Introduce DRM dep queue
Subsystems (9)
Matthew Brost
maintainercover Introduce DRM dep queue
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that drm_gpu_scheduler is no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendered drm_gpu_scheduler unmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.
Key changes from DRM scheduler:
- Unification of drm_gpu_scheduler, drm_sched_entity into drm_dep_queue
- Reference counting
- Clear object-lifetime ownership
- Privatisation of drm dep fence
- Drop scheduled fence from drm dep fence
- Submit workqueue bypass path (optional)
- IRQ removal of jobs (i.e., bypass worker to drop ref to job) (optional)
- Asynchronous teardown
- Extensive lockdep annotations and asserts
- Extensive kernel doc
Xe has been fully converted to drm dep and extensively tested with and without verbose kernel debug Kconfig options. Everything seems to be working locally. All permutations of drm_dep_queue_flags tested.
Compile tested only conversions - Panthor and AMDXDNA.
Other candidates to convert are: Nouveau, PVR, and AMDGPU VM queue.
Forward looking - this design seems suitable for Rust bindings and use by Nova, Tyr given the clean object-lifetime rules and reference counting.
The Xe team is aligned on moving forward with this; hopefully the rest of the community embraces it.
In addition to DRM dep, workqueues have been updated with interfaces to enforce reclaim safety with DRM dep, and Xe is utilizing these interfaces. This part could be split out into a separate series.
I'm also fairly certain that a sane HW queue component could be built on top of DRM dep as well, replacing DRM sched for all drivers they choose.
Matt
Matthew Brost (12):
workqueue: Add interface to teach lockdep to warn on reclaim
violations
drm/dep: Add DRM dependency queue layer
drm/xe: Use WQ_MEM_WARN_ON_RECLAIM on all workqueues in the reclaim
path
drm/xe: Issue GGTT invalidation under lock in ggtt_node_remove drm/xe: Return fence from xe_sched_job_arm and adjust job references drm/xe: Convert to DRM dep queue scheduler layer
drm/xe: Make scheduler message lock IRQ-safe
drm/xe: Rework exec queue object on top of DRM dep drm/xe: Enable IRQ job put in DRM dep
drm/xe: Use DRM dep queue kill semantics
accel/amdxdna: Convert to drm_dep scheduler layer drm/panthor: Convert to drm_dep scheduler layer
drivers/accel/amdxdna/Kconfig | 2 +- drivers/accel/amdxdna/aie2_ctx.c | 144 +- drivers/accel/amdxdna/aie2_pci.h | 4 +- drivers/accel/amdxdna/amdxdna_ctx.c | 5 +- drivers/accel/amdxdna/amdxdna_ctx.h | 4 +- drivers/gpu/drm/Kconfig | 4 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/dep/Makefile | 5 + drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++ drivers/gpu/drm/dep/drm_dep_fence.h | 25 + drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++ drivers/gpu/drm/dep/drm_dep_job.h | 13 + drivers/gpu/drm/dep/drm_dep_queue.c | 1647 ++++++++++++++++++ drivers/gpu/drm/dep/drm_dep_queue.h | 31 + drivers/gpu/drm/panthor/Kconfig | 2 +- drivers/gpu/drm/panthor/panthor_device.c | 5 +- drivers/gpu/drm/panthor/panthor_device.h | 2 +- drivers/gpu/drm/panthor/panthor_drv.c | 35 +- drivers/gpu/drm/panthor/panthor_mmu.c | 160 +- drivers/gpu/drm/panthor/panthor_mmu.h | 14 +- drivers/gpu/drm/panthor/panthor_sched.c | 242 ++- drivers/gpu/drm/panthor/panthor_sched.h | 12 +- drivers/gpu/drm/xe/Kconfig | 2 +- drivers/gpu/drm/xe/tests/xe_migrate.c | 10 +- drivers/gpu/drm/xe/xe_dep_job_types.h | 8 +- drivers/gpu/drm/xe/xe_dep_scheduler.c | 81 +- drivers/gpu/drm/xe/xe_dep_scheduler.h | 7 +- drivers/gpu/drm/xe/xe_device.c | 3 +- drivers/gpu/drm/xe/xe_exec.c | 12 +- drivers/gpu/drm/xe/xe_exec_queue.c | 96 +- drivers/gpu/drm/xe/xe_exec_queue.h | 9 +- drivers/gpu/drm/xe/xe_exec_queue_types.h | 10 +- drivers/gpu/drm/xe/xe_execlist.c | 39 +- drivers/gpu/drm/xe/xe_execlist_types.h | 4 - drivers/gpu/drm/xe/xe_ggtt.c | 12 +- drivers/gpu/drm/xe/xe_gpu_scheduler.c | 66 +- drivers/gpu/drm/xe/xe_gpu_scheduler.h | 67 +- drivers/gpu/drm/xe/xe_gpu_scheduler_types.h | 9 +- drivers/gpu/drm/xe/xe_gsc.c | 5 +- drivers/gpu/drm/xe/xe_gsc_submit.c | 5 +- drivers/gpu/drm/xe/xe_gt.c | 8 +- drivers/gpu/drm/xe/xe_guc_ct.c | 3 +- drivers/gpu/drm/xe/xe_guc_exec_queue_types.h | 14 +- drivers/gpu/drm/xe/xe_guc_submit.c | 360 ++-- drivers/gpu/drm/xe/xe_migrate.c | 27 +- drivers/gpu/drm/xe/xe_oa.c | 5 +- drivers/gpu/drm/xe/xe_pt.c | 2 +- drivers/gpu/drm/xe/xe_pxp_submit.c | 10 +- drivers/gpu/drm/xe/xe_sched_job.c | 59 +- drivers/gpu/drm/xe/xe_sched_job.h | 9 +- drivers/gpu/drm/xe/xe_sched_job_types.h | 8 +- drivers/gpu/drm/xe/xe_sync.c | 2 +- drivers/gpu/drm/xe/xe_tlb_inval.c | 3 +- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 86 +- include/drm/drm_dep.h | 597 +++++++ include/linux/workqueue.h | 3 + include/trace/events/amdxdna.h | 12 +- kernel/workqueue.c | 41 + 58 files changed, 4268 insertions(+), 864 deletions(-) create mode 100644 drivers/gpu/drm/dep/Makefile create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h create mode 100644 include/drm/drm_dep.h
Signature · 1 line
2.34.1
Matthew Brost
maintainerRFC PATCH 1/12 workqueue: Add interface to teach lockdep to warn on reclaim violations
Drivers often use workqueues that run in reclaim paths (e.g., DRM scheduler workqueues). It is useful to teach lockdep that memory allocations which can recurse into reclaim (e.g., GFP_KERNEL) are not allowed on these workqueues. Add an interface that taints a workqueue’s lockdep state with reclaim.
Also add a helper to test whether a workqueue is reclaim annotated, allowing drivers to enforce reclaim-safe behavior.
Example of lockdep splat upon violation below:
[ 60.953095] =============================================
[ 73.023656] Console: switching to colour dummy device 80x25 [ 73.023684] [IGT] xe_exec_reset: executing [ 73.038237] [IGT] xe_exec_reset: starting subtest gt-reset [ 73.044163] xe 0000:03:00.0: [drm] Tile0: GT0: trying reset from force_reset_write [xe] [ 73.044276] xe 0000:03:00.0: [drm] Tile0: GT0: reset queued
[ 73.045963] ======================================================
[ 73.052133] WARNING: possible circular locking dependency detected
[ 73.058302] 7.0.0-rc3-xe+ #31 Tainted: G U
[ 73.063866] ------------------------------------------------------
[ 73.070036] kworker/u64:5/158 is trying to acquire lock:
[ 73.075342] ffffffff829a87a0 (fs_reclaim){+.+.}-{0:0}, at: __kmalloc_cache_noprof+0x39/0x420
[ 73.083791]
but task is already holding lock:
[ 73.089612] ffffc9000152fe60 ((work_completion)(>->reset.worker)){+.+.}-{0:0}, at: process_one_work+0x1d2/0x6a0
[ 73.099852]
which lock already depends on the new lock.[ 73.108013]
the existing dependency chain (in reverse order) is:
[ 73.115481]
-> #2 ((work_completion)(>->reset.worker)){+.+.}-{0:0}:
[ 73.123381] process_one_work+0x1ec/0x6a0
[ 73.127906] worker_thread+0x183/0x330
[ 73.132173] kthread+0xe2/0x120
[ 73.135833] ret_from_fork+0x289/0x2f0
[ 73.140101] ret_from_fork_asm+0x1a/0x30
[ 73.144540]
-> #1 ((wq_completion)gt-ordered-wq){+.+.}-{0:0}:
[ 73.151749] workqueue_warn_on_reclaim.part.0+0x32/0x50
[ 73.157487] alloc_workqueue_noprof+0xef/0x100
[ 73.162445] xe_gt_alloc+0x92/0x220 [xe]
[ 73.166954] xe_pci_probe+0x734/0x1660 [xe]
[ 73.171720] pci_device_probe+0x98/0x140
[ 73.176161] really_probe+0xcf/0x2c0
[ 73.180256] __driver_probe_device+0x6e/0x120
[ 73.185126] driver_probe_device+0x19/0x90
[ 73.189740] __driver_attach+0x89/0x140
[ 73.194091] bus_for_each_dev+0x79/0xd0
[ 73.198446] bus_add_driver+0xe6/0x210
[ 73.202712] driver_register+0x5b/0x110
[ 73.207064] 0xffffffffa00aa0db
[ 73.210724] do_one_initcall+0x59/0x2e0
[ 73.215077] do_init_module+0x5f/0x230
[ 73.219345] init_module_from_file+0xc7/0xe0
[ 73.224128] idempotent_init_module+0x176/0x270
[ 73.229175] __x64_sys_finit_module+0x61/0xb0
[ 73.234047] do_syscall_64+0x9b/0x540
[ 73.238228] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 73.243793]
-> #0 (fs_reclaim){+.+.}-{0:0}:
[ 73.249442] __lock_acquire+0x1496/0x2510
[ 73.253970] lock_acquire+0xbd/0x2f0
[ 73.258062] fs_reclaim_acquire+0x98/0xd0
[ 73.262586] __kmalloc_cache_noprof+0x39/0x420
[ 73.267545] gt_reset_worker+0x27/0x1f0 [xe]
[ 73.272385] process_one_work+0x213/0x6a0
[ 73.276910] worker_thread+0x183/0x330
[ 73.281178] kthread+0xe2/0x120
[ 73.284838] ret_from_fork+0x289/0x2f0
[ 73.289104] ret_from_fork_asm+0x1a/0x30
[ 73.293542]
other info that might help us debug this:[ 73.301528] Chain exists of:
fs_reclaim --> (wq_completion)gt-ordered-wq --> (work_completion)(>->reset.worker)[ 73.314795] Possible unsafe locking scenario:
[ 73.320705] CPU0 CPU1
[ 73.325232] ---- ----
[ 73.329759] lock((work_completion)(>->reset.worker));
[ 73.335148] lock((wq_completion)gt-ordered-wq);
[ 73.342359] lock((work_completion)(>->reset.worker));
[ 73.350259] lock(fs_reclaim);
[ 73.353400]
*** DEADLOCK ***v2:
- Add WQ flag to warn on reclaim violations (Tejun)
- Add a helper function to test if WQ is annotated
--- include/linux/workqueue.h | 3 +++ kernel/workqueue.c | 41 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+)
include/linux/workqueue.h +3 -0
@@ -403,6 +403,7 @@ enum wq_flags {*/ WQ_POWER_EFFICIENT =1 <<7 ,WQ_PERCPU =1 <<8 ,/* bound to a specific cpu */ +WQ_MEM_WARN_ON_RECLAIM =1 <<9 ,/* teach lockdep to warn on reclaim */ __WQ_DESTROYING =1 <<15 ,/* internal: workqueue is destroying */ __WQ_DRAINING = 1 << 16, /* internal: workqueue is draining */
@@ -582,6 +583,8 @@ alloc_workqueue_lockdep_map(const char *fmt, unsigned int flags, int max_active,extern void destroy_workqueue (struct workqueue_struct *wq); +extern bool workqueue_is_reclaim_annotated (struct workqueue_struct *wq); +struct workqueue_attrs *alloc_workqueue_attrs_noprof (void );#define alloc_workqueue_attrs (...) alloc_hooks(alloc_workqueue_attrs_noprof(__VA_ARGS__))
kernel/workqueue.c +41 -0
59 lines not on this page · raw message
Signature · 1 line
2.34.1
Tejun Heo
maintainerRe: [RFC PATCH 01/12] workqueue: Add interface to teach lockdep to warn on reclaim violations
Sorry about the tardiness. Traveling during spring break. Getting more than I can catch up with each day.
include/linux/workqueue.h enum wq_flags { · line 406
Quoted 5 lines · Matthew Brost
include/linux/workqueue.h +1
-0@@ -403,6 +403,7 @@ enum wq_flags {*/ WQ_POWER_EFFICIENT =1 <<7 ,WQ_PERCPU =1 <<8 ,/* bound to a specific cpu */ + WQ_MEM_WARN_ON_RECLAIM = 1 << 9, /* teach lockdep to warn on reclaim */
Shouldn't this require WQ_MEM_RECLAIM?
Quoted 12 lines
kernel/workqueue.c +12
-0+/** +* workqueue_is_reclaim_annotated() - Test whether a workqueue is annotated for +* reclaim safety +* @wq: workqueue to test +* +* Returns true if @wq is flags have both %WQ_MEM_WARN_ON_RECLAIM and +* %WQ_MEM_RECLAIM set. A workqueue marked with these flags indicates that it +* participates in reclaim paths, and therefore must not perform memory +* allocations that can recurse into reclaim (e.g., GFP_KERNEL is not allowed). +* +* Drivers can use this helper to enforce reclaim-safe behavior on workqueues +* that are created or provided elsewhere in the code.
kernel/workqueue.c static struct workqueue_struct *__alloc_workqueue(const char *fmt, · line 5912
Quoted 10 lines
kernel/workqueue.c +10
-0+* +* Return: +* true if the workqueue is reclaim-annotated, false otherwise. +*/ +bool workqueue_is_reclaim_annotated (struct workqueue_struct *wq) +{ +return (wq->flags &WQ_MEM_WARN_ON_RECLAIM ) && + (wq->flags &WQ_MEM_RECLAIM ); +} +EXPORT_SYMBOL_GPL (workqueue_is_reclaim_annotated);
Why is this function necessary? It feels rather odd to use wq as the source of this information. Shouldn't that be an innate knowledge of the code that's using this?
Thanks.
Signature · 1 line
tejun
Matthew Brost
maintainerRe: [RFC PATCH 01/12] workqueue: Add interface to teach lockdep to warn on reclaim violations
Quoted 12 lines · Tejun Heo
Sorry about the tardiness. Traveling during spring break. Getting more than I can catch up with each day.
include/linux/workqueue.h enum wq_flags { · line 406
Quoted 5 lines · Matthew Brost
include/linux/workqueue.h +1
-0@@ -403,6 +403,7 @@ enum wq_flags {*/ WQ_POWER_EFFICIENT =1 <<7 ,WQ_PERCPU =1 <<8 ,/* bound to a specific cpu */ + WQ_MEM_WARN_ON_RECLAIM = 1 << 9, /* teach lockdep to warn on reclaim */Shouldn't this require
WQ_MEM_RECLAIM?
Yes, so what is suggestion here? If WQ_MEM_WARN_ON_RECLAIM is set without WQ_MEM_RECLAIM fail the WQ creation with -EINVAL?
Quoted 18 lines
Quoted 18 lines
kernel/workqueue.c +18
-0+/** +* workqueue_is_reclaim_annotated() - Test whether a workqueue is annotated for +* reclaim safety +* @wq: workqueue to test +* +* Returns true if @wq is flags have both %WQ_MEM_WARN_ON_RECLAIM and +* %WQ_MEM_RECLAIM set. A workqueue marked with these flags indicates that it +* participates in reclaim paths, and therefore must not perform memory +* allocations that can recurse into reclaim (e.g., GFP_KERNEL is not allowed). +* +* Drivers can use this helper to enforce reclaim-safe behavior on workqueues +* that are created or provided elsewhere in the code. +* +* Return: +* true if the workqueue is reclaim-annotated, false otherwise. +*/ +bool workqueue_is_reclaim_annotated (struct workqueue_struct *wq) +{
Quoted 8 lines
kernel/workqueue.c static struct workqueue_struct *__alloc_workqueue(const char *fmt, · line 5912
Quoted 4 lines
kernel/workqueue.c +4
-0+return (wq->flags &WQ_MEM_WARN_ON_RECLAIM ) && + (wq->flags &WQ_MEM_RECLAIM ); +} +EXPORT_SYMBOL_GPL (workqueue_is_reclaim_annotated);Why is this function necessary? It feels rather odd to use wq as the source of this information. Shouldn't that be an innate knowledge of the code that's using this?
This, for example, would be used in DRM sched (the existing scheduler) or DRM dep (the proposed replacement) to ensure that driver-allocated WQs passed into the layers are created with these flags. DRM sched or DRM dep has strict DMA-fencing, thus reclaim rule that we expect DRM drivers to follow. Historically, DRM drivers have broken these rules quite often, and we no longer want to give them the opportunity to do so—lockdep should enforce them.
Matt
Quoted 5 lines
Thanks.
Signature · 1 line
tejun
Re: [RFC PATCH 01/12] workqueue: Add interface to teach lockdep to warn on reclaim violations
Hello,
Quoted 13 lines · Matthew Brost
Quoted 11 lines · Tejun Heo
Sorry about the tardiness. Traveling during spring break. Getting more than I can catch up with each day.
include/linux/workqueue.h enum wq_flags { · line 406
Quoted 5 lines · Matthew Brost
include/linux/workqueue.h +1
-0@@ -403,6 +403,7 @@ enum wq_flags {*/ WQ_POWER_EFFICIENT =1 <<7 ,WQ_PERCPU =1 <<8 ,/* bound to a specific cpu */ + WQ_MEM_WARN_ON_RECLAIM = 1 << 9, /* teach lockdep to warn on reclaim */Shouldn't this require
WQ_MEM_RECLAIM?
Quoted 2 lines
Yes, so what is suggestion here? If
WQ_MEM_WARN_ON_RECLAIMis set withoutWQ_MEM_RECLAIMfail the WQ creation with-EINVAL?
Yes.
Quoted 11 lines
Quoted 3 lines
Why is this function necessary? It feels rather odd to use wq as the source of this information. Shouldn't that be an innate knowledge of the code that's using this?
This, for example, would be used in DRM sched (the existing scheduler) or DRM dep (the proposed replacement) to ensure that driver-allocated WQs passed into the layers are created with these flags. DRM sched or DRM dep has strict DMA-fencing, thus reclaim rule that we expect DRM drivers to follow. Historically, DRM drivers have broken these rules quite often, and we no longer want to give them the opportunity to do so—lockdep should enforce them.
I see. Yeah, that makes sense. Please feel free to add
Please let me know how you wanna route the patch.
Thanks.
Signature · 1 line
tejun
Matthew Brost
maintainerRe: [RFC PATCH 01/12] workqueue: Add interface to teach lockdep to warn on reclaim violations
Quoted 33 lines · Tejun Heo
Hello,
Quoted 13 lines · Matthew Brost
Quoted 11 lines · Tejun Heo
Sorry about the tardiness. Traveling during spring break. Getting more than I can catch up with each day.
include/linux/workqueue.h enum wq_flags { · line 406
Quoted 5 lines · Matthew Brost
include/linux/workqueue.h +1
-0@@ -403,6 +403,7 @@ enum wq_flags {*/ WQ_POWER_EFFICIENT =1 <<7 ,WQ_PERCPU =1 <<8 ,/* bound to a specific cpu */ + WQ_MEM_WARN_ON_RECLAIM = 1 << 9, /* teach lockdep to warn on reclaim */Shouldn't this require
WQ_MEM_RECLAIM?Quoted 2 lines
Yes, so what is suggestion here? If
WQ_MEM_WARN_ON_RECLAIMis set withoutWQ_MEM_RECLAIMfail the WQ creation with-EINVAL?Yes.
Quoted 11 lines
Quoted 3 lines
Why is this function necessary? It feels rather odd to use wq as the source of this information. Shouldn't that be an innate knowledge of the code that's using this?
This, for example, would be used in DRM sched (the existing scheduler) or DRM dep (the proposed replacement) to ensure that driver-allocated WQs passed into the layers are created with these flags. DRM sched or DRM dep has strict DMA-fencing, thus reclaim rule that we expect DRM drivers to follow. Historically, DRM drivers have broken these rules quite often, and we no longer want to give them the opportunity to do so—lockdep should enforce them.
Quoted 6 lines
I see. Yeah, that makes sense. Please feel free to add
Acked-by Tejun Heo <[email protected]>Please let me know how you wanna route the patch.
Can I send an independent patch addressing the WQ_MEM_WARN_ON_RECLAIM usage above for you to pull into 7.01? I’m still working through a few issues on my driver (Xe) side to enable this, and the larger DRM-level changes will take some time since pretty much every driver is doing something wrong.
Alternatively, we could defer this to 7.02 and merge it through DRM, since the 7.01 DRM cycle is closing today. Either option works for me.
Matt
Quoted 4 lines
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: [RFC PATCH 01/12] workqueue: Add interface to teach lockdep to warn on reclaim violations
Quoted 8 lines · Matthew Brost
Can I send an independent patch addressing the
WQ_MEM_WARN_ON_RECLAIMusage above for you to pull into 7.01? I’m still working through a few issues on my driver (Xe) side to enable this, and the larger DRM-level changes will take some time since pretty much every driver is doing something wrong.Alternatively, we could defer this to 7.02 and merge it through DRM, since the 7.01 DRM cycle is closing today. Either option works for me.
It'd probalby make sense to merge it together when DRM side is ready. Whenever you're ready to merge the users of the interface, please let me know. I'll create a separate workqueue branch with the patch on so that DRM can pull it in.
Thanks.
Signature · 1 line
tejun
Matthew Brost
maintainerRFC PATCH 2/12 drm/dep: Add DRM dependency queue layer
In reply to Matthew BrostDiverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that drm_gpu_scheduler is no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendered drm_gpu_scheduler unmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.
Add drm_dep, a lightweight GPU submission queue intended as a replacement for drm_gpu_scheduler for firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlike drm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination, drm_dep merges both roles into a single struct drm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.
Unlike drm_gpu_scheduler, which relies on external locking and lifetime management by the driver, drm_dep uses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until the put_job worker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.
The core objects are:
struct drm_dep_queue - a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.
struct drm_dep_job - a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of input dma_fence dependencies and produce a drm_dep_fence as their finished fence.
struct drm_dep_fence - a dma_fence subclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).
Job lifecycle:
1. drm_dep_job_init() - allocate and initialise; job acquires a queue reference.
2. drm_dep_job_add_dependency() and friends - register input fences; duplicates from the same context are deduplicated.
3. drm_dep_job_arm() - assign sequence number, obtain finished fence.
4. drm_dep_job_push() - submit to queue.
Submission paths under queue lock:
- Bypass path: if DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and the run_job worker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before calling ops->run_job().
Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds q->job.timeout jiffies, calling ops->timedout_job(). drm_dep_queue_trigger_timeout() forces immediate expiry for device teardown.
IRQ-safe completion: queues flagged DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE allow drm_dep_job_done() to be called from hardirq context (e.g. a dma_fence callback). Dependency cleanup is deferred to process context after ops->run_job() returns to avoid calling xa_destroy() from IRQ.
Zombie-state guard: workers use kref_get_unless_zero() on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.
Teardown is always deferred to a module-private workqueue (dep_free_wq) so that destroy_workqueue() is never called from within one of the queue's own workers. Each queue holds a drm_dev_get() reference on its owning struct drm_device, released as the final step of teardown via drm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.
--- drivers/gpu/drm/Kconfig | 4 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/dep/Makefile | 5 + drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++++ drivers/gpu/drm/dep/drm_dep_fence.h | 25 + drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++++++ drivers/gpu/drm/dep/drm_dep_job.h | 13 + drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++ drivers/gpu/drm/dep/drm_dep_queue.h | 31 + include/drm/drm_dep.h | 597 ++++++++++ 10 files changed, 3404 insertions(+) create mode 100644 drivers/gpu/drm/dep/Makefile create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h create mode 100644 include/drm/drm_dep.h
drivers/gpu/drm/Kconfig +4 -0
@@ -276,6 +276,10 @@ config DRM_SCHED tristate depends on DRM +config DRM_DEP + tristate + depends on DRM + # Separate option as not all DRM drivers use it config DRM_PANEL_BACKLIGHT_QUIRKS tristate
drivers/gpu/drm/Makefile +1 -0
@@ -173,6 +173,7 @@ obj-y += clients/ obj-y += display/ obj-$(CONFIG_DRM_TTM) += ttm/ obj-$(CONFIG_DRM_SCHED) += scheduler/ +obj-$(CONFIG_DRM_DEP) += dep/ obj-$(CONFIG_DRM_RADEON)+= radeon/ obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/ obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/
drivers/gpu/drm/dep/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o + +obj-$(CONFIG_DRM_DEP) += drm_dep.o
drivers/gpu/drm/dep/drm_dep_fence.c new file +406 -0
406 lines not on this page · raw message
drivers/gpu/drm/dep/drm_dep_fence.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_FENCE_H_ +#define _DRM_DEP_FENCE_H_ + +#include <linux/dma-fence.h> + +struct drm_dep_fence ; +struct drm_dep_queue ; + +struct drm_dep_fence *drm_dep_fence_alloc (void ); +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q); +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence); +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent); +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result); +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence); +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence); + +#endif /* _DRM_DEP_FENCE_H_ */
drivers/gpu/drm/dep/drm_dep_job.c new file +675 -0
675 lines not on this page · raw message
drivers/gpu/drm/dep/drm_dep_job.h new file +13 -0
@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_JOB_H_ +#define _DRM_DEP_JOB_H_ + +struct drm_dep_queue ; + +void drm_dep_job_drop_dependencies (struct drm_dep_job *job); + +#endif /* _DRM_DEP_JOB_H_ */
drivers/gpu/drm/dep/drm_dep_queue.c new file +1647 -0
1,647 lines not on this page · raw message
drivers/gpu/drm/dep/drm_dep_queue.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_QUEUE_H_ +#define _DRM_DEP_QUEUE_H_ + +#include <linux/types.h> + +struct drm_dep_job ; +struct drm_dep_queue ; + +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job); +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job); +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q); +void drm_dep_queue_push_job_end (struct drm_dep_queue *q); +#else +static inline void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +} +static inline void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +} +#endif + +#endif /* _DRM_DEP_QUEUE_H_ */
include/drm/drm_dep.h new file +597 -0
181 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
Part of the diff is not shown here. View the raw message.
Boris Brezillon
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Hi Matthew,
Quoted 27 lines · On Sun, 15 Mar 2026 21:32:45 -0700 Matthew Brost <[email protected]>…
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until theput_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.
Quoted 6 lines
The core objects are:
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.
First of, I like this idea, and actually think we should have done that from the start rather than trying to bend drm_sched to meet our FW-assisted scheduling model. That's also the direction me and Danilo have been pushing for for the new JobQueue stuff in rust, so I'm glad to see some consensus here.
Now, let's start with the usual naming nitpick :D => can't we find a better prefix than "drm_dep"? I think I get where "dep" comes from (the logic mostly takes care of job deps, and acts as a FIFO otherwise, no real scheduling involved). It's kinda okay for drm_dep_queue, even though, according to the description you've made, jobs seem to stay in that queue even after their deps are met, which, IMHO, is a bit
confusing: dep_queue sounds like a queue in which jobs are placed until their deps are met, and then the job moves to some other queue.
It gets worse for drm_dep_job, which sounds like a dep-only job, rather than a job that's queued to the drm_dep_queue. Same goes for drm_dep_fence, which I find super confusing. What this one does is just proxy the driver fence to provide proper isolation between GPU drivers and fence observers (other drivers).
Since this new model is primarily designed for hardware that have FW-assisted scheduling, how about drm_fw_queue, drm_fw_job, drm_fw_job_fence?
Quoted 19 lines
struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.
Quoted 4 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
I've yet to look at the code, but I must admit I'm less worried about this fast path if it's part of a new model restricted to FW-assisted scheduling. I keep thinking we're not entirely covered for so called real-time GPU contexts that might have jobs that are not dep-free, and if we're going for something new, I'd really like us to consider that case from the start (maybe investigate if kthread_work[er] can be used as a replacement for workqueues, if RT priority on workqueues is not an option).
Quoted 23 lines
- Queued path: job is pushed onto the SPSC queue and the
run_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.
Quoted 6 lines
Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.
Thanks for posting this RFC. I'll try to have a closer look at the code in the coming days, but given the diffstat, it might take me a bit of time...
Regards,
Boris
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 34 lines · Boris Brezillon
Hi Matthew,
Quoted 29 lines · On Sun, 15 Mar 2026 21:32:45 -0700 Matthew Brost <[email protected]>…
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until theput_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.The core objects are:
Quoted 7 lines
Quoted 4 lines
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.First of, I like this idea, and actually think we should have done that from the start rather than trying to bend
drm_schedto meet our
Yes. Tvrtko actually suggested this years ago, and in my naïveté I rejected it. I’m eating my hat here.
Quoted 19 lines
FW-assisted scheduling model. That's also the direction me and Danilo have been pushing for for the new JobQueue stuff in rust, so I'm glad to see some consensus here.
Now, let's start with the usual naming nitpick :D => can't we find a better prefix than "
drm_dep"? I think I get where "dep" comes from (the logic mostly takes care of job deps, and acts as a FIFO otherwise, no real scheduling involved). It's kinda okay fordrm_dep_queue, even though, according to the description you've made, jobs seem to stay in that queue even after their deps are met, which, IMHO, is a bit
confusing:dep_queuesounds like a queue in which jobs are placed until their deps are met, and then the job moves to some other queue.It gets worse for
drm_dep_job, which sounds like a dep-only job, rather than a job that's queued to thedrm_dep_queue. Same goes fordrm_dep_fence, which I find super confusing. What this one does is just proxy the driver fence to provide proper isolation between GPU drivers and fence observers (other drivers).
Quoted 3 lines
Since this new model is primarily designed for hardware that have FW-assisted scheduling, how about
drm_fw_queue,drm_fw_job,drm_fw_job_fence?
We can bikeshed — I’m open to other names, but I believe hardware scheduling can be built quite cleanly on top of this, so drm_fw_* doesn’t really work either. Check out a hardware-scheduler PoC built (today) on top of this in [1].
Quoted 25 lines
Quoted 20 lines
struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.Quoted 4 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
Quoted 9 lines
I've yet to look at the code, but I must admit I'm less worried about this fast path if it's part of a new model restricted to FW-assisted scheduling. I keep thinking we're not entirely covered for so called real-time GPU contexts that might have jobs that are not dep-free, and if we're going for something new, I'd really like us to consider that case from the start (maybe investigate if kthread_work[er] can be used as a replacement for workqueues, if RT priority on workqueues is not an option).
I mostly agree, and I’ll look into whether kthread_work is better suited—if that’s the right model, it should be done up front.
But can you give a use case for real-time GPU contexts that are not dep-free? I personally don’t know of one.
Quoted 30 lines
Quoted 23 lines
- Queued path: job is pushed onto the SPSC queue and the
run_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.Quoted 6 lines
Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.
Quoted 3 lines
Thanks for posting this RFC. I'll try to have a closer look at the code in the coming days, but given the diffstat, it might take me a bit of time...
I understand — I’m a firehose when I get started. Hopefully a sane one, though.
Matt
Quoted 4 lines
Regards,
Boris
Boris Brezillon
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Hi Matthew,
Quoted 65 lines · On Mon, 16 Mar 2026 22:22:30 -0700 Matthew Brost <[email protected]>…
Quoted 34 lines · Boris Brezillon
Hi Matthew,
Quoted 29 lines · On Sun, 15 Mar 2026 21:32:45 -0700 Matthew Brost <[email protected]>…
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until theput_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.The core objects are:
Quoted 7 lines
Quoted 4 lines
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.First of, I like this idea, and actually think we should have done that from the start rather than trying to bend
drm_schedto meet ourYes. Tvrtko actually suggested this years ago, and in my naïveté I rejected it. I’m eating my hat here.
Quoted 19 lines
FW-assisted scheduling model. That's also the direction me and Danilo have been pushing for for the new JobQueue stuff in rust, so I'm glad to see some consensus here.
Now, let's start with the usual naming nitpick :D => can't we find a better prefix than "
drm_dep"? I think I get where "dep" comes from (the logic mostly takes care of job deps, and acts as a FIFO otherwise, no real scheduling involved). It's kinda okay fordrm_dep_queue, even though, according to the description you've made, jobs seem to stay in that queue even after their deps are met, which, IMHO, is a bit
confusing:dep_queuesounds like a queue in which jobs are placed until their deps are met, and then the job moves to some other queue.It gets worse for
drm_dep_job, which sounds like a dep-only job, rather than a job that's queued to thedrm_dep_queue. Same goes fordrm_dep_fence, which I find super confusing. What this one does is just proxy the driver fence to provide proper isolation between GPU drivers and fence observers (other drivers).
Quoted 7 lines
Quoted 3 lines
Since this new model is primarily designed for hardware that have FW-assisted scheduling, how about
drm_fw_queue,drm_fw_job,drm_fw_job_fence?We can bikeshed — I’m open to other names, but I believe hardware scheduling can be built quite cleanly on top of this, so drm_fw_* doesn’t really work either.
I agree drm_fw_ is not great either. I think Philipp's name, JobQueue, is a better fit.
Quoted 2 lines
Check out a hardware-scheduler PoC built
(today) on top of this in [1].
Yeah, I'm pretty sure we can add a layer on top to deal with HW GPU queues (Danilo and Philipp proposed it already), I'm just not sure it should be our primary focus, at least not until we have something usable for the drivers that are currently written in rust. Doesn't mean we shouldn't think about it and design the thing so it can be added later, of course, but most of the time, perfect is the enemy of good.
Quoted 38 lines
Quoted 25 lines
Quoted 20 lines
struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.Quoted 4 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.Quoted 9 lines
I've yet to look at the code, but I must admit I'm less worried about this fast path if it's part of a new model restricted to FW-assisted scheduling. I keep thinking we're not entirely covered for so called real-time GPU contexts that might have jobs that are not dep-free, and if we're going for something new, I'd really like us to consider that case from the start (maybe investigate if kthread_work[er] can be used as a replacement for workqueues, if RT priority on workqueues is not an option).
Quoted 5 lines
I mostly agree, and I’ll look into whether
kthread_workis better suited—if that’s the right model, it should be done up front.But can you give a use case for real-time GPU contexts that are not dep-free? I personally don’t know of one.
Let alone real-time GPU contexts, you still have the concept of context priority in both GL and Vulkan, on which jobs with deps are likely to be queued, and if the dequeuing thread doesn't take this priority into consideration, there's nothing differentiating a low-prio from an high-prio context, thus partially defeating the priority you assign to your FW queue (you can execute fast, but if the queuing to the FW queue is slow, it's pointless).
As for real-time contexts, yes, right now the only use case I can think of is compositors, and apparently those would pass dep-less submissions, but claiming that this is the only use case we'll ever have for those RT contexts is a bit of a risk I'd rather not take. Also, if we solve the problem with kthreads with prios, it might be that we don't even need this fastpath in the first place.
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
- Queued path: job is pushed onto the SPSC queue and the
run_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.
Quoted 7 lines
Quoted 3 lines
Thanks for posting this RFC. I'll try to have a closer look at the code in the coming days, but given the diffstat, it might take me a bit of time...
I understand — I’m a firehose when I get started. Hopefully a sane one, though.
One last note: I've seen a bunch of discussions that start to look like some pissing contest between C and rust advocates. I've only started to look at rust recently, in the context of Tyr (the rust re-implementation of Panthor), and there's niceties in the language that I think make it a good fit for the new JobQueue thing we've been discussing (Daniel listed a bunch of them). I don't really mind who's going to end up winning the trophy of this contest to be honest, but what I certainly don't want is for us to be stuck with no solution at all because each camp claims that their solution is best and none of them give up on their ideas (basically what happened with the AGX driver). So please, please, let's all take a step back, have a look at what each side has done, keep the discussion constructive, and hopefully we'll end up with a mix of ideas that makes the final solution better.
Regards,
Boris
Danilo Krummrich
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Matthew BrostQuoted 8 lines · Matthew Brost
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.
I think we all agree on this and I also think we all agree that this should have been a separate component in the first place -- and just to be clear, I am saying this in retrospective.
In fact, this is also the reason why I proposed building the Rust component differently, i.e. start with a Joqueue (or drm_dep as called in this patch) and expand as needed with a loosely coupled "orchestrator" for drivers with strictly limited software/hardware queues later.
The reason I proposed a new component for Rust, is basically what you also wrote in your cover letter, plus the fact that it prevents us having to build a Rust abstraction layer to the DRM GPU scheduler.
The latter I identified as pretty questionable as building another abstraction layer on top of some infrastructure is really something that you only want to do when it is mature enough in terms of lifetime and ownership model.
I'm not saying it wouldn't be possible, but as mentioned in other threads, I don't think it is a good idea building new features on top of something that has known problems, even less when they are barely resolvable due to other existing dependencies, such as some drivers relying on implementation details historically, etc.
My point is, the justification for a new Jobqueue component in Rust I consider given by the fact that it allows us to avoid building another abstraction layer on top of DRM sched. Additionally, DRM moves to Rust and gathering experience with building native Rust components seems like a good synergy in this context.
Having that said, the obvious question for me for this series is how drm_dep fits into the bigger picture.
I.e. what is the maintainance strategy?
Do we want to support three components allowing users to do the same thing? What happens to DRM sched for 1:1 entity / scheduler relationships?
Is it worth? Do we have enough C users to justify the maintainance of yet another component? (Again, DRM moves into the direction of Rust drivers, so I don't know how many new C drivers we will see.) I.e. having this component won't get us rid of the majority of DRM sched users.
What are the expected improvements? Given the above, I'm not sure it will actually decrease the maintainance burdon of DRM sched.
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 10 lines · Danilo Krummrich
Quoted 8 lines · Matthew Brost
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.
Quoted 3 lines
I think we all agree on this and I also think we all agree that this should have been a separate component in the first place -- and just to be clear, I am saying this in retrospective.
Yes. Tvrtko actually suggested this years ago, and in my naïveté I rejected it. I’m eating my hat here.
Quoted 5 lines
In fact, this is also the reason why I proposed building the Rust component differently, i.e. start with a Joqueue (or
drm_depas called in this patch) and expand as needed with a loosely coupled "orchestrator" for drivers with strictly limited software/hardware queues later.
Yes, I actually have a hardware-scheduling layer built on top of drm_dep
[1] after hacking for several hours today. It’s very unlikely to actually work since I’m typing blind without a test platform, but it conceptually proves that this layer separation works and is clean.
Quoted 9 lines
The reason I proposed a new component for Rust, is basically what you also wrote in your cover letter, plus the fact that it prevents us having to build a Rust abstraction layer to the DRM GPU scheduler.
The latter I identified as pretty questionable as building another abstraction layer on top of some infrastructure is really something that you only want to do when it is mature enough in terms of lifetime and ownership model.
I personally don’t think the language matters that much. I care about lifetime, ownership, and teardown semantics. I believe I’ve made this clear in C, so the Rust bindings should be trivial.
Quoted 6 lines
I'm not saying it wouldn't be possible, but as mentioned in other threads, I don't think it is a good idea building new features on top of something that has known problems, even less when they are barely resolvable due to other existing dependencies, such as some drivers relying on implementation details historically, etc.
It’s a new component, well thought out and without any baggage, so I don’t understand the above statement. Invariants and annotations everywhere (e.g., you cannot abuse this).
Quoted 5 lines
My point is, the justification for a new Jobqueue component in Rust I consider given by the fact that it allows us to avoid building another abstraction layer on top of DRM sched. Additionally, DRM moves to Rust and gathering experience with building native Rust components seems like a good synergy in this context.
If I knew Rust off-hand, I would have written it in Rust :). Perhaps this is an opportunity to learn. But I think the Rust vs. C holy war isn’t in scope here. The real questions are what semantics we want, the timeline, and maintainability. Certainly more people know C, and most drivers are written in C, so having the common component in C makes more sense at this point, in my opinion. If the objection is really about the language, I’ll rewrite it in Rust.
Quoted 5 lines
Having that said, the obvious question for me for this series is how
drm_depfits into the bigger picture.I.e. what is the maintainance strategy?
I will commit to maintaining code I believe in, and immediately write the bindings on top of this so they’re maintained from day one.
Quoted 8 lines
Do we want to support three components allowing users to do the same thing? What happens to DRM sched for 1:1 entity / scheduler relationships?
Is it worth? Do we have enough C users to justify the maintainance of yet another component? (Again, DRM moves into the direction of Rust drivers, so I don't know how many new C drivers we will see.) I.e. having this component won't get us rid of the majority of DRM sched users.
Actually, with [1], I’m fairly certain that pretty much every driver could convert to this new code. Part of the problem, though, is that when looking at this, multiple drivers clearly break dma-fencing rules, so an annotated component like DRM dep would explode their drivers. Not to mention the many driver-side hacks that each individual driver would need to drop (e.g., I would not be receptive to any driver directly touching drm_dep object structs).
Quoted 1 line
What are the expected improvements? Given the above, I'm not sure it will
Clear object model and lifetimes — therefore memory-safe. Bypass paths for compositors, compute, and kernel page-fault handlers. No kicking a worker just to drop a ref; asynchronous teardown (e.g., user hits Ctrl‑C and returns). Reclaim-safe final puts of queues, and built‑in driver-unload barriers. Maintainable, as I understand every single LOC, with verbose documentation (generated with Copilot, but I’ve reviewed it multiple times and it’s correct), etc.
Regardless, given all of the above, at a minimum my driver needs to move on one way or another.
Quoted 1 line
actually decrease the maintainance burdon of DRM sched.
We can deprecate DRM sched, which is now possible as of [1]. I can commit to compile-testing most drivers, aside from the ones with the horrible hacks.
Matt
Danilo Krummrich
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 5 lines · Matthew Brost
Quoted 4 lines · Danilo Krummrich
The reason I proposed a new component for Rust, is basically what you also wrote in your cover letter, plus the fact that it prevents us having to build a Rust abstraction layer to the DRM GPU scheduler.
Quoted 8 lines
Quoted 4 lines
The latter I identified as pretty questionable as building another abstraction layer on top of some infrastructure is really something that you only want to do when it is mature enough in terms of lifetime and ownership model.
I personally don’t think the language matters that much. I care about lifetime, ownership, and teardown semantics. I believe I’ve made this clear in C, so the Rust bindings should be trivial.
No, they won't be trivial -- in fact, in the case of the Jobqueue they may even end up being more complicated than a native implementation.
We still want to build the object model around it that allows us to catch most of the pitfalls at compile time rather than runtime.
For instance, there has been a proposal of having specific work and workqueue types that ensure not to violate DMA fence rules, which the Jobqueue can adopt.
We can also use Klint to ensure correctness for those types at compile time.
So, I can easily see this becoming more complicated when we have to go through an FFI layer that makes us loose additional type information / guarantees.
Anyways, I don't want to argue about this. I don't know why the whole thread took this direction.
This is not about C vs. Rust, and I see the Rust component to be added regardless of this effort.
The question for me is whether we want to have a second component besides the GPU scheduler on the C side or not.
If we can replace the existing scheduler entirely and rework all the drivers that'd be great and you absolutely have my blessing.
But, I don't want to end up in a situation where this is landed, one or two drivers are converted, and everything else is left behind in terms of maintainance / maintainer commitment.
Quoted 6 lines
Quoted 5 lines
My point is, the justification for a new Jobqueue component in Rust I consider given by the fact that it allows us to avoid building another abstraction layer on top of DRM sched. Additionally, DRM moves to Rust and gathering experience with building native Rust components seems like a good synergy in this context.
Quoted 7 lines
If I knew Rust off-hand, I would have written it in Rust :). Perhaps this is an opportunity to learn. But I think the Rust vs. C holy war isn’t in scope here. The real questions are what semantics we want, the timeline, and maintainability. Certainly more people know C, and most drivers are written in C, so having the common component in C makes more sense at this point, in my opinion. If the objection is really about the language, I’ll rewrite it in Rust.
Again, I'm not talking about Rust vs. C. I'm talking about why a new Rust component is much easier to justify maintainance wise than a new C component is.
That is, the existing infrastructure has problems we don't want to build on top of and the abstraction ends up being of a similar magnitude as a native implementation.
A new C implementation alongside the existing one is a whole different question.
Quoted 8 lines
Quoted 5 lines
Having that said, the obvious question for me for this series is how
drm_depfits into the bigger picture.I.e. what is the maintainance strategy?
I will commit to maintaining code I believe in, and immediately write the bindings on top of this so they’re maintained from day one.
This I am sure about, but what about the existing scheduler infrastructure? Are you going to keep this up as well?
Who keeps supporting it for all the drivers that can't switch (due to not having firmware queues) or simply did not switch yet?
Quoted 9 lines
Quoted 8 lines
Do we want to support three components allowing users to do the same thing? What happens to DRM sched for 1:1 entity / scheduler relationships?
Is it worth? Do we have enough C users to justify the maintainance of yet another component? (Again, DRM moves into the direction of Rust drivers, so I don't know how many new C drivers we will see.) I.e. having this component won't get us rid of the majority of DRM sched users.
Quoted 7 lines
Actually, with [1], I’m fairly certain that pretty much every driver could convert to this new code. Part of the problem, though, is that when looking at this, multiple drivers clearly break dma-fencing rules, so an annotated component like DRM dep would explode their drivers. Not to mention the many driver-side hacks that each individual driver would need to drop (e.g., I would not be receptive to any driver directly touching
drm_depobject structs).
I thought the API can't be abused? :) How would you prevent drivers doing this in practice? They need to have the struct definition, and once they have it, you can't do anything about them peeking internals, if not caught through review.
Quoted 3 lines
Maintainable, as I understand every single LOC, with verbose documentation (generated with Copilot, but I’ve reviewed it multiple times and it’s correct), etc.
I'm not sure this is the core criteria for evaluating whether something is maintainable or not.
To be honest, this does not sound very community focused.
Quoted 2 lines
Regardless, given all of the above, at a minimum my driver needs to move on one way or another.
Your driver? What do you mean with "it has to move"?
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 15 lines · Danilo Krummrich
Quoted 5 lines · Matthew Brost
Quoted 4 lines · Danilo Krummrich
The reason I proposed a new component for Rust, is basically what you also wrote in your cover letter, plus the fact that it prevents us having to build a Rust abstraction layer to the DRM GPU scheduler.
Quoted 8 lines
Quoted 4 lines
The latter I identified as pretty questionable as building another abstraction layer on top of some infrastructure is really something that you only want to do when it is mature enough in terms of lifetime and ownership model.
I personally don’t think the language matters that much. I care about lifetime, ownership, and teardown semantics. I believe I’ve made this clear in C, so the Rust bindings should be trivial.
Quoted 6 lines
No, they won't be trivial -- in fact, in the case of the Jobqueue they may even end up being more complicated than a native implementation.
We still want to build the object model around it that allows us to catch most of the pitfalls at compile time rather than runtime.
I do understand this line of thinking.
Quoted 3 lines
For instance, there has been a proposal of having specific work and workqueue types that ensure not to violate DMA fence rules, which the Jobqueue can adopt.
Yes, want to add that too for C (ofc this can't be enforced at compile time).
I've started on this here [1]
https://patchwork.freedesktop.org/patch/711929/?series=163245&rev=1
Quoted 7 lines
We can also use Klint to ensure correctness for those types at compile time.
So, I can easily see this becoming more complicated when we have to go through an FFI layer that makes us loose additional type information / guarantees.
Anyways, I don't want to argue about this. I don't know why the whole thread took this direction.
Yea, my bad. I apologize. This shouldn't be about is Rust or C better.
Quoted 3 lines
This is not about C vs. Rust, and I see the Rust component to be added regardless of this effort.
+1.
Quoted 7 lines
The question for me is whether we want to have a second component besides the GPU scheduler on the C side or not.
If we can replace the existing scheduler entirely and rework all the drivers that'd be great and you absolutely have my blessing.
I'll copy paste my reply from [2].
The only weird case I haven't wrapped my head around quite yet is the ganged submissions that rely on the scheduled fence (PVR, AMDGPU do this). Pretty much every other driver seems like it could be coverted with what I have in place in this series + local work to provide a hardware scheduler...
[2] https://patchwork.freedesktop.org/patch/711933/?series=163245&rev=1#comment_1311418
Quoted 4 lines
But, I don't want to end up in a situation where this is landed, one or two drivers are converted, and everything else is left behind in terms of maintainance / maintainer commitment.
I can get other drivers to compile but I can't do things like test these changes for other vendors. Also if existing drivers break dma-fencing rules, those would need to fixed when converting over too.
Quoted 26 lines
Quoted 6 lines
Quoted 5 lines
My point is, the justification for a new Jobqueue component in Rust I consider given by the fact that it allows us to avoid building another abstraction layer on top of DRM sched. Additionally, DRM moves to Rust and gathering experience with building native Rust components seems like a good synergy in this context.
Quoted 7 lines
If I knew Rust off-hand, I would have written it in Rust :). Perhaps this is an opportunity to learn. But I think the Rust vs. C holy war isn’t in scope here. The real questions are what semantics we want, the timeline, and maintainability. Certainly more people know C, and most drivers are written in C, so having the common component in C makes more sense at this point, in my opinion. If the objection is really about the language, I’ll rewrite it in Rust.
Again, I'm not talking about Rust vs. C. I'm talking about why a new Rust component is much easier to justify maintainance wise than a new C component is.
That is, the existing infrastructure has problems we don't want to build on top of and the abstraction ends up being of a similar magnitude as a native implementation.
A new C implementation alongside the existing one is a whole different question.
Quoted 3 lines
Quoted 3 lines
Having that said, the obvious question for me for this series is how
drm_depfits into the bigger picture.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
I.e. what is the maintainance strategy?
I will commit to maintaining code I believe in, and immediately write the bindings on top of this so they’re maintained from day one.
This I am sure about, but what about the existing scheduler infrastructure? Are
+1
Quoted 1 line
you going to keep this up as well?
The fact is we can't maintain DRM sched as it is now, techincal debt is just too high.
Quoted 4 lines
Who keeps supporting it for all the drivers that can't switch (due to not having firmware queues) or simply did not switch yet?
I'd say I agree with your no new features statement in DRM sched, if you want new features in C, fix your driver to use what I have here.
Ofc if bugs pop up in drm sched, happy help fix those.
Quoted 17 lines
Quoted 9 lines
Quoted 8 lines
Do we want to support three components allowing users to do the same thing? What happens to DRM sched for 1:1 entity / scheduler relationships?
Is it worth? Do we have enough C users to justify the maintainance of yet another component? (Again, DRM moves into the direction of Rust drivers, so I don't know how many new C drivers we will see.) I.e. having this component won't get us rid of the majority of DRM sched users.
Quoted 7 lines
Actually, with [1], I’m fairly certain that pretty much every driver could convert to this new code. Part of the problem, though, is that when looking at this, multiple drivers clearly break dma-fencing rules, so an annotated component like DRM dep would explode their drivers. Not to mention the many driver-side hacks that each individual driver would need to drop (e.g., I would not be receptive to any driver directly touching
drm_depobject structs).
Quoted 4 lines
I thought the API can't be abused? :) How would you prevent drivers doing this in practice? They need to have the struct definition, and once they have it, you can't do anything about them peeking internals, if not caught through review.
I agree with this statement. So it would require disciplined reviews...
e.g., You are touching this struct directly - why? Is is drm dep missing something in the API your driver needs, if it is real issue we add to drm dep.
Quoted 9 lines
Quoted 3 lines
Maintainable, as I understand every single LOC, with verbose documentation (generated with Copilot, but I’ve reviewed it multiple times and it’s correct), etc.
I'm not sure this is the core criteria for evaluating whether something is maintainable or not.
To be honest, this does not sound very community focused.
That is not my intent. I did write this as common component for driver and to used by the community.
Quoted 4 lines
Quoted 2 lines
Regardless, given all of the above, at a minimum my driver needs to move on one way or another.
Your driver? What do you mean with "it has to move"?
DRM sched is not maintainable, thus it is not going to fit the needs of Xe in the longterm. Other drivers like are facing similar issues.
Matt
Daniel Almeida
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Matthew Brost(+cc a few other people + Rust-for-Linux ML)
Hi Matthew,
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Quoted 21 lines · Matthew Brost
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.
Quoted 3 lines
Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue
In a domain that has been plagued by lifetime issues, we really should be enforcing RAII for resource management instead of manual calls.
Quoted 4 lines
reference from init until its last put, and the queue holds a job reference from dispatch until the
put_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.
It makes use-after-free impossible _if_ you’re careful. It is not a property of the type system, and incorrect code will compile just fine.
Quoted 18 lines
The core objects are:
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).
Quoted 7 lines
Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.
You cannot enforce this sequence easily in C code. Once again, we are trusting drivers that it is followed, but in Rust, you can simply reject code that does not follow this order at compile time.
Quoted 10 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and therun_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().
Quoted 3 lines
Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Why can’t we design an API where the driver can refuse jobs in ops->run_job() if there are no resources to run it? This would do away with the credit system that has been in place for quite a while. Has this approach been tried in the past?
Quoted 6 lines
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.
Quoted 8 lines
IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.
In rust, when you queue work, you have to pass a reference-counted pointer (Arc<T>). We simply never have this problem in a Rust design. If there is work queued, the queue is alive.
By the way, why can’t we simply require synchronous teardowns?
Quoted 96 lines
Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.Cc Boris Brezillon <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Rodrigo Vivi <[email protected]>Cc Thomas Hellström <[email protected]>Cc Christian König <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Maarten Lankhorst <[email protected]>Cc Maxime Ripard <[email protected]>Cc Philipp Stanner <[email protected]>Cc Simona Vetter <[email protected]>Cc Sumit Semwal <[email protected]>Cc Thomas Zimmermann <[email protected]>Signed-off-by Matthew Brost <[email protected]>Assisted-by GitHub Copilot:claude-sonnet-4.6---
drivers/gpu/drm/Kconfig | 4 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/dep/Makefile | 5 + drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++++ drivers/gpu/drm/dep/drm_dep_fence.h | 25 + drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++++++ drivers/gpu/drm/dep/drm_dep_job.h | 13 + drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++ drivers/gpu/drm/dep/drm_dep_queue.h | 31 + include/drm/drm_dep.h | 597 ++++++++++ 10 files changed, 3404 insertions(+)create mode 100644
drivers/gpu/drm/dep/Makefilecreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_job.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_job.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.hcreate mode 100644include/drm/drm_dep.hdrivers/gpu/drm/Kconfig +0
-0@@ -276,6 +276,10 @@ config DRM_SCHEDtristate
depends on DRMdrivers/gpu/drm/Kconfig +4
-0+config DRM_DEP + tristate + depends on DRM +# Separate option as not all DRM drivers use it
configDRM_PANEL_BACKLIGHT_QUIRKS
tristatedrivers/gpu/drm/Makefile +0
-0@@ -173,6 +173,7 @@ obj-y += clients/obj-y += display/
obj-$(CONFIG_DRM_TTM) += ttm/
obj-$(CONFIG_DRM_SCHED) += scheduler/
+obj-$(CONFIG_DRM_DEP) += dep/
obj-$(CONFIG_DRM_RADEON)+= radeon/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/drivers/gpu/drm/dep/Makefile new file +5
-0@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o + +obj-$(CONFIG_DRM_DEP) += drm_dep.odrivers/gpu/drm/dep/drm_dep_fence.c new file +6
-0@@ -0,0 +1,406 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +/**
drivers/gpu/drm/dep/drm_dep_fence.c line 16
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+* DOC: DRM dependency fence +* +* Each struct drm_dep_job has an associated struct drm_dep_fence that +* provides a single dma_fence (@finished) signalled when the hardware +* completes the job. +* +* The hardware fence returned by &drm_dep_queue_ops.run_job is stored as +* @parent. @finished is chained to @parent via drm_dep_job_done_cb() and +* is signalled once @parent signals (or immediately if run_job() returns +* NULL or an error).
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Quoted 136 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +136
-0+* +* Drivers should expose @finished as the out-fence for GPU work since it is +* valid from the moment drm_dep_job_arm() returns, whereas the hardware fence +* could be a compound fence, which is disallowed when installed into +* drm_syncobjs or dma-resv. +* +* The fence uses the kernel's inline spinlock (NULL passed to dma_fence_init()) +* so no separate lock allocation is required. +* +* Deadline propagation is supported: if a consumer sets a deadline via +* dma_fence_set_deadline(), it is forwarded to @parent when @parent is set. +* If @parent has not been set yet the deadline is stored in @deadline and +* forwarded at that point. +* +* Memory management: drm_dep_fence objects are allocated with kzalloc() and +* freed via kfree_rcu() once the fence is released, ensuring safety with +* RCU-protected fence accesses. +*/ + +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include "drm_dep_fence.h" + +/** +* DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has been set +* +* Set by the deadline callback on the finished fence to indicate a deadline +* has been set which may need to be propagated to the parent hardware fence. +*/ +#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT (DMA_FENCE_FLAG_USER_BITS + 1) + +/** +* struct drm_dep_fence - fence tracking the completion of a dep job +* +* Contains a single dma_fence (@finished) that is signalled when the +* hardware completes the job. The fence uses the kernel's inline_lock +* (no external spinlock required). +* +* This struct is private to the drm_dep module; external code interacts +* through the accessor functions declared in drm_dep_fence.h. +*/ +struct drm_dep_fence { +/** +* @finished: signalled when the job completes on hardware. +* +* Drivers should use this fence as the out-fence for a job since it +* is available immediately upon drm_dep_job_arm(). +*/ +struct dma_fence finished; + +/** +* @deadline: deadline set on @finished which potentially needs to be +* propagated to @parent. +*/ +ktime_t deadline; + +/** +* @parent: The hardware fence returned by &drm_dep_queue_ops.run_job. +* +* @finished is signaled once @parent is signaled. The initial store is +* performed via smp_store_release to synchronize with deadline handling. +* +* All readers must access this under the fence lock and take a reference to +* it, as @parent is set to NULL under the fence lock when the drm_dep_fence +* signals, and this drop also releases its internal reference. +*/ +struct dma_fence *parent; + +/** +* @q: the queue this fence belongs to. +*/ +struct drm_dep_queue *q; +}; + +static const struct dma_fence_ops drm_dep_fence_ops; + +/** +* to_drm_dep_fence() - cast a dma_fence to its enclosing drm_dep_fence +* @f: dma_fence to cast +* +* Context: No context requirements (inline helper). +* Return: pointer to the enclosing &drm_dep_fence. +*/ +static struct drm_dep_fence *to_drm_dep_fence (struct dma_fence *f) +{ + return container_of(f, struct drm_dep_fence, finished); +} + +/** +* drm_dep_fence_set_parent() - store the hardware fence and propagate +* any deadline +* @dfence: dep fence +* @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or NULL/error +* +* Stores @parent on @dfence under smp_store_release() so that a concurrent +* drm_dep_fence_set_deadline() call sees the parent before checking the +* deadline bit. If a deadline has already been set on @dfence->finished it is +* forwarded to @parent immediately. Does nothing if @parent is NULL or an +* error pointer. +* +* Context: Any context. +*/ +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent) +{ +if (IS_ERR_OR_NULL (parent)) +return ; + +/* +* smp_store_release() to ensure a thread racing us in +* drm_dep_fence_set_deadline() sees the parent set before +* it calls test_bit(HAS_DEADLINE_BIT). +*/ +smp_store_release (&dfence->parent,dma_fence_get (parent)); +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , + &dfence->finished.flags)) +dma_fence_set_deadline (parent, dfence->deadline); +} + +/** +* drm_dep_fence_finished() - signal the finished fence with a result +* @dfence: dep fence to signal +* @result: error code to set, or 0 for success +* +* Sets the fence error to @result if non-zero, then signals +* @dfence->finished. Also removes parent visibility under the fence lock +* and drops the parent reference. Dropping the parent here allows the +* DRM dep fence to be completely decoupled from the DRM dep module. +* +* Context: Any context. +*/ +static void drm_dep_fence_finished (struct drm_dep_fence *dfence,int result) +{ +struct dma_fence *parent; +unsigned long flags; +
drivers/gpu/drm/dep/drm_dep_fence.c line 162
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+dma_fence_lock_irqsave (&dfence->finished, flags); +if (result) +dma_fence_set_error (&dfence->finished, result); +dma_fence_signal_locked (&dfence->finished); + parent = dfence->parent; + dfence->parent =NULL ; +dma_fence_unlock_irqrestore (&dfence->finished, flags); + +dma_fence_put (parent); +}
We should really try to move away from manual locks and unlocks.
Quoted 236 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +234
-0+ +static const char *drm_dep_fence_get_driver_name (struct dma_fence *fence) +{ +return "drm_dep" ; +} + +static const char *drm_dep_fence_get_timeline_name (struct dma_fence *f) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); + +return dfence->q->name; +} + +/** +* drm_dep_fence_get_parent() - get a reference to the parent hardware fence +* @dfence: dep fence to query +* +* Returns a new reference to @dfence->parent, or NULL if the parent has +* already been cleared (i.e. @dfence->finished has signalled and the parent +* reference was dropped under the fence lock). +* +* Uses smp_load_acquire() to pair with the smp_store_release() in +* drm_dep_fence_set_parent(), ensuring that if we race a concurrent +* drm_dep_fence_set_parent() call we observe the parent pointer only after +* the store is fully visible — before set_parent() tests +* %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT. +* +* Caller must hold the fence lock on @dfence->finished. +* +* Context: Any context, fence lock on @dfence->finished must be held. +* Return: a new reference to the parent fence, or NULL. +*/ +static struct dma_fence *drm_dep_fence_get_parent (struct drm_dep_fence *dfence) +{ +dma_fence_assert_held (&dfence->finished); + +return dma_fence_get (smp_load_acquire (&dfence->parent)); +} + +/** +* drm_dep_fence_set_deadline() - dma_fence_ops deadline callback +* @f: fence on which the deadline is being set +* @deadline: the deadline hint to apply +* +* Stores the earliest deadline under the fence lock, then propagates +* it to the parent hardware fence via smp_load_acquire() to race +* safely with drm_dep_fence_set_parent(). +* +* Context: Any context. +*/ +static void drm_dep_fence_set_deadline (struct dma_fence *f,ktime_t deadline) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); +struct dma_fence *parent; +unsigned long flags; + +dma_fence_lock_irqsave (f, flags); + +/* If we already have an earlier deadline, keep it: */ +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags) && +ktime_before (dfence->deadline, deadline)) { +dma_fence_unlock_irqrestore (f, flags); +return ; + } + + dfence->deadline = deadline; +set_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags); + + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (f, flags); + +if (parent) +dma_fence_set_deadline (parent, deadline); + +dma_fence_put (parent); +} + +static const struct dma_fence_ops drm_dep_fence_ops = { + .get_driver_name = drm_dep_fence_get_driver_name, + .get_timeline_name = drm_dep_fence_get_timeline_name, + .set_deadline = drm_dep_fence_set_deadline, +}; + +/** +* drm_dep_fence_alloc() - allocate a dep fence +* +* Allocates a &drm_dep_fence with kzalloc() without initialising the +* dma_fence. Call drm_dep_fence_init() to fully initialise it. +* +* Context: Process context. +* Return: new &drm_dep_fence on success, NULL on allocation failure. +*/ +struct drm_dep_fence *drm_dep_fence_alloc (void ) +{ + return kzalloc_obj(struct drm_dep_fence); +} + +/** +* drm_dep_fence_init() - initialise the dma_fence inside a dep fence +* @dfence: dep fence to initialise +* @q: queue the owning job belongs to +* +* Initialises @dfence->finished using the context and sequence number from @q. +* Passes NULL as the lock so the fence uses its inline spinlock. +* +* Context: Any context. +*/ +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q) +{ +u32 seq = ++q->fence.seqno; + +/* +* XXX: Inline fence hazard: currently all expected users of DRM dep +* hardware fences have a unique lockdep class. If that ever changes, +* we will need to assign a unique lockdep class here so lockdep knows +* this fence is allowed to nest with driver hardware fences. +*/ + + dfence->q = q; +dma_fence_init (&dfence->finished, &drm_dep_fence_ops, +NULL , q->fence.context, seq); +} + +/** +* drm_dep_fence_cleanup() - release a dep fence at job teardown +* @dfence: dep fence to clean up +* +* Called from drm_dep_job_fini(). If the dep fence was armed (refcount > 0) +* it is released via dma_fence_put() and will be freed by the RCU release +* callback once all waiters have dropped their references. If it was never +* armed it is freed directly with kfree(). +* +* Context: Any context. +*/ +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence) +{ +if (drm_dep_fence_is_armed (dfence)) +dma_fence_put (&dfence->finished); +else +kfree (dfence); +} + +/** +* drm_dep_fence_is_armed() - check whether the fence has been armed +* @dfence: dep fence to check +* +* Returns true if drm_dep_job_arm() has been called, i.e. @dfence->finished +* has been initialised and its reference count is non-zero. Used by +* assertions to enforce correct job lifecycle ordering (arm before push, +* add_dependency before arm). +* +* Context: Any context. +* Return: true if the fence is armed, false otherwise. +*/ +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence) +{ +return !!kref_read (&dfence->finished.refcount); +} + +/** +* drm_dep_fence_is_finished() - test whether the finished fence has signalled +* @dfence: dep fence to check +* +* Uses dma_fence_test_signaled_flag() to read %DMA_FENCE_FLAG_SIGNALED_BIT +* directly without invoking the fence's ->signaled() callback or triggering +* any signalling side-effects. +* +* Context: Any context. +* Return: true if @dfence->finished has been signalled, false otherwise. +*/ +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence) +{ +return dma_fence_test_signaled_flag (&dfence->finished); +} + +/** +* drm_dep_fence_is_complete() - test whether the job has completed +* @dfence: dep fence to check +* +* Takes the fence lock on @dfence->finished and calls +* drm_dep_fence_get_parent() to safely obtain a reference to the parent +* hardware fence — or NULL if the parent has already been cleared after +* signalling. Calls dma_fence_is_signaled() on @parent outside the lock, +* which may invoke the fence's ->signaled() callback and trigger signalling +* side-effects if the fence has completed but the signalled flag has not yet +* been set. The finished fence is tested via dma_fence_test_signaled_flag(), +* without side-effects. +* +* May only be called on a stopped queue (see drm_dep_queue_is_stopped()). +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is complete, false otherwise. +*/ +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence) +{ +struct dma_fence *parent; +unsigned long flags; +bool complete; + +dma_fence_lock_irqsave (&dfence->finished, flags); + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (&dfence->finished, flags); + + complete = (parent &&dma_fence_is_signaled (parent)) || +dma_fence_test_signaled_flag (&dfence->finished); + +dma_fence_put (parent); + +return complete; +} + +/** +* drm_dep_fence_to_dma() - return the finished dma_fence for a dep fence +* @dfence: dep fence to query +* +* No reference is taken; the caller must hold its own reference to the owning +* &drm_dep_job for the duration of the access. +* +* Context: Any context. +* Return: the finished &dma_fence. +*/ +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence) +{ +return &dfence->finished; +} + +/** +* drm_dep_fence_done() - signal the finished fence on job completion +* @dfence: dep fence to signal +* @result: job error code, or 0 on success +* +* Gets a temporary reference to @dfence->finished to guard against a racing +* last-put, signals the fence with @result, then drops the temporary +* reference. Called from drm_dep_job_done() in the queue core when a
drivers/gpu/drm/dep/drm_dep_fence.c line 406
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+* hardware completion callback fires or when run_job() returns immediately. +* +* Context: Any context. +*/ +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result) +{ +dma_fence_get (&dfence->finished); +drm_dep_fence_finished (dfence, result); +dma_fence_put (&dfence->finished); +}
Proper refcounting is automated (and enforced) in Rust.
Quoted 111 lines
drivers/gpu/drm/dep/drm_dep_fence.h new file +25
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_FENCE_H_ +#define _DRM_DEP_FENCE_H_ + +#include <linux/dma-fence.h> + +struct drm_dep_fence ; +struct drm_dep_queue ; + +struct drm_dep_fence *drm_dep_fence_alloc (void ); +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q); +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence); +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent); +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result); +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence); +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence); + +#endif /* _DRM_DEP_FENCE_H_ */ drivers/gpu/drm/dep/drm_dep_job.c new file +74
-0@@ -0,0 +1,675 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency job +* +* A struct drm_dep_job represents a single unit of GPU work associated with +* a struct drm_dep_queue. The lifecycle of a job is: +* +* 1. **Allocation**: the driver allocates memory for the job (typically by +* embedding struct drm_dep_job in a larger structure) and calls +* drm_dep_job_init() to initialise it. On success the job holds one +* kref reference and a reference to its queue. +* +* 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(), +* drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(), +* or drm_dep_job_add_implicit_dependencies() to register dma_fence objects +* that must be signalled before the job can run. Duplicate fences from the +* same fence context are deduplicated automatically. +* +* 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence, +* consuming a sequence number from the queue. After arming, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* 4. **Submission**: drm_dep_job_push() submits the job to the queue. The +* queue takes a reference that it holds until the job's finished fence +* signals and the job is freed by the put_job worker. +* +* 5. **Completion**: when the job's hardware work finishes its finished fence +* is signalled and drm_dep_job_put() is called by the queue. The driver +* must release any driver-private resources in &drm_dep_job_ops.release. +* +* Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The +* internal drm_dep_job_fini() tears down the dependency xarray and fence +* objects before the driver's release callback is invoked. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include <drm/drm_file.h> +#include <drm/drm_gem.h> +#include <drm/drm_syncobj.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/** +* drm_dep_job_init() - initialise a dep job +* @job: dep job to initialise
drivers/gpu/drm/dep/drm_dep_job.c line 84
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* @args: initialisation arguments +* +* Initialises @job with the queue, ops and credit count from @args. Acquires +* a reference to @args->q via drm_dep_queue_get(); this reference is held for +* the lifetime of the job and released by drm_dep_job_release() when the last +* job reference is dropped. +* +* Resources are released automatically when the last reference is dropped +* via drm_dep_job_put(), which must be called to release the job; drivers +* must not free the job directly.
Again, can’t enforce that in C.
Quoted 49 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +49
-0+* +* Context: Process context. Allocates memory with GFP_KERNEL. +* Return: 0 on success, -%EINVAL if credits is 0, +* -%ENOMEM on fence allocation failure. +*/ +int drm_dep_job_init (struct drm_dep_job *job, +const struct drm_dep_job_init_args *args) +{ +if (unlikely (!args->credits)) { +pr_err ("drm_dep: %s: credits cannot be 0\n" , __func__); +return -EINVAL ; + } + +memset (job,0 ,sizeof (*job)); + + job->dfence =drm_dep_fence_alloc (); +if (!job->dfence) +return -ENOMEM ; + + job->ops = args->ops; + job->q =drm_dep_queue_get (args->q); + job->credits = args->credits; + +kref_init (&job->refcount); +xa_init_flags (&job->dependencies,XA_FLAGS_ALLOC ); +INIT_LIST_HEAD (&job->pending_link); + +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_init); + +/** +* drm_dep_job_drop_dependencies() - release all input dependency fences +* @job: dep job whose dependency xarray to drain +* +* Walks @job->dependencies, puts each fence, and destroys the xarray. +* Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel — +* i.e. slots that were pre-allocated but never replaced — are silently +* skipped; the sentinel carries no reference. Called from +* drm_dep_queue_run_job() in process context immediately after +* @ops->run_job() returns, before the final drm_dep_job_put(). Releasing +* dependencies here — while still in process context — avoids calling +* xa_destroy() from IRQ context if the job's last reference is later +* dropped from a dma_fence callback. +* +* Context: Process context. +*/ +void drm_dep_job_drop_dependencies (struct drm_dep_job *job) +{
drivers/gpu/drm/dep/drm_dep_job.c line 143
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+struct dma_fence *fence; +unsigned long index; + +xa_for_each (&job->dependencies, index, fence) { +if (unlikely (fence ==DRM_DEP_JOB_FENCE_PREALLOC )) +continue ; +dma_fence_put (fence); + } +xa_destroy (&job->dependencies); +}
This is automated in Rust. You also can’t “forget” to call this.
Quoted 25 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +25
-0+ +/** +* drm_dep_job_fini() - clean up a dep job +* @job: dep job to clean up +* +* Cleans up the dep fence and drops the queue reference held by @job. +* +* If the job was never armed (e.g. init failed before drm_dep_job_arm()), +* the dependency xarray is also released here. For armed jobs the xarray +* has already been drained by drm_dep_job_drop_dependencies() in process +* context immediately after run_job(), so it is left untouched to avoid +* calling xa_destroy() from IRQ context. +* +* Warns if @job is still linked on the queue's pending list, which would +* indicate a bug in the teardown ordering. +* +* Context: Any context. +*/ +static void drm_dep_job_fini (struct drm_dep_job *job) +{ +bool armed =drm_dep_fence_is_armed (job->dfence); + +WARN_ON (!list_empty (&job->pending_link)); + +drm_dep_fence_cleanup (job->dfence);
drivers/gpu/drm/dep/drm_dep_job.c line 178
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ job->dfence =NULL ; + +/* +* Armed jobs have their dependencies drained by +* drm_dep_job_drop_dependencies() in process context after run_job(). +* Skip here to avoid calling xa_destroy() from IRQ context. +*/ +if (!armed) +drm_dep_job_drop_dependencies (job); +}
Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +6
-0+ +/** +* drm_dep_job_get() - acquire a reference to a dep job +* @job: dep job to acquire a reference on, or NULL +* +* Context: Any context.
drivers/gpu/drm/dep/drm_dep_job.c line 194
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* Return: @job with an additional reference held, or NULL if @job is NULL. +*/ +struct drm_dep_job *drm_dep_job_get (struct drm_dep_job *job) +{ +if (job) +kref_get (&job->refcount); +return job; +} +EXPORT_SYMBOL (drm_dep_job_get); +
Same here.
Quoted 19 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +19
-0+/** +* drm_dep_job_release() - kref release callback for a dep job +* @kref: kref embedded in the dep job +* +* Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Finally, releases the queue reference +* that was acquired by drm_dep_job_init() via drm_dep_queue_put(). The +* queue put is performed last to ensure no queue state is accessed after +* the job memory is freed. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the +* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +static void drm_dep_job_release (struct kref *kref) +{ +struct drm_dep_job *job = + container_of(kref, struct drm_dep_job, refcount); +struct drm_dep_queue *q = job->q;
drivers/gpu/drm/dep/drm_dep_job.c line 223
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ +drm_dep_job_fini (job); + +if (job->ops && job->ops->release) + job->ops->release (job); +else +kfree (job); + +drm_dep_queue_put (q); +}
Same here.
Quoted 9 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +9
-0+ +/** +* drm_dep_job_put() - release a reference to a dep job +* @job: dep job to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Does nothing if @job is NULL. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
drivers/gpu/drm/dep/drm_dep_job.c line 242
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +void drm_dep_job_put (struct drm_dep_job *job) +{ +if (job) +kref_put (&job->refcount, drm_dep_job_release); +} +EXPORT_SYMBOL (drm_dep_job_put); +
Same here.
Quoted 65 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +65
-0+/** +* drm_dep_job_arm() - arm a dep job for submission +* @job: dep job to arm +* +* Initialises the finished fence on @job->dfence, assigning +* it a sequence number from the job's queue. Must be called after +* drm_dep_job_init() and before drm_dep_job_push(). Once armed, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* Begins the DMA fence signalling path via dma_fence_begin_signalling(). +* After this point, memory allocations that could trigger reclaim are +* forbidden; lockdep enforces this. arm() must always be paired with +* drm_dep_job_push(); lockdep also enforces this pairing. +* +* Warns if the job has already been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_arm (struct drm_dep_job *job) +{ +drm_dep_queue_push_job_begin (job->q); +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +drm_dep_fence_init (job->dfence, job->q); + job->signalling_cookie =dma_fence_begin_signalling (); +} +EXPORT_SYMBOL (drm_dep_job_arm); + +/** +* drm_dep_job_push() - submit a job to its queue for execution +* @job: dep job to push +* +* Submits @job to the queue it was initialised with. Must be called after +* drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue, +* held until the queue is fully done with it. The reference is released +* directly in the finished-fence dma_fence callback for queues with +* %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run +* from hardirq context), or via the put_job work item on the submit +* workqueue otherwise. +* +* Ends the DMA fence signalling path begun by drm_dep_job_arm() via +* dma_fence_end_signalling(). This must be paired with arm(); lockdep +* enforces the pairing. +* +* Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for +* @job exactly once, even if the queue is killed or torn down before the +* job reaches the head of the queue. Drivers can use this guarantee to +* perform bookkeeping cleanup; the actual backend operation should be +* skipped when drm_dep_queue_is_killed() returns true. +* +* If the queue does not support the bypass path, the job is pushed directly +* onto the SPSC submission queue via drm_dep_queue_push_job() without holding +* @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either +* run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or +* enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item. +* +* Warns if the job has not been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_push (struct drm_dep_job *job)
drivers/gpu/drm/dep/drm_dep_job.c line 317
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+{ +struct drm_dep_queue *q = job->q; + +WARN_ON (!drm_dep_fence_is_armed (job->dfence)); + +drm_dep_job_get (job); + +if (!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED )) { +drm_dep_queue_push_job (q, job); +dma_fence_end_signalling (job->signalling_cookie);
Signaling is enforced in a more thorough way in Rust. I’ll expand on this later in this patch.
Quoted 167 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +166
-0+drm_dep_queue_push_job_end (job->q); +return ; + } + + scoped_guard(mutex, &q->sched.lock) { +if (drm_dep_queue_can_job_bypass (q, job)) +drm_dep_queue_run_job (q, job); +else +drm_dep_queue_push_job (q, job); + } + +dma_fence_end_signalling (job->signalling_cookie); +drm_dep_queue_push_job_end (job->q); +} +EXPORT_SYMBOL (drm_dep_job_push); + +/** +* drm_dep_job_add_dependency() - adds the fence as a job dependency +* @job: dep job to add the dependencies to +* @fence: the dma_fence to add to the list of dependencies, or +* %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later. +* +* Note that @fence is consumed in both the success and error cases (except +* when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference). +* +* Signalled fences and fences belonging to the same queue as @job (i.e. where +* fence->context matches the queue's finished fence context) are silently +* dropped; the job need not wait on its own queue's output. +* +* Warns if the job has already been armed (dependencies must be added before +* drm_dep_job_arm()). +* +* **Pre-allocation pattern** +* +* When multiple jobs across different queues must be prepared and submitted +* together in a single atomic commit — for example, where job A's finished +* fence is an input dependency of job B — all jobs must be armed and pushed +* within a single dma_fence_begin_signalling() / dma_fence_end_signalling() +* region. Once that region has started no memory allocation is permitted. +* +* To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation +* phase (before arming any job, while GFP_KERNEL allocation is still allowed) +* to pre-allocate a slot in @job->dependencies. The slot index assigned by +* the underlying xarray must be tracked by the caller separately (e.g. it is +* always index 0 when the dependency array is empty, as Xe relies on). +* After all jobs have been armed and the finished fences are available, call +* drm_dep_job_replace_dependency() with that index and the real fence. +* drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be +* called from atomic or signalling context. +* +* The sentinel slot is never skipped by the signalled-fence fast-path, +* ensuring a slot is always allocated even when the real fence is not yet +* known. +* +* **Example: bind job feeding TLB invalidation jobs** +* +* Consider a GPU with separate queues for page-table bind operations and for +* TLB invalidation. A single atomic commit must: +* +* 1. Run a bind job that modifies page tables. +* 2. Run one TLB-invalidation job per MMU that depends on the bind +* completing, so stale translations are flushed before the engines +* continue. +* +* Because all jobs must be armed and pushed inside a signalling region (where +* GFP_KERNEL is forbidden), pre-allocate slots before entering the region:: +* +* // Phase 1 — process context, GFP_KERNEL allowed +* drm_dep_job_init(bind_job, bind_queue, ops); +* for_each_mmu(mmu) { +* drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops); +* // Pre-allocate slot at index 0; real fence not available yet +* drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC); +* } +* +* // Phase 2 — inside signalling region, no GFP_KERNEL +* dma_fence_begin_signalling(); +* drm_dep_job_arm(bind_job); +* for_each_mmu(mmu) { +* // Swap sentinel for bind job's finished fence +* drm_dep_job_replace_dependency(tlb_job[mmu], 0, +* dma_fence_get(bind_job->finished)); +* drm_dep_job_arm(tlb_job[mmu]); +* } +* drm_dep_job_push(bind_job); +* for_each_mmu(mmu) +* drm_dep_job_push(tlb_job[mmu]); +* dma_fence_end_signalling(); +* +* Context: Process context. May allocate memory with GFP_KERNEL. +* Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on +* success, else 0 on success, or a negative error code. +*/ +int drm_dep_job_add_dependency (struct drm_dep_job *job,struct dma_fence *fence) +{ +struct drm_dep_queue *q = job->q; +struct dma_fence *entry; +unsigned long index; +u32 id =0 ; +int ret; + +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +might_alloc (GFP_KERNEL ); + +if (!fence) +return 0 ; + +if (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) + goto add_fence; + +/* +* Ignore signalled fences or fences from our own queue — finished +* fences use q->fence.context. +*/ +if (dma_fence_test_signaled_flag (fence) || + fence->context == q->fence.context) { +dma_fence_put (fence); +return 0 ; + } + +/* Deduplicate if we already depend on a fence from the same context. +* This lets the size of the array of deps scale with the number of +* engines involved, rather than the number of BOs. +*/ +xa_for_each (&job->dependencies, index, entry) { +if (entry ==DRM_DEP_JOB_FENCE_PREALLOC || + entry->context != fence->context) +continue ; + +if (dma_fence_is_later (fence, entry)) { +dma_fence_put (entry); +xa_store (&job->dependencies, index, fence,GFP_KERNEL ); + }else { +dma_fence_put (fence); + } +return 0 ; + } + +add_fence: + ret =xa_alloc (&job->dependencies, &id, fence, xa_limit_32b, +GFP_KERNEL ); +if (ret !=0 ) { +if (fence !=DRM_DEP_JOB_FENCE_PREALLOC ) +dma_fence_put (fence); +return ret; + } + +return (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) ? id :0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_dependency); + +/** +* drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot +* @job: dep job to update +* @index: xarray index of the slot to replace, as returned when the sentinel +* was originally inserted via drm_dep_job_add_dependency() +* @fence: the real dma_fence to store; its reference is always consumed +* +* Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in +* @job->dependencies with @fence. The slot must have been pre-allocated by +* passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the +* existing entry is asserted to be the sentinel. +* +* This is the second half of the pre-allocation pattern described in +* drm_dep_job_add_dependency(). It is intended to be called inside a +* dma_fence_begin_signalling() / dma_fence_end_signalling() region where
drivers/gpu/drm/dep/drm_dep_job.c line 493
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* memory allocation with GFP_KERNEL is forbidden. It uses GFP_NOWAIT +* internally so it is safe to call from atomic or signalling context, but +* since the slot has been pre-allocated no actual memory allocation occurs. +* +* If @fence is already signalled the slot is erased rather than storing a +* redundant dependency. The successful store is asserted — if the store +* fails it indicates a programming error (slot index out of range or +* concurrent modification). +* +* Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
Can’t enforce this in C. Also, how is the fence “consumed” ? You can’t enforce that the user can’t access the fence anymore after this function returns, like we can do at compile time in Rust.
Quoted 163 lines
+ *
+ * Context: Any context. DMA fence signaling path.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +160
-0+void drm_dep_job_replace_dependency (struct drm_dep_job *job,u32 index, +struct dma_fence *fence) +{ +WARN_ON (xa_load (&job->dependencies, index) != +DRM_DEP_JOB_FENCE_PREALLOC ); + +if (dma_fence_test_signaled_flag (fence)) { +xa_erase (&job->dependencies, index); +dma_fence_put (fence); +return ; + } + +if (WARN_ON (xa_is_err (xa_store (&job->dependencies, index, fence, +GFP_NOWAIT )))) { +dma_fence_put (fence); +return ; + } +} +EXPORT_SYMBOL (drm_dep_job_replace_dependency); + +/** +* drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a +* job dependency +* @job: dep job to add the dependencies to +* @file: drm file private pointer +* @handle: syncobj handle to lookup +* @point: timeline point +* +* This adds the fence matching the given syncobj to @job. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_syncobj_dependency (struct drm_dep_job *job, +struct drm_file *file,u32 handle, +u32 point) +{ +struct dma_fence *fence; +int ret; + + ret =drm_syncobj_find_fence (file, handle, point,0 , &fence); +if (ret) +return ret; + +return drm_dep_job_add_dependency (job, fence); +} +EXPORT_SYMBOL (drm_dep_job_add_syncobj_dependency); + +/** +* drm_dep_job_add_resv_dependencies() - add all fences from the resv to the job +* @job: dep job to add the dependencies to +* @resv: the dma_resv object to get the fences from +* @usage: the dma_resv_usage to use to filter the fences +* +* This adds all fences matching the given usage from @resv to @job. +* Must be called with the @resv lock held. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_resv_dependencies (struct drm_dep_job *job, +struct dma_resv *resv, +enum dma_resv_usage usage) +{ +struct dma_resv_iter cursor; +struct dma_fence *fence; +int ret; + +dma_resv_assert_held (resv); + +dma_resv_for_each_fence (&cursor, resv, usage, fence) { +/* +* As drm_dep_job_add_dependency always consumes the fence +* reference (even when it fails), and dma_resv_for_each_fence +* is not obtaining one, we need to grab one before calling. +*/ + ret =drm_dep_job_add_dependency (job,dma_fence_get (fence)); +if (ret) +return ret; + } +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_resv_dependencies); + +/** +* drm_dep_job_add_implicit_dependencies() - adds implicit dependencies +* as job dependencies +* @job: dep job to add the dependencies to +* @obj: the gem object to add new dependencies from. +* @write: whether the job might write the object (so we need to depend on +* shared fences in the reservation object). +* +* This should be called after drm_gem_lock_reservations() on your array of +* GEM objects used in the job but before updating the reservations with your +* own fences. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_implicit_dependencies (struct drm_dep_job *job, +struct drm_gem_object *obj, +bool write) +{ +return drm_dep_job_add_resv_dependencies (job, obj->resv, +dma_resv_usage_rw (write)); +} +EXPORT_SYMBOL (drm_dep_job_add_implicit_dependencies); + +/** +* drm_dep_job_is_signaled() - check whether a dep job has completed +* @job: dep job to check +* +* Determines whether @job has signalled. The queue should be stopped before +* calling this to obtain a stable snapshot of state. Both the parent hardware +* fence and the finished software fence are checked. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is signalled, false otherwise. +*/ +bool drm_dep_job_is_signaled (struct drm_dep_job *job) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q)); +return drm_dep_fence_is_complete (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_signaled); + +/** +* drm_dep_job_is_finished() - test whether a dep job's finished fence has signalled +* @job: dep job to check +* +* Tests whether the job's software finished fence has been signalled, using +* dma_fence_test_signaled_flag() to avoid any signalling side-effects. Unlike +* drm_dep_job_is_signaled(), this does not require the queue to be stopped and +* does not check the parent hardware fence — it is a lightweight test of the +* finished fence only. +* +* Context: Any context. +* Return: true if the job's finished fence has been signalled, false otherwise. +*/ +bool drm_dep_job_is_finished (struct drm_dep_job *job) +{ +return drm_dep_fence_is_finished (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_finished); + +/** +* drm_dep_job_invalidate_job() - increment the invalidation count for a job +* @job: dep job to invalidate +* @threshold: threshold above which the job is considered invalidated +* +* Increments @job->invalidate_count and returns true if it exceeds @threshold, +* indicating the job should be considered hung and discarded. The queue must +* be stopped before calling this function. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if @job->invalidate_count exceeds @threshold, false otherwise. +*/ +bool drm_dep_job_invalidate_job (struct drm_dep_job *job,int threshold) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q));
drivers/gpu/drm/dep/drm_dep_job.c line 666
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+return ++job->invalidate_count > threshold; +} +EXPORT_SYMBOL (drm_dep_job_invalidate_job); + +/** +* drm_dep_job_finished_fence() - return the finished fence for a job +* @job: dep job to query +* +* No reference is taken on the returned fence; the caller must hold its own +* reference to @job for the duration of any access.
Can’t enforce this in C.
Quoted 111 lines
+ *
+ * Context: Any context.
+ * Return: the finished&dma_fencefor @job.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +5
-0+struct dma_fence *drm_dep_job_finished_fence (struct drm_dep_job *job) +{ +return drm_dep_fence_to_dma (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_finished_fence);drivers/gpu/drm/dep/drm_dep_job.h new file +13
-0@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_JOB_H_ +#define _DRM_DEP_JOB_H_ + +struct drm_dep_queue ; + +void drm_dep_job_drop_dependencies (struct drm_dep_job *job); + +#endif /* _DRM_DEP_JOB_H_ */ drivers/gpu/drm/dep/drm_dep_queue.c new file +77
-0@@ -0,0 +1,1647 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency queue +* +* The drm_dep subsystem provides a lightweight GPU submission queue that +* combines the roles of drm_gpu_scheduler and drm_sched_entity into a +* single object (struct drm_dep_queue). Each queue owns its own ordered +* submit workqueue, timeout workqueue, and TDR delayed-work. +* +* **Job lifecycle** +* +* 1. Allocate and initialise a job with drm_dep_job_init(). +* 2. Add dependency fences with drm_dep_job_add_dependency() and friends. +* 3. Arm the job with drm_dep_job_arm() to obtain its out-fences. +* 4. Submit with drm_dep_job_push(). +* +* **Submission paths** +* +* drm_dep_job_push() decides between two paths under @q->sched.lock: +* +* - **Bypass path** (drm_dep_queue_can_job_bypass()): if +* %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not stopped, +* the SPSC queue is empty, the job has no dependency fences, and credits +* are available, the job is submitted inline on the calling thread without +* touching the submit workqueue. +* +* - **Queued path** (drm_dep_queue_push_job()): the job is pushed onto an +* SPSC queue and the run_job worker is kicked. The run_job worker pops the +* job, resolves any remaining dependency fences (installing wakeup +* callbacks for unresolved ones), and calls drm_dep_queue_run_job(). +* +* **Running a job** +* +* drm_dep_queue_run_job() accounts credits, appends the job to the pending +* list (starting the TDR timer only when the list was previously empty), +* calls @ops->run_job(), stores the returned hardware fence as the parent +* of the job's dep fence, then installs a callback on it. When the hardware +* fence fires (or the job completes synchronously), drm_dep_job_done() +* signals the finished fence, returns credits, and kicks the put_job worker +* to free the job. +* +* **Timeout detection and recovery (TDR)** +* +* A delayed work item fires when a job on the pending list takes longer than +* @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on the +* returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB). +* drm_dep_queue_trigger_timeout() forces the timer to fire immediately (without +* changing the stored timeout), for example during device teardown. +* +* **Reference counting** +* +* Jobs and queues are both reference counted.
drivers/gpu/drm/dep/drm_dep_queue.c line 87
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* +* A job holds a reference to its queue from drm_dep_job_init() until +* drm_dep_job_put() drops the job's last reference and its release callback +* runs. This ensures the queue remains valid for the entire lifetime of any +* job that was submitted to it. +* +* The queue holds its own reference to a job for as long as the job is +* internally tracked: from the moment the job is added to the pending list +* in drm_dep_queue_run_job() until drm_dep_job_done() kicks the put_job +* worker, which calls drm_dep_job_put() to release that reference.
Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
Quoted 7 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +7
-0+* +* **Hazard: use-after-free from within a worker** +* +* Because a job holds a queue reference, drm_dep_job_put() dropping the last +* job reference will also drop a queue reference via the job's release path. +* If that happens to be the last queue reference, drm_dep_queue_fini() can be +* called, which queues @q->free_work on dep_free_wq and returns immediately.
drivers/gpu/drm/dep/drm_dep_queue.c line 104
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* free_work calls disable_work_sync() / disable_delayed_work_sync() on the +* queue's own workers before destroying its workqueues, so in practice a +* running worker always completes before the queue memory is freed. +* +* However, there is a secondary hazard: a worker can be queued while the +* queue is in a "zombie" state — refcount has already reached zero and async +* teardown is in flight, but the work item has not yet been disabled by +* free_work. To guard against this every worker uses +* drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero +* the worker bails immediately without touching the queue state.
Again, this problem is gone in Rust.
Quoted 34 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +34
-0+* +* Because all actual teardown (disable_*_sync, destroy_workqueue) runs on +* dep_free_wq — which is independent of the queue's own submit/timeout +* workqueues — there is no deadlock risk. Each queue holds a drm_dev_get() +* reference on its owning &drm_device, which is released as the last step of +* teardown. This ensures the driver module cannot be unloaded while any queue +* is still alive. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/module.h> +#include <linux/overflow.h> +#include <linux/slab.h> +#include <linux/wait.h> +#include <linux/workqueue.h> +#include <drm/drm_dep.h> +#include <drm/drm_drv.h> +#include <drm/drm_print.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/* +* Dedicated workqueue for deferred drm_dep_queue teardown. Using a +* module-private WQ instead of system_percpu_wq keeps teardown isolated +* from unrelated kernel subsystems. +*/ +static struct workqueue_struct *dep_free_wq; + +/** +* drm_dep_queue_flags_set() - set a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to set (one of &enum drm_dep_queue_flags)
drivers/gpu/drm/dep/drm_dep_queue.c line 148
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* +* Sets @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_set (struct drm_dep_queue *q, +enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock);
We can enforce this in Rust at compile-time. The code does not compile if the lock is not taken. Same here and everywhere else where the sched lock has to be taken.
Quoted 124 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +124
-0+ q->sched.flags |= flag; +} + +/** +* drm_dep_queue_flags_clear() - clear a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to clear (one of &enum drm_dep_queue_flags) +* +* Clears @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_clear (struct drm_dep_queue *q, +enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock); + q->sched.flags &= ~flag; +} + +/** +* drm_dep_queue_has_credits() - check whether the queue has enough credits +* @q: dep queue +* @job: job requesting credits +* +* Checks whether the queue has enough available credits to dispatch +* @job. If @job->credits exceeds the queue's credit limit, it is +* clamped with a WARN. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if available credits >= @job->credits, false otherwise. +*/ +static bool drm_dep_queue_has_credits (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +u32 available; + +lockdep_assert_held (&q->sched.lock); + +if (job->credits > q->credit.limit) { +drm_warn (q->drm, +"Jobs may not exceed the credit limit, truncate.\n" ); + job->credits = q->credit.limit; + } + +WARN_ON (check_sub_overflow (q->credit.limit, +atomic_read (&q->credit.count), + &available)); + +return available >= job->credits; +} + +/** +* drm_dep_queue_run_job_queue() - kick the run-job worker +* @q: dep queue +* +* Queues @q->sched.run_job on @q->sched.submit_wq unless the queue is stopped +* or the job queue is empty. The empty-queue check avoids queueing a work item +* that would immediately return with nothing to do. +* +* Context: Any context. +*/ +static void drm_dep_queue_run_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q) &&spsc_queue_count (&q->job.queue)) +queue_work (q->sched.submit_wq, &q->sched.run_job); +} + +/** +* drm_dep_queue_put_job_queue() - kick the put-job worker +* @q: dep queue +* +* Queues @q->sched.put_job on @q->sched.submit_wq unless the queue +* is stopped. +* +* Context: Any context. +*/ +static void drm_dep_queue_put_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q)) +queue_work (q->sched.submit_wq, &q->sched.put_job); +} + +/** +* drm_queue_start_timeout() - arm or re-arm the TDR delayed work +* @q: dep queue +* +* Arms the TDR delayed work with @q->job.timeout. No-op if +* @q->ops->timedout_job is NULL, the timeout is MAX_SCHEDULE_TIMEOUT, +* or the pending list is empty. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_queue_start_timeout (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->job.lock); + +if (!q->ops->timedout_job || + q->job.timeout ==MAX_SCHEDULE_TIMEOUT || +list_empty (&q->job.pending)) +return ; + +mod_delayed_work (q->sched.timeout_wq, &q->sched.tdr, q->job.timeout); +} + +/** +* drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock +* @q: dep queue +* +* Acquires @q->job.lock with irq and calls +* drm_queue_start_timeout(). +* +* Context: Process context (workqueue). +*/ +static void drm_queue_start_timeout_unlocked (struct drm_dep_queue *q) +{ +guard (spinlock_irq)(&q->job.lock); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_remove_dependency() - clear the active dependency and wake +* the run-job worker +* @q: dep queue
drivers/gpu/drm/dep/drm_dep_queue.c line 282
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @f: the dependency fence being removed +* +* Stores @f into @q->dep.removed_fence via smp_store_release() so that the +* run-job worker can drop the reference to it in drm_dep_queue_is_ready(), +* paired with smp_load_acquire(). Clears @q->dep.fence and kicks the +* run-job worker. +* +* The fence reference is not dropped here; it is deferred to the run-job +* worker via @q->dep.removed_fence to keep this path suitable dma_fence +* callback removal in drm_dep_queue_kill().
This is a comment in C, but in Rust this is encoded directly in the type system.
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_remove_dependency (struct drm_dep_queue *q, +struct dma_fence *f) +{ +/* removed_fence must be visible to the reader before &q->dep.fence */ +smp_store_release (&q->dep.removed_fence, f); + +WRITE_ONCE (q->dep.fence,NULL ); +drm_dep_queue_run_job_queue (q); +}
drivers/gpu/drm/dep/drm_dep_queue.c line 304
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +/** +* drm_dep_queue_wakeup() - dma_fence callback to wake the run-job worker +* @f: the signalled dependency fence +* @cb: callback embedded in the dep queue +* +* Called from dma_fence_signal() when the active dependency fence signals. +* Delegates to drm_dep_queue_remove_dependency() to clear @q->dep.fence and +* kick the run-job worker. The fence reference is not dropped here; it is +* deferred to the run-job worker via @q->dep.removed_fence.
Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +6
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_wakeup (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_queue *q =
drivers/gpu/drm/dep/drm_dep_queue.c line 320
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ container_of(cb, struct drm_dep_queue, dep.cb); + +drm_dep_queue_remove_dependency (q, f); +} + +/** +* drm_dep_queue_is_ready() - check whether the queue has a dispatchable job +* @q: dep queue +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
Can’t call this in Rust if the lock is not taken.
Quoted 158 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +158
-0+* Return: true if SPSC queue non-empty and no dep fence pending, +* false otherwise. +*/ +static bool drm_dep_queue_is_ready (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->sched.lock); + +if (!spsc_queue_count (&q->job.queue)) +return false; + +if (READ_ONCE (q->dep.fence)) +return false; + +/* Paired with smp_store_release in drm_dep_queue_remove_dependency() */ +dma_fence_put (smp_load_acquire (&q->dep.removed_fence)); + + q->dep.removed_fence =NULL ; + +return true; +} + +/** +* drm_dep_queue_is_killed() - check whether a dep queue has been killed +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_killed (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_KILLED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_killed); + +/** +* drm_dep_queue_is_initialized() - check whether a dep queue has been initialized +* @q: dep queue to check +* +* A queue is considered initialized once its ops pointer has been set by a +* successful call to drm_dep_queue_init(). Drivers that embed a +* &drm_dep_queue inside a larger structure may call this before attempting any +* other queue operation to confirm that initialization has taken place. +* drm_dep_queue_put() must be called if this function returns true to drop the +* initialization reference from drm_dep_queue_init(). +* +* Return: true if @q has been initialized, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_initialized (struct drm_dep_queue *q) +{ +return !!q->ops; +} +EXPORT_SYMBOL (drm_dep_queue_is_initialized); + +/** +* drm_dep_queue_set_stopped() - pre-mark a queue as stopped before first use +* @q: dep queue to mark +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going through the +* normal drm_dep_queue_stop() path. This is only valid during the driver-side +* queue initialisation sequence — i.e. after drm_dep_queue_init() returns but +* before the queue is made visible to other threads (e.g. before it is added +* to any lookup structures). Using this after the queue is live is a driver +* bug; use drm_dep_queue_stop() instead. +* +* Context: Process context, queue not yet visible to other threads. +*/ +void drm_dep_queue_set_stopped (struct drm_dep_queue *q) +{ + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_STOPPED ; +} +EXPORT_SYMBOL (drm_dep_queue_set_stopped); + +/** +* drm_dep_queue_refcount() - read the current reference count of a queue +* @q: dep queue to query +* +* Returns the instantaneous kref value. The count may change immediately +* after this call; callers must not make safety decisions based solely on +* the returned value. Intended for diagnostic snapshots and debugfs output. +* +* Context: Any context. +* Return: current reference count. +*/ +unsigned int drm_dep_queue_refcount (const struct drm_dep_queue *q) +{ +return kref_read (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_refcount); + +/** +* drm_dep_queue_timeout() - read the per-job TDR timeout for a queue +* @q: dep queue to query +* +* Returns the per-job timeout in jiffies as set at init time. +* %MAX_SCHEDULE_TIMEOUT means no timeout is configured. +* +* Context: Any context. +* Return: timeout in jiffies. +*/ +long drm_dep_queue_timeout (const struct drm_dep_queue *q) +{ +return q->job.timeout; +} +EXPORT_SYMBOL (drm_dep_queue_timeout); + +/** +* drm_dep_queue_is_job_put_irq_safe() - test whether job-put from IRQ is allowed +* @q: dep queue +* +* Context: Any context. +* Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set, +* false otherwise. +*/ +static bool drm_dep_queue_is_job_put_irq_safe (const struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE ); +} + +/** +* drm_dep_queue_job_dependency() - get next unresolved dep fence +* @q: dep queue +* @job: job whose dependencies to advance +* +* Returns NULL immediately if the queue has been killed via +* drm_dep_queue_kill(), bypassing all dependency waits so that jobs +* drain through run_job as quickly as possible. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next unresolved &dma_fence with a new reference, or NULL +* when all dependencies have been consumed (or the queue is killed). +*/ +static struct dma_fence * +drm_dep_queue_job_dependency (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *f; + +lockdep_assert_held (&q->sched.lock); + +if (drm_dep_queue_is_killed (q)) +return NULL ; + + f =xa_load (&job->dependencies, job->last_dependency); +if (f) { + job->last_dependency++; +if (WARN_ON (DRM_DEP_JOB_FENCE_PREALLOC == f)) +return dma_fence_get_stub (); +return dma_fence_get (f); + } + +return NULL ; +} + +/** +* drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence
drivers/gpu/drm/dep/drm_dep_queue.c line 488
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @q: dep queue +* @job: job whose dependency fence is stored in @q->dep.fence +* +* Installs a wakeup callback on @q->dep.fence. Returns true if the +* callback was installed (the queue must wait), false if the fence is +* already signalled or is a self-fence from the same queue context. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if callback installed, false if fence already done. +*/
In Rust, we can encode the signaling paths with a “token type”. So any sections that are part of the signaling path can simply take this token as an argument. This type also enforces that end_signaling() is called automatically when it goes out of scope.
By the way, we can easily offer an irq handler type where we enforce this:
fn handle_threaded_irq(&self, device: &Device<Bound>) -> IrqReturn {
let _annotation = DmaFenceSignallingAnnotation::new(); // Calls begin_signaling() self.driver.handle_threaded_irq(device)
// end_signaling() is called here automatically.
}
Same for workqueues:
fn work_fn(&self, device: &Device<Bound>) {
let _annotation = DmaFenceSignallingAnnotation::new(); // Calls begin_signaling() self.driver.work_fn(device)
// end_signaling() is called here automatically.
}
This is not Rust-specific, of course, but it is more ergonomic to write in Rust.
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+static bool drm_dep_queue_add_dep_cb (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *fence = q->dep.fence; + +lockdep_assert_held (&q->sched.lock); + +if (WARN_ON (fence->context == q->fence.context)) { +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; +return false; + }
drivers/gpu/drm/dep/drm_dep_queue.c line 510
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +if (!dma_fence_add_callback (q->dep.fence, &q->dep.cb, + drm_dep_queue_wakeup)) +return true; + +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; + +return false; +}
In rust we can enforce that all callbacks take a reference to the fence automatically. If the callback is “forgotten” in a buggy path, it is automatically removed, and the fence is automatically signaled with -ECANCELED.
Quoted 36 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +36
-0+ +/** +* drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC queue +* @q: dep queue +* +* Peeks at the head of the SPSC queue and drains all resolved +* dependencies. If a dependency is still pending, installs a wakeup +* callback and returns NULL. On success pops the job and returns it. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next dispatchable job, or NULL if a dep is still pending. +*/ +static struct drm_dep_job *drm_dep_queue_pop_job (struct drm_dep_queue *q) +{ +struct spsc_node *node; +struct drm_dep_job *job; + +lockdep_assert_held (&q->sched.lock); + + node =spsc_queue_peek (&q->job.queue); +if (!node) +return NULL ; + + job = container_of(node, struct drm_dep_job, queue_node); + +while ((q->dep.fence =drm_dep_queue_job_dependency (q, job))) { +if (drm_dep_queue_add_dep_cb (q, job)) +return NULL ; + } + +spsc_queue_pop (&q->job.queue); + +return job; +} + +/*
drivers/gpu/drm/dep/drm_dep_queue.c line 556
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* drm_dep_queue_get_unless_zero() - try to acquire a queue reference +* +* Workers use this instead of drm_dep_queue_get() to guard against the zombie +* state: the queue's refcount has already reached zero (async teardown is in +* flight) but a work item was queued before free_work had a chance to cancel +* it. If kref_get_unless_zero() fails the caller must bail immediately. +* +* Context: Any context. +* Returns true if the reference was acquired, false if the queue is zombie. +*/
Again, this function is totally gone in Rust.
Quoted 1,021 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +1021
-0+bool drm_dep_queue_get_unless_zero (struct drm_dep_queue *q) +{ +return kref_get_unless_zero (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_get_unless_zero); + +/** +* drm_dep_queue_run_job_work() - run-job worker +* @work: work item embedded in the dep queue +* +* Acquires @q->sched.lock, checks stopped state, queue readiness and +* available credits, pops the next job via drm_dep_queue_pop_job(), +* dispatches it via drm_dep_queue_run_job(), then re-kicks itself. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_run_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.run_job); +struct spsc_node *node; +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +mutex_lock (&q->sched.lock); + +if (drm_dep_queue_is_stopped (q)) + goto put_queue; + +if (!drm_dep_queue_is_ready (q)) + goto put_queue; + +/* Peek to check credits before committing to pop and dep resolution */ + node =spsc_queue_peek (&q->job.queue); +if (!node) + goto put_queue; + + job = container_of(node, struct drm_dep_job, queue_node); +if (!drm_dep_queue_has_credits (q, job)) + goto put_queue; + + job =drm_dep_queue_pop_job (q); +if (!job) + goto put_queue; + +drm_dep_queue_run_job (q, job); +drm_dep_queue_run_job_queue (q); + +put_queue: +mutex_unlock (&q->sched.lock); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/* +* drm_dep_queue_remove_job() - unlink a job from the pending list and reset TDR +* @q: dep queue owning @job +* @job: job to remove +* +* Splices @job out of @q->job.pending, cancels any pending TDR delayed work, +* and arms the timeout for the new list head (if any). +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_remove_job (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->job.lock); + +list_del_init (&job->pending_link); +cancel_delayed_work (&q->sched.tdr); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_get_finished_job() - dequeue a finished job +* @q: dep queue +* +* Under @q->job.lock checks the head of the pending list for a +* finished dep fence. If found, removes the job from the list, +* cancels the TDR, and re-arms it for the new head. +* +* Context: Process context (workqueue). DMA fence signaling path. +* Return: the finished &drm_dep_job, or NULL if none is ready. +*/ +static struct drm_dep_job * +drm_dep_queue_get_finished_job (struct drm_dep_queue *q) +{ +struct drm_dep_job *job; + +guard (spinlock_irq)(&q->job.lock); + + job = list_first_entry_or_null(&q->job.pending, struct drm_dep_job, + pending_link); +if (job &&drm_dep_fence_is_finished (job->dfence)) +drm_dep_queue_remove_job (q, job); +else + job =NULL ; + +return job; +} + +/** +* drm_dep_queue_put_job_work() - put-job worker +* @work: work item embedded in the dep queue +* +* Drains all finished jobs by calling drm_dep_job_put() in a loop, +* then kicks the run-job worker. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because workqueue is shared with other items in the fence signaling path. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_put_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.put_job); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +while ((job =drm_dep_queue_get_finished_job (q))) +drm_dep_job_put (job); + +drm_dep_queue_run_job_queue (q); + +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_queue_tdr_work() - TDR worker +* @work: work item embedded in the delayed TDR work +* +* Removes the head job from the pending list under @q->job.lock, +* asserts @q->ops->timedout_job is non-NULL, calls it outside the lock, +* requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the +* queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and always +* restarts the TDR timer after handling the job (unless @q is stopping). +* Any other return value triggers a WARN. +* +* The TDR is never armed when @q->ops->timedout_job is NULL, so firing +* this worker without a timedout_job callback is a driver bug. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because timedout_job() is expected to signal the guilty job's fence as part +* of reset. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_tdr_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.tdr.work); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + + scoped_guard(spinlock_irq, &q->job.lock) { + job =list_first_entry_or_null (&q->job.pending, + struct drm_dep_job, + pending_link); +if (job) +/* +* Remove from pending so it cannot be freed +* concurrently by drm_dep_queue_get_finished_job() or +* .drm_dep_job_done(). +*/ +list_del_init (&job->pending_link); + } + +if (job) { +enum drm_dep_timedout_stat status; + +if (WARN_ON (!q->ops->timedout_job)) { +drm_dep_job_put (job); + goto out; + } + + status = q->ops->timedout_job (job); + +switch (status) { +case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB : + scoped_guard(spinlock_irq, &q->job.lock) +list_add (&job->pending_link, &q->job.pending); +drm_dep_queue_put_job_queue (q); +break ; +case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED : +drm_dep_job_put (job); +break ; +default : +WARN_ON ("invalid drm_dep_timedout_stat" ); +break ; + } + } + +out: +drm_queue_start_timeout_unlocked (q); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue +* @name: name for the workqueue +* @flags: DRM_DEP_QUEUE_FLAGS_* flags +* +* Allocates an ordered workqueue for job submission with %WQ_MEM_RECLAIM and +* %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use from +* memory reclaim context and properly annotated for lockdep taint tracking. +* Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When +* CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct * +drm_dep_alloc_submit_wq (const char *name,enum drm_dep_queue_flags flags) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +if (flags &DRM_DEP_QUEUE_FLAGS_HIGHPRI ) + wq_flags |=WQ_HIGHPRI ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_submit_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue +* @name: name for the workqueue +* +* Allocates an ordered workqueue for timeout detection and recovery with +* %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring consistent taint +* annotation with the submit workqueue. When CONFIG_LOCKDEP is enabled, uses +* a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct *drm_dep_alloc_timeout_wq (const char *name) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_timeout_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_queue_init() - initialize a dep queue +* @q: dep queue to initialize +* @args: initialization arguments +* +* Initializes all fields of @q from @args. If @args->submit_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds one kref +* reference and drm_dep_queue_put() must be called to drop this reference +* (i.e., drivers cannot directly free the queue). +* +* When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against the +* fs_reclaim pseudo-lock so that lockdep can detect any lock ordering +* inversion between @sched.lock and memory reclaim. +* +* Return: 0 on success, %-EINVAL when @args->credit_limit is zero, @args->ops +* is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when +* @args->submit_wq or @args->timeout_wq is non-NULL but was not allocated with +* %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails. +* +* Context: Process context. May allocate memory and create workqueues. +*/ +int drm_dep_queue_init (struct drm_dep_queue *q, +const struct drm_dep_queue_init_args *args) +{ +if (!args->credit_limit || !args->drm || !args->ops || + !args->ops->run_job) +return -EINVAL ; + +if (args->submit_wq && !workqueue_is_reclaim_annotated (args->submit_wq)) +return -EINVAL ; + +if (args->timeout_wq && + !workqueue_is_reclaim_annotated (args->timeout_wq)) +return -EINVAL ; + +memset (q,0 ,sizeof (*q)); + + q->name = args->name; + q->drm = args->drm; + q->credit.limit = args->credit_limit; + q->job.timeout = args->timeout ? args->timeout :MAX_SCHEDULE_TIMEOUT ; + +init_rcu_head (&q->rcu); +INIT_LIST_HEAD (&q->job.pending); +spin_lock_init (&q->job.lock); +spsc_queue_init (&q->job.queue); + +mutex_init (&q->sched.lock); +if (IS_ENABLED (CONFIG_LOCKDEP )) { +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&q->sched.lock); +fs_reclaim_release (GFP_KERNEL ); + } + +if (args->submit_wq) { + q->sched.submit_wq = args->submit_wq; + }else { + q->sched.submit_wq =drm_dep_alloc_submit_wq (args->name ?:"drm_dep" , + args->flags); +if (!q->sched.submit_wq) +return -ENOMEM ; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ; + } + +if (args->timeout_wq) { + q->sched.timeout_wq = args->timeout_wq; + }else { + q->sched.timeout_wq =drm_dep_alloc_timeout_wq (args->name ?:"drm_dep" ); +if (!q->sched.timeout_wq) + goto err_submit_wq; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ; + } + + q->sched.flags |= args->flags & + ~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ | +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ); + +INIT_DELAYED_WORK (&q->sched.tdr, drm_dep_queue_tdr_work); +INIT_WORK (&q->sched.run_job, drm_dep_queue_run_job_work); +INIT_WORK (&q->sched.put_job, drm_dep_queue_put_job_work); + + q->fence.context =dma_fence_context_alloc (1 ); + +kref_init (&q->refcount); + q->ops = args->ops; +drm_dev_get (q->drm); + +return 0 ; + +err_submit_wq: +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); +mutex_destroy (&q->sched.lock); + +return -ENOMEM ; +} +EXPORT_SYMBOL (drm_dep_queue_init); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* drm_dep_queue_push_job_begin() - mark the start of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the start of drm_dep_job_arm() and warns if the push context is +* already owned by another task, which would indicate concurrent arm/push on +* the same queue. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner); + q->job.push.owner = current; +} + +/** +* drm_dep_queue_push_job_end() - mark the end of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the end of drm_dep_job_push() and warns if the push context is not +* owned by the current task, which would indicate a mismatched begin/end pair +* or a push from the wrong thread. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner != current); + q->job.push.owner =NULL ; +} +#endif + +/** +* drm_dep_queue_assert_teardown_invariants() - assert teardown invariants +* @q: dep queue being torn down +* +* Warns if the pending-job list, the SPSC submission queue, or the credit +* counter is non-zero when called, or if the queue still has a non-zero +* reference count. +* +* Context: Any context. +*/ +static void drm_dep_queue_assert_teardown_invariants (struct drm_dep_queue *q) +{ +WARN_ON (!list_empty (&q->job.pending)); +WARN_ON (spsc_queue_count (&q->job.queue)); +WARN_ON (atomic_read (&q->credit.count)); +WARN_ON (drm_dep_queue_refcount (q)); +} + +/** +* drm_dep_queue_release() - final internal cleanup of a dep queue +* @q: dep queue to clean up +* +* Asserts teardown invariants and destroys internal resources allocated by +* drm_dep_queue_init() that cannot be torn down earlier in the teardown +* sequence. Currently this destroys @q->sched.lock. +* +* Drivers that implement &drm_dep_queue_ops.release **must** call this +* function after removing @q from any internal bookkeeping (e.g. lookup +* tables or lists) but before freeing the memory that contains @q. When +* &drm_dep_queue_ops.release is NULL, drm_dep follows the default teardown +* path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_release (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); +mutex_destroy (&q->sched.lock); +} +EXPORT_SYMBOL (drm_dep_queue_release); + +/** +* drm_dep_queue_free() - final cleanup of a dep queue +* @q: dep queue to free +* +* Invokes &drm_dep_queue_ops.release if set, in which case the driver is +* responsible for calling drm_dep_queue_release() and freeing @q itself. +* If &drm_dep_queue_ops.release is NULL, calls drm_dep_queue_release() +* and then frees @q with kfree_rcu(). +* +* In either case, releases the drm_dev_get() reference taken at init time +* via drm_dev_put(), allowing the owning &drm_device to be unloaded once +* all queues have been freed. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free (struct drm_dep_queue *q) +{ +struct drm_device *drm = q->drm; + +if (q->ops->release) { + q->ops->release (q); + }else { +drm_dep_queue_release (q); +kfree_rcu (q, rcu); + } +drm_dev_put (drm); +} + +/** +* drm_dep_queue_free_work() - deferred queue teardown worker +* @work: free_work item embedded in the dep queue +* +* Runs on dep_free_wq. Disables all work items synchronously +* (preventing re-queue and waiting for in-flight instances), +* destroys any owned workqueues, then calls drm_dep_queue_free(). +* Running on dep_free_wq ensures destroy_workqueue() is never +* called from within one of the queue's own workers (deadlock) +* and disable_*_sync() cannot deadlock either. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, free_work); + +drm_dep_queue_assert_teardown_invariants (q); + +disable_delayed_work_sync (&q->sched.tdr); +disable_work_sync (&q->sched.run_job); +disable_work_sync (&q->sched.put_job); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ) +destroy_workqueue (q->sched.timeout_wq); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); + +drm_dep_queue_free (q); +} + +/** +* drm_dep_queue_fini() - tear down a dep queue +* @q: dep queue to tear down +* +* Asserts teardown invariants and nitiates teardown of @q by queuing the +* deferred free work onto tht module-private dep_free_wq workqueue. The work +* item disables any pending TDR and run/put-job work synchronously, destroys +* any workqueues that were allocated by drm_dep_queue_init(), and then releases +* the queue memory. +* +* Running teardown from dep_free_wq ensures that destroy_workqueue() is never +* called from within one of the queue's own workers (e.g. via +* drm_dep_queue_put()), which would deadlock. +* +* Drivers can wait for all outstanding deferred work to complete by waiting +* for the last drm_dev_put() reference on their &drm_device, which is +* released as the final step of each queue's teardown. +* +* Drivers that implement &drm_dep_queue_ops.fini **must** call this +* function after removing @q from any device bookkeeping but before freeing the +* memory that contains @q. When &drm_dep_queue_ops.fini is NULL, drm_dep +* follows the default teardown path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_fini (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); + +INIT_WORK (&q->free_work, drm_dep_queue_free_work); +queue_work (dep_free_wq, &q->free_work); +} +EXPORT_SYMBOL (drm_dep_queue_fini); + +/** +* drm_dep_queue_get() - acquire a reference to a dep queue +* @q: dep queue to acquire a reference on, or NULL +* +* Return: @q with an additional reference held, or NULL if @q is NULL. +* +* Context: Any context. +*/ +struct drm_dep_queue *drm_dep_queue_get (struct drm_dep_queue *q) +{ +if (q) +kref_get (&q->refcount); +return q; +} +EXPORT_SYMBOL (drm_dep_queue_get); + +/** +* __drm_dep_queue_release() - kref release callback for a dep queue +* @kref: kref embedded in the dep queue +* +* Calls &drm_dep_queue_ops.fini if set, otherwise calls +* drm_dep_queue_fini() to initiate deferred teardown. +* +* Context: Any context. +*/ +static void __drm_dep_queue_release (struct kref *kref) +{ +struct drm_dep_queue *q = + container_of(kref, struct drm_dep_queue, refcount); + +if (q->ops->fini) + q->ops->fini (q); +else +drm_dep_queue_fini (q); +} + +/** +* drm_dep_queue_put() - release a reference to a dep queue +* @q: dep queue to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_queue_ops.fini if set, +* otherwise calls drm_dep_queue_fini(). Final memory release is handled by +* &drm_dep_queue_ops.release (which must call drm_dep_queue_release()) if set, +* or drm_dep_queue_release() followed by kfree_rcu() otherwise. +* Does nothing if @q is NULL. +* +* Context: Any context. +*/ +void drm_dep_queue_put (struct drm_dep_queue *q) +{ +if (q) +kref_put (&q->refcount, __drm_dep_queue_release); +} +EXPORT_SYMBOL (drm_dep_queue_put); + +/** +* drm_dep_queue_stop() - stop a dep queue from processing new jobs +* @q: dep queue to stop +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from finished +* fenced signaling context. Then cancels any in-flight run_job and put_job work +* items. Once stopped, the bypass path and the submit workqueue will not +* dispatch further jobs nor will any jobs be removed from the pending list. +* Call drm_dep_queue_start() to resume processing. +* +* Context: Process context. Waits for in-flight workers to complete. +*/ +void drm_dep_queue_stop (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +cancel_work_sync (&q->sched.run_job); +cancel_work_sync (&q->sched.put_job); +} +EXPORT_SYMBOL (drm_dep_queue_stop); + +/** +* drm_dep_queue_start() - resume a stopped dep queue +* @q: dep queue to start +* +* Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from IRQ +* context. Then re-queues the run_job and put_job work items so that any jobs +* pending since the queue was stopped are processed. Must only be called after +* drm_dep_queue_stop(). +* +* Context: Process context. +*/ +void drm_dep_queue_start (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_clear (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +drm_dep_queue_run_job_queue (q); +drm_dep_queue_put_job_queue (q); +} +EXPORT_SYMBOL (drm_dep_queue_start); + +/** +* drm_dep_queue_trigger_timeout() - trigger the TDR immediately for +* all pending jobs +* @q: dep queue to trigger timeout on +* +* Sets @q->job.timeout to 1 and arms the TDR delayed work with a one-jiffy +* delay, causing it to fire almost immediately without hot-spinning at zero +* delay. This is used to force-expire any pendind jobs on the queue, for +* example when the device is being torn down or has encountered an +* unrecoverable error. +* +* It is suggested that when this function is used, the first timedout_job call +* causes the driver to kick the queue off the hardware and signal all pending +* job fences. Subsequent calls continue to signal all pending job fences. +* +* Has no effect if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_trigger_timeout (struct drm_dep_queue *q) +{ +guard (spinlock_irqsave)(&q->job.lock); + q->job.timeout =1 ; +drm_queue_start_timeout (q); +} +EXPORT_SYMBOL (drm_dep_queue_trigger_timeout); + +/** +* drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait +* for it to finish +* @q: dep queue whose TDR to cancel +* +* Cancels the TDR delayed work item if it has not yet started, and waits for +* it to complete if it is already running. After this call returns, the TDR +* worker is guaranteed not to be executing and will not fire again until +* explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by a new +* job being submitted). +* +* Useful during error recovery or queue teardown when the caller needs to +* know that no timeout handling races with its own reset logic. +* +* Context: Process context. May sleep waiting for the TDR worker to finish. +*/ +void drm_dep_queue_cancel_tdr_sync (struct drm_dep_queue *q) +{ +cancel_delayed_work_sync (&q->sched.tdr); +} +EXPORT_SYMBOL (drm_dep_queue_cancel_tdr_sync); + +/** +* drm_dep_queue_resume_timeout() - restart the TDR timer with the +* configured timeout +* @q: dep queue to resume the timeout for +* +* Restarts the TDR delayed work using @q->job.timeout. Called after device +* recovery to give pending jobs a fresh full timeout window. Has no effect +* if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_resume_timeout (struct drm_dep_queue *q) +{ +drm_queue_start_timeout_unlocked (q); +} +EXPORT_SYMBOL (drm_dep_queue_resume_timeout); + +/** +* drm_dep_queue_is_stopped() - check whether a dep queue is stopped +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_stopped (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_STOPPED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_stopped); + +/** +* drm_dep_queue_kill() - kill a dep queue and flush all pending jobs +* @q: dep queue to kill +* +* Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock. If a +* dependency fence is currently being waited on, its callback is removed and +* the run-job worker is kicked immediately so that the blocked job drains +* without waiting. +* +* Once killed, drm_dep_queue_job_dependency() returns NULL for all jobs, +* bypassing dependency waits so that every queued job drains through +* &drm_dep_queue_ops.run_job without blocking. +* +* The &drm_dep_queue_ops.run_job callback is guaranteed to be called for every +* job that was pushed before or after drm_dep_queue_kill(), even during queue +* teardown. Drivers should use this guarantee to perform any necessary +* bookkeeping cleanup without executing the actual backend operation when the +* queue is killed. +* +* Unlike drm_dep_queue_stop(), killing is one-way: there is no corresponding +* start function. +* +* **Driver safety requirement** +* +* drm_dep_queue_kill() must only be called once the driver can guarantee that +* no job in the queue will touch memory associated with any of its fences +* (i.e., the queue has been removed from the device and will never be put back +* on). +* +* Context: Process context. +*/ +void drm_dep_queue_kill (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { +struct dma_fence *fence; + +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_KILLED ); + +/* +* Holding &q->sched.lock guarantees that the run-job work item +* cannot drop its reference to q->dep.fence concurrently, so +* reading q->dep.fence here is safe. +*/ + fence =READ_ONCE (q->dep.fence); +if (fence &&dma_fence_remove_callback (fence, &q->dep.cb)) +drm_dep_queue_remove_dependency (q, fence); + } +} +EXPORT_SYMBOL (drm_dep_queue_kill); + +/** +* drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Drivers may use this to queue their own work items alongside the queue's +* internal run-job and put-job workers — for example to process incoming +* messages in the same serialisation domain. +* +* Prefer drm_dep_queue_work_enqueue() when the only need is to enqueue a +* work item, as it additionally checks the stopped state. Use this accessor +* when the workqueue itself is required (e.g. for alloc_ordered_workqueue +* replacement or drain_workqueue calls). +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for job submission. +*/ +struct workqueue_struct *drm_dep_queue_submit_wq (struct drm_dep_queue *q) +{ +return q->sched.submit_wq; +} +EXPORT_SYMBOL (drm_dep_queue_submit_wq); + +/** +* drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Returns the workqueue used by @q to run TDR (timeout detection and recovery) +* work. Drivers may use this to queue their own timeout-domain work items, or +* to call drain_workqueue() when tearing down and needing to ensure all pending +* timeout callbacks have completed before proceeding. +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for TDR work. +*/ +struct workqueue_struct *drm_dep_queue_timeout_wq (struct drm_dep_queue *q) +{ +return q->sched.timeout_wq; +} +EXPORT_SYMBOL (drm_dep_queue_timeout_wq); + +/** +* drm_dep_queue_work_enqueue() - queue work on the dep queue's submit workqueue +* @q: dep queue to enqueue work on +* @work: work item to enqueue +* +* Queues @work on @q->sched.submit_wq if the queue is not stopped. This +* allows drivers to schedule custom work items that run serialised with the +* queue's own run-job and put-job workers. +* +* Return: true if the work was queued, false if the queue is stopped or the +* work item was already pending. +* +* Context: Any context. +*/ +bool drm_dep_queue_work_enqueue (struct drm_dep_queue *q, +struct work_struct *work) +{ +if (drm_dep_queue_is_stopped (q)) +return false; + +return queue_work (q->sched.submit_wq, work); +} +EXPORT_SYMBOL (drm_dep_queue_work_enqueue); + +/** +* drm_dep_queue_can_job_bypass() - test whether a job can skip the SPSC queue +* @q: dep queue +* @job: job to test +* +* A job may bypass the submit workqueue and run inline on the calling thread +* if all of the following hold: +* +* - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue +* - the queue is not stopped +* - the SPSC submission queue is empty (no other jobs waiting) +* - the queue has enough credits for @job +* - @job has no unresolved dependency fences +* +* Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). +* Return: true if the job may be run inline, false otherwise. +*/ +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->sched.lock); + +return q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED && + !drm_dep_queue_is_stopped (q) && + !spsc_queue_count (&q->job.queue) && +drm_dep_queue_has_credits (q, job) && +xa_empty (&job->dependencies); +} + +/** +* drm_dep_job_done() - mark a job as complete +* @job: the job that finished +* @result: error code to propagate, or 0 for success +* +* Subtracts @job->credits from the queue credit counter, then signals the +* job's dep fence with @result. +* +* When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), a +* temporary extra reference is taken on @job before signalling the fence. +* This prevents a concurrent put-job worker — which may be woken by timeouts or +* queue starting — from freeing the job while this function still holds a +* pointer to it. The extra reference is released at the end of the function. +* +* After signalling, the IRQ-safe path removes the job from the pending list +* under @q->job.lock, provided the queue is not stopped. Removal is skipped +* when the queue is stopped so that drm_dep_queue_for_each_pending_job() can +* iterate the list without racing with the completion path. On successful +* removal, kicks the run-job worker so the next queued job can be dispatched +* immediately, then drops the job reference. If the job was already removed +* by TDR, or removal was skipped because the queue is stopped, kicks the +* put-job worker instead to allow the deferred put to complete. +* +* Context: Any context. +*/ +static void drm_dep_job_done (struct drm_dep_job *job,int result) +{ +struct drm_dep_queue *q = job->q; +bool irq_safe =drm_dep_queue_is_job_put_irq_safe (q), removed = false; + +/* +* Local ref to ensure the put worker—which may be woken by external +* forces (TDR, driver-side queue starting)—doesn't free the job behind +* this function's back after drm_dep_fence_done() while it is still on +* the pending list. +*/ +if (irq_safe) +drm_dep_job_get (job); + +atomic_sub (job->credits, &q->credit.count); +drm_dep_fence_done (job->dfence, result); + +/* Only safe to touch job after fence signal if we have a local ref. */ + +if (irq_safe) { + scoped_guard(spinlock_irqsave, &q->job.lock) { + removed = !list_empty (&job->pending_link) && + !drm_dep_queue_is_stopped (q); + +/* Guard against TDR operating on job */ +if (removed) +drm_dep_queue_remove_job (q, job); + } + } + +if (removed) { +drm_dep_queue_run_job_queue (q); +drm_dep_job_put (job); + }else { +drm_dep_queue_put_job_queue (q); + } + +if (irq_safe) +drm_dep_job_put (job); +} + +/** +* drm_dep_job_done_cb() - dma_fence callback to complete a job +* @f: the hardware fence that signalled +* @cb: fence callback embedded in the dep job +* +* Extracts the job from @cb and calls drm_dep_job_done() with +* @f->error as the result. +* +* Context: Any context, but with IRQ disabled. May not sleep. +*/ +static void drm_dep_job_done_cb (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_job *job = container_of(cb, struct drm_dep_job, cb); + +drm_dep_job_done (job, f->error); +} + +/** +* drm_dep_queue_run_job() - submit a job to hardware and set up +* completion tracking +* @q: dep queue +* @job: job to run +* +* Accounts @job->credits against the queue, appends the job to the pending +* list, then calls @q->ops->run_job(). The TDR timer is started only when +* @job is the first entry on the pending list; subsequent jobs added while +* a TDR is already in flight do not reset the timer (which would otherwise +* extend the deadline for the already-running head job). Stores the returned +* hardware fence as the parent of the job's dep fence, then installs +* drm_dep_job_done_cb() on it. If the hardware fence is already signalled +* (%-ENOENT from dma_fence_add_callback()) or run_job() returns NULL/error, +* the job is completed immediately. Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). DMA fence +* signaling path. +*/ +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +struct dma_fence *fence; +int r; + +lockdep_assert_held (&q->sched.lock); + +drm_dep_job_get (job); +atomic_add (job->credits, &q->credit.count); + + scoped_guard(spinlock_irq, &q->job.lock) { +bool first =list_empty (&q->job.pending); + +list_add_tail (&job->pending_link, &q->job.pending); +if (first) +drm_queue_start_timeout (q); + } + + fence = q->ops->run_job (job); +drm_dep_fence_set_parent (job->dfence, fence); + +if (!IS_ERR_OR_NULL (fence)) { + r =dma_fence_add_callback (fence, &job->cb, + drm_dep_job_done_cb); +if (r == -ENOENT ) +drm_dep_job_done (job, fence->error); +else if (r) +drm_err (q->drm,"fence add callback failed (%d)\n" , r); +dma_fence_put (fence);
drivers/gpu/drm/dep/drm_dep_queue.c line 1587
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ }else { +drm_dep_job_done (job,IS_ERR (fence) ?PTR_ERR (fence) :0 ); + } + +/* +* Drop all input dependency fences now, in process context, before the +* final job put. Once the job is on the pending list its last reference +* may be dropped from a dma_fence callback (IRQ context), where calling +* xa_destroy() would be unsafe. +*/
I assume that “pending” is the list of jobs that have been handed to the driver via ops->run_job()?
Can’t this problem be solved by not doing anything inside a dma_fence callback other than scheduling the queue worker?
Quoted 201 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +60
-0+drm_dep_job_drop_dependencies (job); +drm_dep_job_put (job); +} + +/** +* drm_dep_queue_push_job() - enqueue a job on the SPSC submission queue +* @q: dep queue +* @job: job to push +* +* Pushes @job onto the SPSC queue. If the queue was previously empty +* (i.e. this is the first pending job), kicks the run_job worker so it +* processes the job promptly without waiting for the next wakeup. +* May be called with or without @q->sched.lock held. +* +* Context: Any context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +/* +* spsc_queue_push() returns true if the queue was previously empty, +* i.e. this is the first pending job. Kick the run_job worker so it +* picks it up without waiting for the next wakeup. +*/ +if (spsc_queue_push (&q->job.queue, &job->queue_node)) +drm_dep_queue_run_job_queue (q); +} + +/** +* drm_dep_init() - module initialiser +* +* Allocates the module-private dep_free_wq unbound workqueue used for +* deferred queue teardown. +* +* Return: 0 on success, %-ENOMEM if workqueue allocation fails. +*/ +static int __init drm_dep_init (void ) +{ + dep_free_wq =alloc_workqueue ("drm_dep_free" ,WQ_UNBOUND ,0 ); +if (!dep_free_wq) +return -ENOMEM ; + +return 0 ; +} + +/** +* drm_dep_exit() - module exit +* +* Destroys the module-private dep_free_wq workqueue. +*/ +static void __exit drm_dep_exit(void) +{ +destroy_workqueue (dep_free_wq); + dep_free_wq =NULL ; +} + +module_init (drm_dep_init); +module_exit (drm_dep_exit); + +MODULE_DESCRIPTION ("DRM dependency queue" ); +MODULE_LICENSE ("Dual MIT/GPL" );drivers/gpu/drm/dep/drm_dep_queue.h new file +31
-0@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_QUEUE_H_ +#define _DRM_DEP_QUEUE_H_ + +#include <linux/types.h> + +struct drm_dep_job ; +struct drm_dep_queue ; + +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job); +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job); +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q); +void drm_dep_queue_push_job_end (struct drm_dep_queue *q); +#else +static inline void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +} +static inline void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +} +#endif + +#endif /* _DRM_DEP_QUEUE_H_ */ include/drm/drm_dep.h new file +98
-0@@ -0,0 +1,597 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_H_ +#define _DRM_DEP_H_ + +#include <drm/spsc_queue.h> +#include <linux/dma-fence.h> +#include <linux/xarray.h> +#include <linux/workqueue.h> + +enum dma_resv_usage ; +struct dma_resv ; +struct drm_dep_fence ; +struct drm_dep_job ; +struct drm_dep_queue ; +struct drm_file ; +struct drm_gem_object ; + +/** +* enum drm_dep_timedout_stat - return value of &drm_dep_queue_ops.timedout_job +* @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's finished +* fence during reset; drm_dep may safely drop its reference to the job. +* @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; reinsert the +* job at the head of the pending list so it can complete normally. +*/ +enum drm_dep_timedout_stat { +DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED , +DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB , +}; + +/** +* struct drm_dep_queue_ops - driver callbacks for a dep queue +*/ +struct drm_dep_queue_ops { +/** +* @run_job: submit the job to hardware. Returns the hardware completion +* fence (with a reference held for the scheduler), or NULL/ERR_PTR on +* synchronous completion or error. +*/ +struct dma_fence *(*run_job)(struct drm_dep_job *job); + +/** +* @timedout_job: called when the TDR fires for the head job. Must stop +* the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if the +* job's fence was signalled during reset, or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or +* signalling was otherwise delayed, and the job should be re-inserted +* at the head of the pending list. Any other value triggers a WARN. +*/ +enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job); + +/** +* @release: called when the last kref on the queue is dropped and +* drm_dep_queue_fini() has completed. The driver is responsible for +* removing @q from any internal bookkeeping, calling +* drm_dep_queue_release(), and then freeing the memory containing @q +* (e.g. via kfree_rcu() using @q->rcu). If NULL, drm_dep calls +* drm_dep_queue_release() and frees @q automatically via kfree_rcu(). +* Use this when the queue is embedded in a larger structure. +*/ +void (*release)(struct drm_dep_queue *q); + +/** +* @fini: if set, called instead of drm_dep_queue_fini() when the last +* kref is dropped. The driver is responsible for calling +* drm_dep_queue_fini() itself after it is done with the queue. Use this +* when additional teardown logic must run before fini (e.g., cleanup +* firmware resources associated with the queue). +*/ +void (*fini)(struct drm_dep_queue *q); +}; + +/** +* enum drm_dep_queue_flags - flags for &drm_dep_queue and +* &drm_dep_queue_init_args
include/drm/drm_dep.h line 108
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* +* Flags are divided into three categories: +* +* - **Private static**: set internally at init time and never changed. +* Drivers must not read or write these. +* %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ, +* %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ. +* +* - **Public dynamic**: toggled at runtime by drivers via accessors. +* Any modification must be performed under &drm_dep_queue.sched.lock.
Can’t enforce that in C.
include/drm/drm_dep.h line 115
Quoted 8 lines
include/drm/drm_dep.h new file +7
-0+* Accessor functions provide unstable reads. +* %DRM_DEP_QUEUE_FLAGS_STOPPED, +* %DRM_DEP_QUEUE_FLAGS_KILLED. +* +* - **Public static**: supplied by the driver in +* &drm_dep_queue_init_args.flags at queue creation time and not modified +* thereafter.
Same here.
Quoted 47 lines
+ * %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED, + * %DRM_DEP_QUEUE_FLAGS_HIGHPRI, + * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE.+ *
+ * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit workqueue was
+ * allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout workqueue
+ * was allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is stopped and+ * will not dispatch new jobs or remove jobs from the pending list, dropping + * the drm_dep-owned reference. Set by drm_dep_queue_stop(), cleared by + * drm_dep_queue_start(). + * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been killed + * via drm_dep_queue_kill(). Any active dependency wait is cancelled + * immediately. Jobs continue to flow through run_job for bookkeeping + * cleanup, but dependency waiting is skipped so that queued work drains + * as quickly as possible. + * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue supports + * the bypass path where eligible jobs skip the SPSC queue and run inline. + * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit workqueue owned + * by the queue is created with %WQ_HIGHPRI, causing run-job and put-job + * workers to execute at elevated priority. Only privileged clients (e.g. + * drivers managing time-critical or real-time GPU contexts) should request + * this flag; granting it to unprivileged userspace would allow priority + * inversion attacks. + * @drm_dep_queue_init_args.submit_wq is provided. + * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set, + * drm_dep_job_done() may be called from hardirq context (e.g. from a + * hardware-signalled dma_fence callback). drm_dep_job_done() will directly + * dequeue the job and call drm_dep_job_put() without deferring to a + * workqueue. The driver's &drm_dep_job_ops.release callback must therefore + * be safe to invoke from IRQ context.+ */
include/drm/drm_dep.h new file +11
-0+enum drm_dep_queue_flags { +DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ =BIT (0 ), +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ =BIT (1 ), +DRM_DEP_QUEUE_FLAGS_STOPPED =BIT (2 ), +DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED =BIT (3 ), +DRM_DEP_QUEUE_FLAGS_HIGHPRI =BIT (4 ), +DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE =BIT (5 ), +DRM_DEP_QUEUE_FLAGS_KILLED =BIT (6 ), +}; + +/**
include/drm/drm_dep.h line 171
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* struct drm_dep_queue - a dependency-tracked GPU submission queue +* +* Combines the role of &drm_gpu_scheduler and &drm_sched_entity into a single +* object. Each queue owns a submit workqueue (or borrows one), a timeout +* workqueue, an SPSC submission queue, and a pending-job list used for TDR. +* +* Initialise with drm_dep_queue_init(), tear down with drm_dep_queue_fini(). +* Reference counted via drm_dep_queue_get() / drm_dep_queue_put(). +* +* All fields are **opaque to drivers**. Do not read or write any field
Can’t enforce this in C.
Quoted 23 lines
+ * directly; use the provided helper functions instead. The sole exception
+ * is @rcu, which drivers may pass tokfree_rcu()when the queue is embedded
+ * inside a larger driver-managed structure and the &drm_dep_queue_ops.release
+ * vfunc performs an RCU-deferred free.+ */
+structdrm_dep_queue{
+ /** @ops: driver callbacks, set at init time. */
+ const structdrm_dep_queue_ops*ops;
+ /** @name: human-readable name used for workqueue and fence naming. */
+ const char *name;
+ /** @drm: owning DRM device; adrm_dev_get()reference is held for the
+ * lifetime of the queue to prevent module unload while queues are live.
+ */
+ structdrm_device*drm;
+ /** @refcount: reference count; use drm_dep_queue_get/put(). */
+ struct kref refcount;
+ /**
+ * @free_work: deferred teardown work queued unconditionally by
+ *drm_dep_queue_fini()onto the module-privatedep_free_wq. The work
+ * item disables pending workers synchronously and destroys any owned
+ * workqueues before releasing the queue memory and dropping the
+ *drm_dev_get()reference. Running ondep_free_wqensures
Quoted 8 lines
+ *
destroy_workqueue()is never called from within one of the queue's
+ * own workers.
+ */
+ structwork_structfree_work;
+ /**
+ * @rcu: RCU head for deferred freeing.
+ *
+ * This is the **only** field drivers may access directly. When the
We can enforce this in Rust at compile time.
Quoted 88 lines
+ * queue is embedded in a larger structure, implement
+ * &drm_dep_queue_ops.release, calldrm_dep_queue_release()to destroy
+ * internal resources, then pass this field tokfree_rcu()so that any
+ * in-flight RCU readers referencing the queue'sdma_fencetimeline name
+ * complete before the memory is returned. All other fields must be
+ * accessed through the provided helpers.
+ */
+ structrcu_headrcu;include/drm/drm_dep.h new file +80
-0+ +/** @sched: scheduling and workqueue state. */ +struct { +/** @sched.submit_wq: ordered workqueue for run/put-job work. */ +struct workqueue_struct *submit_wq; +/** @sched.timeout_wq: workqueue for the TDR delayed work. */ +struct workqueue_struct *timeout_wq; +/** +* @sched.run_job: work item that dispatches the next queued +* job. +*/ +struct work_struct run_job; +/** @sched.put_job: work item that frees finished jobs. */ +struct work_struct put_job; +/** @sched.tdr: delayed work item for timeout/reset (TDR). */ +struct delayed_work tdr; +/** +* @sched.lock: mutex serialising job dispatch, bypass +* decisions, stop/start, and flag updates. +*/ +struct mutex lock; +/** +* @sched.flags: bitmask of &enum drm_dep_queue_flags. +* Any modification after drm_dep_queue_init() must be +* performed under @sched.lock. +*/ +enum drm_dep_queue_flags flags; + } sched; + +/** @job: pending-job tracking state. */ +struct { +/** +* @job.pending: list of jobs that have been dispatched to +* hardware and not yet freed. Protected by @job.lock. +*/ +struct list_head pending; +/** +* @job.queue: SPSC queue of jobs waiting to be dispatched. +* Producers push via drm_dep_queue_push_job(); the run_job +* work item pops from the consumer side. +*/ +struct spsc_queue queue; +/** +* @job.lock: spinlock protecting @job.pending, TDR start, and +* the %DRM_DEP_QUEUE_FLAGS_STOPPED flag. Always acquired with +* irqsave (spin_lock_irqsave / spin_unlock_irqrestore) to +* support %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE queues where +* drm_dep_job_done() may run from hardirq context. +*/ +spinlock_t lock; +/** Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
include/drm/drm_dep.h line 270
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* @job.timeout: per-job TDR timeout in jiffies. +* %MAX_SCHEDULE_TIMEOUT means no timeout. +*/ +long timeout; +#if IS_ENABLED(CONFIG_PROVE_LOCKING) + /** + * @job.push: lockdep annotation tracking the arm-to-push + * critical section. + */ + struct {
We can enforce this in Rust automatically.
include/drm/drm_dep.h
Quoted 73 lines
+ */
+ structdma_fence*removed_fence;
+ /** @dep.cb: callback installed on @dep.fence. */
+ structdma_fence_cbcb;
+ } dep;include/drm/drm_dep.h new file +68
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
And if they don’t?
By the way, we can also enforce this in Rust.
include/drm/drm_dep.h
Quoted 88 lines
+ *
drm_dep_job_put()will callkfree()on the job directly.
+ */
+ void (*release)(structdrm_dep_job*job);include/drm/drm_dep.h new file +85
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
So if you’re not careful, you have just introduced a race :/
include/drm/drm_dep.h
Quoted 89 lines
+ *
+ * **Constraint: only fromsubmit_wqworker context**
+ *
+ * This guard must only be acquired from a work item running on the queue's
+ * submit workqueue (@q->sched.submit_wq) by drivers.
+ *
+ * Context: Process context only; must be called fromsubmit_wqwork by
+ * drivers.
+ */include/drm/drm_dep.h new file +80
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Not enforceable in C.
Quoted 51 lines
include/drm/drm_dep.h new file +48
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1
By the way:
I invite you to have a look at this implementation [0]. It currently works in real hardware i.e.: our downstream "Tyr" driver for Arm Mali is using that at the moment. It is a mere prototype that we’ve put together to test different approaches, so it’s not meant to be a “solution” at all. It’s a mere data point for further discussion.
Philip Stanner is working on this “Job Queue” concept too, but from an upstream perspective.
[0]: https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61
Part of the diff is not shown here. View the raw message.
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 8 lines · Daniel Almeida
(+cc a few other people + Rust-for-Linux ML)
Hi Matthew,
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Holy war? See my reply to Danilo — I’ll write this in Rust if needed, but it’s not my first choice since I’m not yet a native speaker.
Quoted 21 lines
Quoted 21 lines · Matthew Brost
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.
Quoted 5 lines
Quoted 3 lines
Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queueIn a domain that has been plagued by lifetime issues, we really should be
Yes, drm sched is a mess. I’ve been suggesting we fix it for years and have met pushback. This, however (drm dep), isn’t plagued by lifetime issues — that’s the primary focus here.
Quoted 2 lines
enforcing RAII for resource management instead of manual calls.
You can do RAII in C - see cleanup.h. Clear object lifetimes and ownership are what is important. Disciplined coding is the only to do this regardless of language. RAII doesn't help with help with bad object models / ownership / lifetime models either.
I don't buy the Rust solves everything argument but again non-native speaker.
Quoted 8 lines
Quoted 4 lines
reference from init until its last put, and the queue holds a job reference from dispatch until the
put_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.It makes use-after-free impossible _if_ you’re careful. It is not a property of the type system, and incorrect code will compile just fine.
Sure. If a driver puts a drm_dep object reference on a resource that drm_dep owns, it will explode. That’s effectively putting a reference on a resource the driver doesn’t own. A driver can write to any physical memory and crash the system anyway, so I’m not really sure what we’re talking about here. Rust doesn’t solve anything in this scenario — you can always use an unsafe block and put a reference on a resource you don’t own.
Object model, ownership, and lifetimes are what is important and that is what drm dep is built around.
Quoted 26 lines
Quoted 18 lines
The core objects are:
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).Quoted 7 lines
Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.
Quoted 4 lines
You cannot enforce this sequence easily in C code. Once again, we are trusting drivers that it is followed, but in Rust, you can simply reject code that does not follow this order at compile time.
I don’t know Rust, but yes — you can enforce this in C. It’s called lockdep and annotations. It’s not compile-time, but all of this is strictly enforced. e.g., write some code that doesn't follow this and report back if the kernel doesn't explode. It will, if doesn't I'll fix it to complain.
Quoted 11 lines
Quoted 11 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and therun_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().
Quoted 9 lines
Quoted 3 lines
Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Why can’t we design an API where the driver can refuse jobs in
ops->run_job()if there are no resources to run it? This would do away with the credit system that has been in place for quite a while. Has this approach been tried in the past?
That seems possible if this is the preferred option. -EAGAIN is the way to do this. I’m open to the idea, but we also need to weigh the cost of converting drivers against the number of changes required.
Partial - reply with catch up the rest later.
Appreciate the feedback.
Matt
Quoted 3,695 lines
Quoted 7 lines
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.Quoted 8 lines
IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.In rust, when you queue work, you have to pass a reference-counted pointer (Arc<T>). We simply never have this problem in a Rust design. If there is work queued, the queue is alive.
By the way, why can’t we simply require synchronous teardowns?
Quoted 96 lines
Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.Cc Boris Brezillon <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Rodrigo Vivi <[email protected]>Cc Thomas Hellström <[email protected]>Cc Christian König <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Maarten Lankhorst <[email protected]>Cc Maxime Ripard <[email protected]>Cc Philipp Stanner <[email protected]>Cc Simona Vetter <[email protected]>Cc Sumit Semwal <[email protected]>Cc Thomas Zimmermann <[email protected]>Signed-off-by Matthew Brost <[email protected]>Assisted-by GitHub Copilot:claude-sonnet-4.6---
drivers/gpu/drm/Kconfig | 4 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/dep/Makefile | 5 + drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++++ drivers/gpu/drm/dep/drm_dep_fence.h | 25 + drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++++++ drivers/gpu/drm/dep/drm_dep_job.h | 13 + drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++ drivers/gpu/drm/dep/drm_dep_queue.h | 31 + include/drm/drm_dep.h | 597 ++++++++++ 10 files changed, 3404 insertions(+)create mode 100644
drivers/gpu/drm/dep/Makefilecreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_job.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_job.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.hcreate mode 100644include/drm/drm_dep.hdrivers/gpu/drm/Kconfig +0
-0@@ -276,6 +276,10 @@ config DRM_SCHEDtristate
depends on DRMdrivers/gpu/drm/Kconfig +4
-0+config DRM_DEP + tristate + depends on DRM +# Separate option as not all DRM drivers use it
configDRM_PANEL_BACKLIGHT_QUIRKS
tristatedrivers/gpu/drm/Makefile +0
-0@@ -173,6 +173,7 @@ obj-y += clients/obj-y += display/
obj-$(CONFIG_DRM_TTM) += ttm/
obj-$(CONFIG_DRM_SCHED) += scheduler/
+obj-$(CONFIG_DRM_DEP) += dep/
obj-$(CONFIG_DRM_RADEON)+= radeon/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/drivers/gpu/drm/dep/Makefile new file +5
-0@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o + +obj-$(CONFIG_DRM_DEP) += drm_dep.odrivers/gpu/drm/dep/drm_dep_fence.c new file +6
-0@@ -0,0 +1,406 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +/** drivers/gpu/drm/dep/drm_dep_fence.c line 16
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+* DOC: DRM dependency fence +* +* Each struct drm_dep_job has an associated struct drm_dep_fence that +* provides a single dma_fence (@finished) signalled when the hardware +* completes the job. +* +* The hardware fence returned by &drm_dep_queue_ops.run_job is stored as +* @parent. @finished is chained to @parent via drm_dep_job_done_cb() and +* is signalled once @parent signals (or immediately if run_job() returns +* NULL or an error). I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Quoted 136 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +136
-0+* +* Drivers should expose @finished as the out-fence for GPU work since it is +* valid from the moment drm_dep_job_arm() returns, whereas the hardware fence +* could be a compound fence, which is disallowed when installed into +* drm_syncobjs or dma-resv. +* +* The fence uses the kernel's inline spinlock (NULL passed to dma_fence_init()) +* so no separate lock allocation is required. +* +* Deadline propagation is supported: if a consumer sets a deadline via +* dma_fence_set_deadline(), it is forwarded to @parent when @parent is set. +* If @parent has not been set yet the deadline is stored in @deadline and +* forwarded at that point. +* +* Memory management: drm_dep_fence objects are allocated with kzalloc() and +* freed via kfree_rcu() once the fence is released, ensuring safety with +* RCU-protected fence accesses. +*/ + +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include "drm_dep_fence.h" + +/** +* DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has been set +* +* Set by the deadline callback on the finished fence to indicate a deadline +* has been set which may need to be propagated to the parent hardware fence. +*/ +#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT (DMA_FENCE_FLAG_USER_BITS + 1) + +/** +* struct drm_dep_fence - fence tracking the completion of a dep job +* +* Contains a single dma_fence (@finished) that is signalled when the +* hardware completes the job. The fence uses the kernel's inline_lock +* (no external spinlock required). +* +* This struct is private to the drm_dep module; external code interacts +* through the accessor functions declared in drm_dep_fence.h. +*/ +struct drm_dep_fence { +/** +* @finished: signalled when the job completes on hardware. +* +* Drivers should use this fence as the out-fence for a job since it +* is available immediately upon drm_dep_job_arm(). +*/ +struct dma_fence finished; + +/** +* @deadline: deadline set on @finished which potentially needs to be +* propagated to @parent. +*/ +ktime_t deadline; + +/** +* @parent: The hardware fence returned by &drm_dep_queue_ops.run_job. +* +* @finished is signaled once @parent is signaled. The initial store is +* performed via smp_store_release to synchronize with deadline handling. +* +* All readers must access this under the fence lock and take a reference to +* it, as @parent is set to NULL under the fence lock when the drm_dep_fence +* signals, and this drop also releases its internal reference. +*/ +struct dma_fence *parent; + +/** +* @q: the queue this fence belongs to. +*/ +struct drm_dep_queue *q; +}; + +static const struct dma_fence_ops drm_dep_fence_ops; + +/** +* to_drm_dep_fence() - cast a dma_fence to its enclosing drm_dep_fence +* @f: dma_fence to cast +* +* Context: No context requirements (inline helper). +* Return: pointer to the enclosing &drm_dep_fence. +*/ +static struct drm_dep_fence *to_drm_dep_fence (struct dma_fence *f) +{ + return container_of(f, struct drm_dep_fence, finished); +} + +/** +* drm_dep_fence_set_parent() - store the hardware fence and propagate +* any deadline +* @dfence: dep fence +* @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or NULL/error +* +* Stores @parent on @dfence under smp_store_release() so that a concurrent +* drm_dep_fence_set_deadline() call sees the parent before checking the +* deadline bit. If a deadline has already been set on @dfence->finished it is +* forwarded to @parent immediately. Does nothing if @parent is NULL or an +* error pointer. +* +* Context: Any context. +*/ +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent) +{ +if (IS_ERR_OR_NULL (parent)) +return ; + +/* +* smp_store_release() to ensure a thread racing us in +* drm_dep_fence_set_deadline() sees the parent set before +* it calls test_bit(HAS_DEADLINE_BIT). +*/ +smp_store_release (&dfence->parent,dma_fence_get (parent)); +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , + &dfence->finished.flags)) +dma_fence_set_deadline (parent, dfence->deadline); +} + +/** +* drm_dep_fence_finished() - signal the finished fence with a result +* @dfence: dep fence to signal +* @result: error code to set, or 0 for success +* +* Sets the fence error to @result if non-zero, then signals +* @dfence->finished. Also removes parent visibility under the fence lock +* and drops the parent reference. Dropping the parent here allows the +* DRM dep fence to be completely decoupled from the DRM dep module. +* +* Context: Any context. +*/ +static void drm_dep_fence_finished (struct drm_dep_fence *dfence,int result) +{ +struct dma_fence *parent; +unsigned long flags; +drivers/gpu/drm/dep/drm_dep_fence.c line 162
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+dma_fence_lock_irqsave (&dfence->finished, flags); +if (result) +dma_fence_set_error (&dfence->finished, result); +dma_fence_signal_locked (&dfence->finished); + parent = dfence->parent; + dfence->parent =NULL ; +dma_fence_unlock_irqrestore (&dfence->finished, flags); + +dma_fence_put (parent); +}We should really try to move away from manual locks and unlocks.
Quoted 236 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +234
-0+ +static const char *drm_dep_fence_get_driver_name (struct dma_fence *fence) +{ +return "drm_dep" ; +} + +static const char *drm_dep_fence_get_timeline_name (struct dma_fence *f) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); + +return dfence->q->name; +} + +/** +* drm_dep_fence_get_parent() - get a reference to the parent hardware fence +* @dfence: dep fence to query +* +* Returns a new reference to @dfence->parent, or NULL if the parent has +* already been cleared (i.e. @dfence->finished has signalled and the parent +* reference was dropped under the fence lock). +* +* Uses smp_load_acquire() to pair with the smp_store_release() in +* drm_dep_fence_set_parent(), ensuring that if we race a concurrent +* drm_dep_fence_set_parent() call we observe the parent pointer only after +* the store is fully visible — before set_parent() tests +* %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT. +* +* Caller must hold the fence lock on @dfence->finished. +* +* Context: Any context, fence lock on @dfence->finished must be held. +* Return: a new reference to the parent fence, or NULL. +*/ +static struct dma_fence *drm_dep_fence_get_parent (struct drm_dep_fence *dfence) +{ +dma_fence_assert_held (&dfence->finished); + +return dma_fence_get (smp_load_acquire (&dfence->parent)); +} + +/** +* drm_dep_fence_set_deadline() - dma_fence_ops deadline callback +* @f: fence on which the deadline is being set +* @deadline: the deadline hint to apply +* +* Stores the earliest deadline under the fence lock, then propagates +* it to the parent hardware fence via smp_load_acquire() to race +* safely with drm_dep_fence_set_parent(). +* +* Context: Any context. +*/ +static void drm_dep_fence_set_deadline (struct dma_fence *f,ktime_t deadline) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); +struct dma_fence *parent; +unsigned long flags; + +dma_fence_lock_irqsave (f, flags); + +/* If we already have an earlier deadline, keep it: */ +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags) && +ktime_before (dfence->deadline, deadline)) { +dma_fence_unlock_irqrestore (f, flags); +return ; + } + + dfence->deadline = deadline; +set_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags); + + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (f, flags); + +if (parent) +dma_fence_set_deadline (parent, deadline); + +dma_fence_put (parent); +} + +static const struct dma_fence_ops drm_dep_fence_ops = { + .get_driver_name = drm_dep_fence_get_driver_name, + .get_timeline_name = drm_dep_fence_get_timeline_name, + .set_deadline = drm_dep_fence_set_deadline, +}; + +/** +* drm_dep_fence_alloc() - allocate a dep fence +* +* Allocates a &drm_dep_fence with kzalloc() without initialising the +* dma_fence. Call drm_dep_fence_init() to fully initialise it. +* +* Context: Process context. +* Return: new &drm_dep_fence on success, NULL on allocation failure. +*/ +struct drm_dep_fence *drm_dep_fence_alloc (void ) +{ + return kzalloc_obj(struct drm_dep_fence); +} + +/** +* drm_dep_fence_init() - initialise the dma_fence inside a dep fence +* @dfence: dep fence to initialise +* @q: queue the owning job belongs to +* +* Initialises @dfence->finished using the context and sequence number from @q. +* Passes NULL as the lock so the fence uses its inline spinlock. +* +* Context: Any context. +*/ +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q) +{ +u32 seq = ++q->fence.seqno; + +/* +* XXX: Inline fence hazard: currently all expected users of DRM dep +* hardware fences have a unique lockdep class. If that ever changes, +* we will need to assign a unique lockdep class here so lockdep knows +* this fence is allowed to nest with driver hardware fences. +*/ + + dfence->q = q; +dma_fence_init (&dfence->finished, &drm_dep_fence_ops, +NULL , q->fence.context, seq); +} + +/** +* drm_dep_fence_cleanup() - release a dep fence at job teardown +* @dfence: dep fence to clean up +* +* Called from drm_dep_job_fini(). If the dep fence was armed (refcount > 0) +* it is released via dma_fence_put() and will be freed by the RCU release +* callback once all waiters have dropped their references. If it was never +* armed it is freed directly with kfree(). +* +* Context: Any context. +*/ +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence) +{ +if (drm_dep_fence_is_armed (dfence)) +dma_fence_put (&dfence->finished); +else +kfree (dfence); +} + +/** +* drm_dep_fence_is_armed() - check whether the fence has been armed +* @dfence: dep fence to check +* +* Returns true if drm_dep_job_arm() has been called, i.e. @dfence->finished +* has been initialised and its reference count is non-zero. Used by +* assertions to enforce correct job lifecycle ordering (arm before push, +* add_dependency before arm). +* +* Context: Any context. +* Return: true if the fence is armed, false otherwise. +*/ +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence) +{ +return !!kref_read (&dfence->finished.refcount); +} + +/** +* drm_dep_fence_is_finished() - test whether the finished fence has signalled +* @dfence: dep fence to check +* +* Uses dma_fence_test_signaled_flag() to read %DMA_FENCE_FLAG_SIGNALED_BIT +* directly without invoking the fence's ->signaled() callback or triggering +* any signalling side-effects. +* +* Context: Any context. +* Return: true if @dfence->finished has been signalled, false otherwise. +*/ +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence) +{ +return dma_fence_test_signaled_flag (&dfence->finished); +} + +/** +* drm_dep_fence_is_complete() - test whether the job has completed +* @dfence: dep fence to check +* +* Takes the fence lock on @dfence->finished and calls +* drm_dep_fence_get_parent() to safely obtain a reference to the parent +* hardware fence — or NULL if the parent has already been cleared after +* signalling. Calls dma_fence_is_signaled() on @parent outside the lock, +* which may invoke the fence's ->signaled() callback and trigger signalling +* side-effects if the fence has completed but the signalled flag has not yet +* been set. The finished fence is tested via dma_fence_test_signaled_flag(), +* without side-effects. +* +* May only be called on a stopped queue (see drm_dep_queue_is_stopped()). +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is complete, false otherwise. +*/ +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence) +{ +struct dma_fence *parent; +unsigned long flags; +bool complete; + +dma_fence_lock_irqsave (&dfence->finished, flags); + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (&dfence->finished, flags); + + complete = (parent &&dma_fence_is_signaled (parent)) || +dma_fence_test_signaled_flag (&dfence->finished); + +dma_fence_put (parent); + +return complete; +} + +/** +* drm_dep_fence_to_dma() - return the finished dma_fence for a dep fence +* @dfence: dep fence to query +* +* No reference is taken; the caller must hold its own reference to the owning +* &drm_dep_job for the duration of the access. +* +* Context: Any context. +* Return: the finished &dma_fence. +*/ +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence) +{ +return &dfence->finished; +} + +/** +* drm_dep_fence_done() - signal the finished fence on job completion +* @dfence: dep fence to signal +* @result: job error code, or 0 on success +* +* Gets a temporary reference to @dfence->finished to guard against a racing +* last-put, signals the fence with @result, then drops the temporary +* reference. Called from drm_dep_job_done() in the queue core when a drivers/gpu/drm/dep/drm_dep_fence.c line 406
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10
-0+* hardware completion callback fires or when run_job() returns immediately. +* +* Context: Any context. +*/ +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result) +{ +dma_fence_get (&dfence->finished); +drm_dep_fence_finished (dfence, result); +dma_fence_put (&dfence->finished); +}Proper refcounting is automated (and enforced) in Rust.
Quoted 111 lines
drivers/gpu/drm/dep/drm_dep_fence.h new file +25
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_FENCE_H_ +#define _DRM_DEP_FENCE_H_ + +#include <linux/dma-fence.h> + +struct drm_dep_fence ; +struct drm_dep_queue ; + +struct drm_dep_fence *drm_dep_fence_alloc (void ); +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q); +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence); +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent); +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result); +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence); +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence); + +#endif /* _DRM_DEP_FENCE_H_ */ drivers/gpu/drm/dep/drm_dep_job.c new file +74
-0@@ -0,0 +1,675 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency job +* +* A struct drm_dep_job represents a single unit of GPU work associated with +* a struct drm_dep_queue. The lifecycle of a job is: +* +* 1. **Allocation**: the driver allocates memory for the job (typically by +* embedding struct drm_dep_job in a larger structure) and calls +* drm_dep_job_init() to initialise it. On success the job holds one +* kref reference and a reference to its queue. +* +* 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(), +* drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(), +* or drm_dep_job_add_implicit_dependencies() to register dma_fence objects +* that must be signalled before the job can run. Duplicate fences from the +* same fence context are deduplicated automatically. +* +* 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence, +* consuming a sequence number from the queue. After arming, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* 4. **Submission**: drm_dep_job_push() submits the job to the queue. The +* queue takes a reference that it holds until the job's finished fence +* signals and the job is freed by the put_job worker. +* +* 5. **Completion**: when the job's hardware work finishes its finished fence +* is signalled and drm_dep_job_put() is called by the queue. The driver +* must release any driver-private resources in &drm_dep_job_ops.release. +* +* Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The +* internal drm_dep_job_fini() tears down the dependency xarray and fence +* objects before the driver's release callback is invoked. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include <drm/drm_file.h> +#include <drm/drm_gem.h> +#include <drm/drm_syncobj.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/** +* drm_dep_job_init() - initialise a dep job +* @job: dep job to initialise drivers/gpu/drm/dep/drm_dep_job.c line 84
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* @args: initialisation arguments +* +* Initialises @job with the queue, ops and credit count from @args. Acquires +* a reference to @args->q via drm_dep_queue_get(); this reference is held for +* the lifetime of the job and released by drm_dep_job_release() when the last +* job reference is dropped. +* +* Resources are released automatically when the last reference is dropped +* via drm_dep_job_put(), which must be called to release the job; drivers +* must not free the job directly. Again, can’t enforce that in C.
Quoted 49 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +49
-0+* +* Context: Process context. Allocates memory with GFP_KERNEL. +* Return: 0 on success, -%EINVAL if credits is 0, +* -%ENOMEM on fence allocation failure. +*/ +int drm_dep_job_init (struct drm_dep_job *job, +const struct drm_dep_job_init_args *args) +{ +if (unlikely (!args->credits)) { +pr_err ("drm_dep: %s: credits cannot be 0\n" , __func__); +return -EINVAL ; + } + +memset (job,0 ,sizeof (*job)); + + job->dfence =drm_dep_fence_alloc (); +if (!job->dfence) +return -ENOMEM ; + + job->ops = args->ops; + job->q =drm_dep_queue_get (args->q); + job->credits = args->credits; + +kref_init (&job->refcount); +xa_init_flags (&job->dependencies,XA_FLAGS_ALLOC ); +INIT_LIST_HEAD (&job->pending_link); + +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_init); + +/** +* drm_dep_job_drop_dependencies() - release all input dependency fences +* @job: dep job whose dependency xarray to drain +* +* Walks @job->dependencies, puts each fence, and destroys the xarray. +* Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel — +* i.e. slots that were pre-allocated but never replaced — are silently +* skipped; the sentinel carries no reference. Called from +* drm_dep_queue_run_job() in process context immediately after +* @ops->run_job() returns, before the final drm_dep_job_put(). Releasing +* dependencies here — while still in process context — avoids calling +* xa_destroy() from IRQ context if the job's last reference is later +* dropped from a dma_fence callback. +* +* Context: Process context. +*/ +void drm_dep_job_drop_dependencies (struct drm_dep_job *job) +{drivers/gpu/drm/dep/drm_dep_job.c line 143
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+struct dma_fence *fence; +unsigned long index; + +xa_for_each (&job->dependencies, index, fence) { +if (unlikely (fence ==DRM_DEP_JOB_FENCE_PREALLOC )) +continue ; +dma_fence_put (fence); + } +xa_destroy (&job->dependencies); +}This is automated in Rust. You also can’t “forget” to call this.
Quoted 25 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +25
-0+ +/** +* drm_dep_job_fini() - clean up a dep job +* @job: dep job to clean up +* +* Cleans up the dep fence and drops the queue reference held by @job. +* +* If the job was never armed (e.g. init failed before drm_dep_job_arm()), +* the dependency xarray is also released here. For armed jobs the xarray +* has already been drained by drm_dep_job_drop_dependencies() in process +* context immediately after run_job(), so it is left untouched to avoid +* calling xa_destroy() from IRQ context. +* +* Warns if @job is still linked on the queue's pending list, which would +* indicate a bug in the teardown ordering. +* +* Context: Any context. +*/ +static void drm_dep_job_fini (struct drm_dep_job *job) +{ +bool armed =drm_dep_fence_is_armed (job->dfence); + +WARN_ON (!list_empty (&job->pending_link)); + +drm_dep_fence_cleanup (job->dfence);drivers/gpu/drm/dep/drm_dep_job.c line 178
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ job->dfence =NULL ; + +/* +* Armed jobs have their dependencies drained by +* drm_dep_job_drop_dependencies() in process context after run_job(). +* Skip here to avoid calling xa_destroy() from IRQ context. +*/ +if (!armed) +drm_dep_job_drop_dependencies (job); +}Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +6
-0+ +/** +* drm_dep_job_get() - acquire a reference to a dep job +* @job: dep job to acquire a reference on, or NULL +* +* Context: Any context. drivers/gpu/drm/dep/drm_dep_job.c line 194
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* Return: @job with an additional reference held, or NULL if @job is NULL. +*/ +struct drm_dep_job *drm_dep_job_get (struct drm_dep_job *job) +{ +if (job) +kref_get (&job->refcount); +return job; +} +EXPORT_SYMBOL (drm_dep_job_get); +Same here.
Quoted 19 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +19
-0+/** +* drm_dep_job_release() - kref release callback for a dep job +* @kref: kref embedded in the dep job +* +* Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Finally, releases the queue reference +* that was acquired by drm_dep_job_init() via drm_dep_queue_put(). The +* queue put is performed last to ensure no queue state is accessed after +* the job memory is freed. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the +* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +static void drm_dep_job_release (struct kref *kref) +{ +struct drm_dep_job *job = + container_of(kref, struct drm_dep_job, refcount); +struct drm_dep_queue *q = job->q;drivers/gpu/drm/dep/drm_dep_job.c line 223
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ +drm_dep_job_fini (job); + +if (job->ops && job->ops->release) + job->ops->release (job); +else +kfree (job); + +drm_dep_queue_put (q); +}Same here.
Quoted 9 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +9
-0+ +/** +* drm_dep_job_put() - release a reference to a dep job +* @job: dep job to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Does nothing if @job is NULL. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the drivers/gpu/drm/dep/drm_dep_job.c line 242
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +void drm_dep_job_put (struct drm_dep_job *job) +{ +if (job) +kref_put (&job->refcount, drm_dep_job_release); +} +EXPORT_SYMBOL (drm_dep_job_put); +Same here.
Quoted 65 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +65
-0+/** +* drm_dep_job_arm() - arm a dep job for submission +* @job: dep job to arm +* +* Initialises the finished fence on @job->dfence, assigning +* it a sequence number from the job's queue. Must be called after +* drm_dep_job_init() and before drm_dep_job_push(). Once armed, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* Begins the DMA fence signalling path via dma_fence_begin_signalling(). +* After this point, memory allocations that could trigger reclaim are +* forbidden; lockdep enforces this. arm() must always be paired with +* drm_dep_job_push(); lockdep also enforces this pairing. +* +* Warns if the job has already been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_arm (struct drm_dep_job *job) +{ +drm_dep_queue_push_job_begin (job->q); +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +drm_dep_fence_init (job->dfence, job->q); + job->signalling_cookie =dma_fence_begin_signalling (); +} +EXPORT_SYMBOL (drm_dep_job_arm); + +/** +* drm_dep_job_push() - submit a job to its queue for execution +* @job: dep job to push +* +* Submits @job to the queue it was initialised with. Must be called after +* drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue, +* held until the queue is fully done with it. The reference is released +* directly in the finished-fence dma_fence callback for queues with +* %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run +* from hardirq context), or via the put_job work item on the submit +* workqueue otherwise. +* +* Ends the DMA fence signalling path begun by drm_dep_job_arm() via +* dma_fence_end_signalling(). This must be paired with arm(); lockdep +* enforces the pairing. +* +* Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for +* @job exactly once, even if the queue is killed or torn down before the +* job reaches the head of the queue. Drivers can use this guarantee to +* perform bookkeeping cleanup; the actual backend operation should be +* skipped when drm_dep_queue_is_killed() returns true. +* +* If the queue does not support the bypass path, the job is pushed directly +* onto the SPSC submission queue via drm_dep_queue_push_job() without holding +* @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either +* run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or +* enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item. +* +* Warns if the job has not been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_push (struct drm_dep_job *job)drivers/gpu/drm/dep/drm_dep_job.c line 317
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+{ +struct drm_dep_queue *q = job->q; + +WARN_ON (!drm_dep_fence_is_armed (job->dfence)); + +drm_dep_job_get (job); + +if (!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED )) { +drm_dep_queue_push_job (q, job); +dma_fence_end_signalling (job->signalling_cookie);Signaling is enforced in a more thorough way in Rust. I’ll expand on this later in this patch.
Quoted 167 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +166
-0+drm_dep_queue_push_job_end (job->q); +return ; + } + + scoped_guard(mutex, &q->sched.lock) { +if (drm_dep_queue_can_job_bypass (q, job)) +drm_dep_queue_run_job (q, job); +else +drm_dep_queue_push_job (q, job); + } + +dma_fence_end_signalling (job->signalling_cookie); +drm_dep_queue_push_job_end (job->q); +} +EXPORT_SYMBOL (drm_dep_job_push); + +/** +* drm_dep_job_add_dependency() - adds the fence as a job dependency +* @job: dep job to add the dependencies to +* @fence: the dma_fence to add to the list of dependencies, or +* %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later. +* +* Note that @fence is consumed in both the success and error cases (except +* when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference). +* +* Signalled fences and fences belonging to the same queue as @job (i.e. where +* fence->context matches the queue's finished fence context) are silently +* dropped; the job need not wait on its own queue's output. +* +* Warns if the job has already been armed (dependencies must be added before +* drm_dep_job_arm()). +* +* **Pre-allocation pattern** +* +* When multiple jobs across different queues must be prepared and submitted +* together in a single atomic commit — for example, where job A's finished +* fence is an input dependency of job B — all jobs must be armed and pushed +* within a single dma_fence_begin_signalling() / dma_fence_end_signalling() +* region. Once that region has started no memory allocation is permitted. +* +* To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation +* phase (before arming any job, while GFP_KERNEL allocation is still allowed) +* to pre-allocate a slot in @job->dependencies. The slot index assigned by +* the underlying xarray must be tracked by the caller separately (e.g. it is +* always index 0 when the dependency array is empty, as Xe relies on). +* After all jobs have been armed and the finished fences are available, call +* drm_dep_job_replace_dependency() with that index and the real fence. +* drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be +* called from atomic or signalling context. +* +* The sentinel slot is never skipped by the signalled-fence fast-path, +* ensuring a slot is always allocated even when the real fence is not yet +* known. +* +* **Example: bind job feeding TLB invalidation jobs** +* +* Consider a GPU with separate queues for page-table bind operations and for +* TLB invalidation. A single atomic commit must: +* +* 1. Run a bind job that modifies page tables. +* 2. Run one TLB-invalidation job per MMU that depends on the bind +* completing, so stale translations are flushed before the engines +* continue. +* +* Because all jobs must be armed and pushed inside a signalling region (where +* GFP_KERNEL is forbidden), pre-allocate slots before entering the region:: +* +* // Phase 1 — process context, GFP_KERNEL allowed +* drm_dep_job_init(bind_job, bind_queue, ops); +* for_each_mmu(mmu) { +* drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops); +* // Pre-allocate slot at index 0; real fence not available yet +* drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC); +* } +* +* // Phase 2 — inside signalling region, no GFP_KERNEL +* dma_fence_begin_signalling(); +* drm_dep_job_arm(bind_job); +* for_each_mmu(mmu) { +* // Swap sentinel for bind job's finished fence +* drm_dep_job_replace_dependency(tlb_job[mmu], 0, +* dma_fence_get(bind_job->finished)); +* drm_dep_job_arm(tlb_job[mmu]); +* } +* drm_dep_job_push(bind_job); +* for_each_mmu(mmu) +* drm_dep_job_push(tlb_job[mmu]); +* dma_fence_end_signalling(); +* +* Context: Process context. May allocate memory with GFP_KERNEL. +* Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on +* success, else 0 on success, or a negative error code. +*/ +int drm_dep_job_add_dependency (struct drm_dep_job *job,struct dma_fence *fence) +{ +struct drm_dep_queue *q = job->q; +struct dma_fence *entry; +unsigned long index; +u32 id =0 ; +int ret; + +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +might_alloc (GFP_KERNEL ); + +if (!fence) +return 0 ; + +if (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) + goto add_fence; + +/* +* Ignore signalled fences or fences from our own queue — finished +* fences use q->fence.context. +*/ +if (dma_fence_test_signaled_flag (fence) || + fence->context == q->fence.context) { +dma_fence_put (fence); +return 0 ; + } + +/* Deduplicate if we already depend on a fence from the same context. +* This lets the size of the array of deps scale with the number of +* engines involved, rather than the number of BOs. +*/ +xa_for_each (&job->dependencies, index, entry) { +if (entry ==DRM_DEP_JOB_FENCE_PREALLOC || + entry->context != fence->context) +continue ; + +if (dma_fence_is_later (fence, entry)) { +dma_fence_put (entry); +xa_store (&job->dependencies, index, fence,GFP_KERNEL ); + }else { +dma_fence_put (fence); + } +return 0 ; + } + +add_fence: + ret =xa_alloc (&job->dependencies, &id, fence, xa_limit_32b, +GFP_KERNEL ); +if (ret !=0 ) { +if (fence !=DRM_DEP_JOB_FENCE_PREALLOC ) +dma_fence_put (fence); +return ret; + } + +return (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) ? id :0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_dependency); + +/** +* drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot +* @job: dep job to update +* @index: xarray index of the slot to replace, as returned when the sentinel +* was originally inserted via drm_dep_job_add_dependency() +* @fence: the real dma_fence to store; its reference is always consumed +* +* Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in +* @job->dependencies with @fence. The slot must have been pre-allocated by +* passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the +* existing entry is asserted to be the sentinel. +* +* This is the second half of the pre-allocation pattern described in +* drm_dep_job_add_dependency(). It is intended to be called inside a +* dma_fence_begin_signalling() / dma_fence_end_signalling() region where drivers/gpu/drm/dep/drm_dep_job.c line 493
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* memory allocation with GFP_KERNEL is forbidden. It uses GFP_NOWAIT +* internally so it is safe to call from atomic or signalling context, but +* since the slot has been pre-allocated no actual memory allocation occurs. +* +* If @fence is already signalled the slot is erased rather than storing a +* redundant dependency. The successful store is asserted — if the store +* fails it indicates a programming error (slot index out of range or +* concurrent modification). +* +* Must be called before drm_dep_job_arm(). @fence is consumed in all cases. Can’t enforce this in C. Also, how is the fence “consumed” ? You can’t enforce that the user can’t access the fence anymore after this function returns, like we can do at compile time in Rust.
Quoted 163 lines
+ *
+ * Context: Any context. DMA fence signaling path.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +160
-0+void drm_dep_job_replace_dependency (struct drm_dep_job *job,u32 index, +struct dma_fence *fence) +{ +WARN_ON (xa_load (&job->dependencies, index) != +DRM_DEP_JOB_FENCE_PREALLOC ); + +if (dma_fence_test_signaled_flag (fence)) { +xa_erase (&job->dependencies, index); +dma_fence_put (fence); +return ; + } + +if (WARN_ON (xa_is_err (xa_store (&job->dependencies, index, fence, +GFP_NOWAIT )))) { +dma_fence_put (fence); +return ; + } +} +EXPORT_SYMBOL (drm_dep_job_replace_dependency); + +/** +* drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a +* job dependency +* @job: dep job to add the dependencies to +* @file: drm file private pointer +* @handle: syncobj handle to lookup +* @point: timeline point +* +* This adds the fence matching the given syncobj to @job. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_syncobj_dependency (struct drm_dep_job *job, +struct drm_file *file,u32 handle, +u32 point) +{ +struct dma_fence *fence; +int ret; + + ret =drm_syncobj_find_fence (file, handle, point,0 , &fence); +if (ret) +return ret; + +return drm_dep_job_add_dependency (job, fence); +} +EXPORT_SYMBOL (drm_dep_job_add_syncobj_dependency); + +/** +* drm_dep_job_add_resv_dependencies() - add all fences from the resv to the job +* @job: dep job to add the dependencies to +* @resv: the dma_resv object to get the fences from +* @usage: the dma_resv_usage to use to filter the fences +* +* This adds all fences matching the given usage from @resv to @job. +* Must be called with the @resv lock held. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_resv_dependencies (struct drm_dep_job *job, +struct dma_resv *resv, +enum dma_resv_usage usage) +{ +struct dma_resv_iter cursor; +struct dma_fence *fence; +int ret; + +dma_resv_assert_held (resv); + +dma_resv_for_each_fence (&cursor, resv, usage, fence) { +/* +* As drm_dep_job_add_dependency always consumes the fence +* reference (even when it fails), and dma_resv_for_each_fence +* is not obtaining one, we need to grab one before calling. +*/ + ret =drm_dep_job_add_dependency (job,dma_fence_get (fence)); +if (ret) +return ret; + } +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_resv_dependencies); + +/** +* drm_dep_job_add_implicit_dependencies() - adds implicit dependencies +* as job dependencies +* @job: dep job to add the dependencies to +* @obj: the gem object to add new dependencies from. +* @write: whether the job might write the object (so we need to depend on +* shared fences in the reservation object). +* +* This should be called after drm_gem_lock_reservations() on your array of +* GEM objects used in the job but before updating the reservations with your +* own fences. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_implicit_dependencies (struct drm_dep_job *job, +struct drm_gem_object *obj, +bool write) +{ +return drm_dep_job_add_resv_dependencies (job, obj->resv, +dma_resv_usage_rw (write)); +} +EXPORT_SYMBOL (drm_dep_job_add_implicit_dependencies); + +/** +* drm_dep_job_is_signaled() - check whether a dep job has completed +* @job: dep job to check +* +* Determines whether @job has signalled. The queue should be stopped before +* calling this to obtain a stable snapshot of state. Both the parent hardware +* fence and the finished software fence are checked. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is signalled, false otherwise. +*/ +bool drm_dep_job_is_signaled (struct drm_dep_job *job) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q)); +return drm_dep_fence_is_complete (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_signaled); + +/** +* drm_dep_job_is_finished() - test whether a dep job's finished fence has signalled +* @job: dep job to check +* +* Tests whether the job's software finished fence has been signalled, using +* dma_fence_test_signaled_flag() to avoid any signalling side-effects. Unlike +* drm_dep_job_is_signaled(), this does not require the queue to be stopped and +* does not check the parent hardware fence — it is a lightweight test of the +* finished fence only. +* +* Context: Any context. +* Return: true if the job's finished fence has been signalled, false otherwise. +*/ +bool drm_dep_job_is_finished (struct drm_dep_job *job) +{ +return drm_dep_fence_is_finished (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_finished); + +/** +* drm_dep_job_invalidate_job() - increment the invalidation count for a job +* @job: dep job to invalidate +* @threshold: threshold above which the job is considered invalidated +* +* Increments @job->invalidate_count and returns true if it exceeds @threshold, +* indicating the job should be considered hung and discarded. The queue must +* be stopped before calling this function. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if @job->invalidate_count exceeds @threshold, false otherwise. +*/ +bool drm_dep_job_invalidate_job (struct drm_dep_job *job,int threshold) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q));drivers/gpu/drm/dep/drm_dep_job.c line 666
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+return ++job->invalidate_count > threshold; +} +EXPORT_SYMBOL (drm_dep_job_invalidate_job); + +/** +* drm_dep_job_finished_fence() - return the finished fence for a job +* @job: dep job to query +* +* No reference is taken on the returned fence; the caller must hold its own +* reference to @job for the duration of any access. Can’t enforce this in C.
Quoted 111 lines
+ *
+ * Context: Any context.
+ * Return: the finished&dma_fencefor @job.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +5
-0+struct dma_fence *drm_dep_job_finished_fence (struct drm_dep_job *job) +{ +return drm_dep_fence_to_dma (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_finished_fence);drivers/gpu/drm/dep/drm_dep_job.h new file +13
-0@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_JOB_H_ +#define _DRM_DEP_JOB_H_ + +struct drm_dep_queue ; + +void drm_dep_job_drop_dependencies (struct drm_dep_job *job); + +#endif /* _DRM_DEP_JOB_H_ */ drivers/gpu/drm/dep/drm_dep_queue.c new file +77
-0@@ -0,0 +1,1647 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency queue +* +* The drm_dep subsystem provides a lightweight GPU submission queue that +* combines the roles of drm_gpu_scheduler and drm_sched_entity into a +* single object (struct drm_dep_queue). Each queue owns its own ordered +* submit workqueue, timeout workqueue, and TDR delayed-work. +* +* **Job lifecycle** +* +* 1. Allocate and initialise a job with drm_dep_job_init(). +* 2. Add dependency fences with drm_dep_job_add_dependency() and friends. +* 3. Arm the job with drm_dep_job_arm() to obtain its out-fences. +* 4. Submit with drm_dep_job_push(). +* +* **Submission paths** +* +* drm_dep_job_push() decides between two paths under @q->sched.lock: +* +* - **Bypass path** (drm_dep_queue_can_job_bypass()): if +* %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not stopped, +* the SPSC queue is empty, the job has no dependency fences, and credits +* are available, the job is submitted inline on the calling thread without +* touching the submit workqueue. +* +* - **Queued path** (drm_dep_queue_push_job()): the job is pushed onto an +* SPSC queue and the run_job worker is kicked. The run_job worker pops the +* job, resolves any remaining dependency fences (installing wakeup +* callbacks for unresolved ones), and calls drm_dep_queue_run_job(). +* +* **Running a job** +* +* drm_dep_queue_run_job() accounts credits, appends the job to the pending +* list (starting the TDR timer only when the list was previously empty), +* calls @ops->run_job(), stores the returned hardware fence as the parent +* of the job's dep fence, then installs a callback on it. When the hardware +* fence fires (or the job completes synchronously), drm_dep_job_done() +* signals the finished fence, returns credits, and kicks the put_job worker +* to free the job. +* +* **Timeout detection and recovery (TDR)** +* +* A delayed work item fires when a job on the pending list takes longer than +* @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on the +* returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB). +* drm_dep_queue_trigger_timeout() forces the timer to fire immediately (without +* changing the stored timeout), for example during device teardown. +* +* **Reference counting** +* +* Jobs and queues are both reference counted. drivers/gpu/drm/dep/drm_dep_queue.c line 87
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* +* A job holds a reference to its queue from drm_dep_job_init() until +* drm_dep_job_put() drops the job's last reference and its release callback +* runs. This ensures the queue remains valid for the entire lifetime of any +* job that was submitted to it. +* +* The queue holds its own reference to a job for as long as the job is +* internally tracked: from the moment the job is added to the pending list +* in drm_dep_queue_run_job() until drm_dep_job_done() kicks the put_job +* worker, which calls drm_dep_job_put() to release that reference. Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
Quoted 7 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +7
-0+* +* **Hazard: use-after-free from within a worker** +* +* Because a job holds a queue reference, drm_dep_job_put() dropping the last +* job reference will also drop a queue reference via the job's release path. +* If that happens to be the last queue reference, drm_dep_queue_fini() can be +* called, which queues @q->free_work on dep_free_wq and returns immediately. drivers/gpu/drm/dep/drm_dep_queue.c line 104
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* free_work calls disable_work_sync() / disable_delayed_work_sync() on the +* queue's own workers before destroying its workqueues, so in practice a +* running worker always completes before the queue memory is freed. +* +* However, there is a secondary hazard: a worker can be queued while the +* queue is in a "zombie" state — refcount has already reached zero and async +* teardown is in flight, but the work item has not yet been disabled by +* free_work. To guard against this every worker uses +* drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero +* the worker bails immediately without touching the queue state. Again, this problem is gone in Rust.
Quoted 34 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +34
-0+* +* Because all actual teardown (disable_*_sync, destroy_workqueue) runs on +* dep_free_wq — which is independent of the queue's own submit/timeout +* workqueues — there is no deadlock risk. Each queue holds a drm_dev_get() +* reference on its owning &drm_device, which is released as the last step of +* teardown. This ensures the driver module cannot be unloaded while any queue +* is still alive. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/module.h> +#include <linux/overflow.h> +#include <linux/slab.h> +#include <linux/wait.h> +#include <linux/workqueue.h> +#include <drm/drm_dep.h> +#include <drm/drm_drv.h> +#include <drm/drm_print.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/* +* Dedicated workqueue for deferred drm_dep_queue teardown. Using a +* module-private WQ instead of system_percpu_wq keeps teardown isolated +* from unrelated kernel subsystems. +*/ +static struct workqueue_struct *dep_free_wq; + +/** +* drm_dep_queue_flags_set() - set a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to set (one of &enum drm_dep_queue_flags) drivers/gpu/drm/dep/drm_dep_queue.c line 148
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* +* Sets @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_set (struct drm_dep_queue *q, +enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock);We can enforce this in Rust at compile-time. The code does not compile if the lock is not taken. Same here and everywhere else where the sched lock has to be taken.
Quoted 124 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +124
-0+ q->sched.flags |= flag; +} + +/** +* drm_dep_queue_flags_clear() - clear a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to clear (one of &enum drm_dep_queue_flags) +* +* Clears @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_clear (struct drm_dep_queue *q, +enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock); + q->sched.flags &= ~flag; +} + +/** +* drm_dep_queue_has_credits() - check whether the queue has enough credits +* @q: dep queue +* @job: job requesting credits +* +* Checks whether the queue has enough available credits to dispatch +* @job. If @job->credits exceeds the queue's credit limit, it is +* clamped with a WARN. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if available credits >= @job->credits, false otherwise. +*/ +static bool drm_dep_queue_has_credits (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +u32 available; + +lockdep_assert_held (&q->sched.lock); + +if (job->credits > q->credit.limit) { +drm_warn (q->drm, +"Jobs may not exceed the credit limit, truncate.\n" ); + job->credits = q->credit.limit; + } + +WARN_ON (check_sub_overflow (q->credit.limit, +atomic_read (&q->credit.count), + &available)); + +return available >= job->credits; +} + +/** +* drm_dep_queue_run_job_queue() - kick the run-job worker +* @q: dep queue +* +* Queues @q->sched.run_job on @q->sched.submit_wq unless the queue is stopped +* or the job queue is empty. The empty-queue check avoids queueing a work item +* that would immediately return with nothing to do. +* +* Context: Any context. +*/ +static void drm_dep_queue_run_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q) &&spsc_queue_count (&q->job.queue)) +queue_work (q->sched.submit_wq, &q->sched.run_job); +} + +/** +* drm_dep_queue_put_job_queue() - kick the put-job worker +* @q: dep queue +* +* Queues @q->sched.put_job on @q->sched.submit_wq unless the queue +* is stopped. +* +* Context: Any context. +*/ +static void drm_dep_queue_put_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q)) +queue_work (q->sched.submit_wq, &q->sched.put_job); +} + +/** +* drm_queue_start_timeout() - arm or re-arm the TDR delayed work +* @q: dep queue +* +* Arms the TDR delayed work with @q->job.timeout. No-op if +* @q->ops->timedout_job is NULL, the timeout is MAX_SCHEDULE_TIMEOUT, +* or the pending list is empty. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_queue_start_timeout (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->job.lock); + +if (!q->ops->timedout_job || + q->job.timeout ==MAX_SCHEDULE_TIMEOUT || +list_empty (&q->job.pending)) +return ; + +mod_delayed_work (q->sched.timeout_wq, &q->sched.tdr, q->job.timeout); +} + +/** +* drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock +* @q: dep queue +* +* Acquires @q->job.lock with irq and calls +* drm_queue_start_timeout(). +* +* Context: Process context (workqueue). +*/ +static void drm_queue_start_timeout_unlocked (struct drm_dep_queue *q) +{ +guard (spinlock_irq)(&q->job.lock); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_remove_dependency() - clear the active dependency and wake +* the run-job worker +* @q: dep queue drivers/gpu/drm/dep/drm_dep_queue.c line 282
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @f: the dependency fence being removed +* +* Stores @f into @q->dep.removed_fence via smp_store_release() so that the +* run-job worker can drop the reference to it in drm_dep_queue_is_ready(), +* paired with smp_load_acquire(). Clears @q->dep.fence and kicks the +* run-job worker. +* +* The fence reference is not dropped here; it is deferred to the run-job +* worker via @q->dep.removed_fence to keep this path suitable dma_fence +* callback removal in drm_dep_queue_kill(). This is a comment in C, but in Rust this is encoded directly in the type system.
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_remove_dependency (struct drm_dep_queue *q, +struct dma_fence *f) +{ +/* removed_fence must be visible to the reader before &q->dep.fence */ +smp_store_release (&q->dep.removed_fence, f); + +WRITE_ONCE (q->dep.fence,NULL ); +drm_dep_queue_run_job_queue (q); +}drivers/gpu/drm/dep/drm_dep_queue.c line 304
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +/** +* drm_dep_queue_wakeup() - dma_fence callback to wake the run-job worker +* @f: the signalled dependency fence +* @cb: callback embedded in the dep queue +* +* Called from dma_fence_signal() when the active dependency fence signals. +* Delegates to drm_dep_queue_remove_dependency() to clear @q->dep.fence and +* kick the run-job worker. The fence reference is not dropped here; it is +* deferred to the run-job worker via @q->dep.removed_fence. Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +6
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_wakeup (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_queue *q =drivers/gpu/drm/dep/drm_dep_queue.c line 320
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ container_of(cb, struct drm_dep_queue, dep.cb); + +drm_dep_queue_remove_dependency (q, f); +} + +/** +* drm_dep_queue_is_ready() - check whether the queue has a dispatchable job +* @q: dep queue +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. Can’t call this in Rust if the lock is not taken.
Quoted 158 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +158
-0+* Return: true if SPSC queue non-empty and no dep fence pending, +* false otherwise. +*/ +static bool drm_dep_queue_is_ready (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->sched.lock); + +if (!spsc_queue_count (&q->job.queue)) +return false; + +if (READ_ONCE (q->dep.fence)) +return false; + +/* Paired with smp_store_release in drm_dep_queue_remove_dependency() */ +dma_fence_put (smp_load_acquire (&q->dep.removed_fence)); + + q->dep.removed_fence =NULL ; + +return true; +} + +/** +* drm_dep_queue_is_killed() - check whether a dep queue has been killed +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_killed (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_KILLED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_killed); + +/** +* drm_dep_queue_is_initialized() - check whether a dep queue has been initialized +* @q: dep queue to check +* +* A queue is considered initialized once its ops pointer has been set by a +* successful call to drm_dep_queue_init(). Drivers that embed a +* &drm_dep_queue inside a larger structure may call this before attempting any +* other queue operation to confirm that initialization has taken place. +* drm_dep_queue_put() must be called if this function returns true to drop the +* initialization reference from drm_dep_queue_init(). +* +* Return: true if @q has been initialized, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_initialized (struct drm_dep_queue *q) +{ +return !!q->ops; +} +EXPORT_SYMBOL (drm_dep_queue_is_initialized); + +/** +* drm_dep_queue_set_stopped() - pre-mark a queue as stopped before first use +* @q: dep queue to mark +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going through the +* normal drm_dep_queue_stop() path. This is only valid during the driver-side +* queue initialisation sequence — i.e. after drm_dep_queue_init() returns but +* before the queue is made visible to other threads (e.g. before it is added +* to any lookup structures). Using this after the queue is live is a driver +* bug; use drm_dep_queue_stop() instead. +* +* Context: Process context, queue not yet visible to other threads. +*/ +void drm_dep_queue_set_stopped (struct drm_dep_queue *q) +{ + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_STOPPED ; +} +EXPORT_SYMBOL (drm_dep_queue_set_stopped); + +/** +* drm_dep_queue_refcount() - read the current reference count of a queue +* @q: dep queue to query +* +* Returns the instantaneous kref value. The count may change immediately +* after this call; callers must not make safety decisions based solely on +* the returned value. Intended for diagnostic snapshots and debugfs output. +* +* Context: Any context. +* Return: current reference count. +*/ +unsigned int drm_dep_queue_refcount (const struct drm_dep_queue *q) +{ +return kref_read (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_refcount); + +/** +* drm_dep_queue_timeout() - read the per-job TDR timeout for a queue +* @q: dep queue to query +* +* Returns the per-job timeout in jiffies as set at init time. +* %MAX_SCHEDULE_TIMEOUT means no timeout is configured. +* +* Context: Any context. +* Return: timeout in jiffies. +*/ +long drm_dep_queue_timeout (const struct drm_dep_queue *q) +{ +return q->job.timeout; +} +EXPORT_SYMBOL (drm_dep_queue_timeout); + +/** +* drm_dep_queue_is_job_put_irq_safe() - test whether job-put from IRQ is allowed +* @q: dep queue +* +* Context: Any context. +* Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set, +* false otherwise. +*/ +static bool drm_dep_queue_is_job_put_irq_safe (const struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE ); +} + +/** +* drm_dep_queue_job_dependency() - get next unresolved dep fence +* @q: dep queue +* @job: job whose dependencies to advance +* +* Returns NULL immediately if the queue has been killed via +* drm_dep_queue_kill(), bypassing all dependency waits so that jobs +* drain through run_job as quickly as possible. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next unresolved &dma_fence with a new reference, or NULL +* when all dependencies have been consumed (or the queue is killed). +*/ +static struct dma_fence * +drm_dep_queue_job_dependency (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *f; + +lockdep_assert_held (&q->sched.lock); + +if (drm_dep_queue_is_killed (q)) +return NULL ; + + f =xa_load (&job->dependencies, job->last_dependency); +if (f) { + job->last_dependency++; +if (WARN_ON (DRM_DEP_JOB_FENCE_PREALLOC == f)) +return dma_fence_get_stub (); +return dma_fence_get (f); + } + +return NULL ; +} + +/** +* drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence drivers/gpu/drm/dep/drm_dep_queue.c line 488
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @q: dep queue +* @job: job whose dependency fence is stored in @q->dep.fence +* +* Installs a wakeup callback on @q->dep.fence. Returns true if the +* callback was installed (the queue must wait), false if the fence is +* already signalled or is a self-fence from the same queue context. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if callback installed, false if fence already done. +*/ In Rust, we can encode the signaling paths with a “token type”. So any sections that are part of the signaling path can simply take this token as an argument. This type also enforces that
end_signaling()is called automatically when it goes out of scope.By the way, we can easily offer an irq handler type where we enforce this:
fn
handle_threaded_irq(&self, device: &Device<Bound>)-> IrqReturn {
let_annotation=DmaFenceSignallingAnnotation::new(); // Callsbegin_signaling()self.driver.handle_threaded_irq(device)//
end_signaling()is called here automatically.
}Same for workqueues:
fn
work_fn(&self, device: &Device<Bound>){
let_annotation=DmaFenceSignallingAnnotation::new(); // Callsbegin_signaling()self.driver.work_fn(device)//
end_signaling()is called here automatically.
}This is not Rust-specific, of course, but it is more ergonomic to write in Rust.
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+static bool drm_dep_queue_add_dep_cb (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *fence = q->dep.fence; + +lockdep_assert_held (&q->sched.lock); + +if (WARN_ON (fence->context == q->fence.context)) { +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; +return false; + }drivers/gpu/drm/dep/drm_dep_queue.c line 510
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +if (!dma_fence_add_callback (q->dep.fence, &q->dep.cb, + drm_dep_queue_wakeup)) +return true; + +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; + +return false; +}In rust we can enforce that all callbacks take a reference to the fence automatically. If the callback is “forgotten” in a buggy path, it is automatically removed, and the fence is automatically signaled with
-ECANCELED.Quoted 36 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +36
-0+ +/** +* drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC queue +* @q: dep queue +* +* Peeks at the head of the SPSC queue and drains all resolved +* dependencies. If a dependency is still pending, installs a wakeup +* callback and returns NULL. On success pops the job and returns it. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next dispatchable job, or NULL if a dep is still pending. +*/ +static struct drm_dep_job *drm_dep_queue_pop_job (struct drm_dep_queue *q) +{ +struct spsc_node *node; +struct drm_dep_job *job; + +lockdep_assert_held (&q->sched.lock); + + node =spsc_queue_peek (&q->job.queue); +if (!node) +return NULL ; + + job = container_of(node, struct drm_dep_job, queue_node); + +while ((q->dep.fence =drm_dep_queue_job_dependency (q, job))) { +if (drm_dep_queue_add_dep_cb (q, job)) +return NULL ; + } + +spsc_queue_pop (&q->job.queue); + +return job; +} + +/* drivers/gpu/drm/dep/drm_dep_queue.c line 556
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* drm_dep_queue_get_unless_zero() - try to acquire a queue reference +* +* Workers use this instead of drm_dep_queue_get() to guard against the zombie +* state: the queue's refcount has already reached zero (async teardown is in +* flight) but a work item was queued before free_work had a chance to cancel +* it. If kref_get_unless_zero() fails the caller must bail immediately. +* +* Context: Any context. +* Returns true if the reference was acquired, false if the queue is zombie. +*/ Again, this function is totally gone in Rust.
Quoted 1,021 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +1021
-0+bool drm_dep_queue_get_unless_zero (struct drm_dep_queue *q) +{ +return kref_get_unless_zero (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_get_unless_zero); + +/** +* drm_dep_queue_run_job_work() - run-job worker +* @work: work item embedded in the dep queue +* +* Acquires @q->sched.lock, checks stopped state, queue readiness and +* available credits, pops the next job via drm_dep_queue_pop_job(), +* dispatches it via drm_dep_queue_run_job(), then re-kicks itself. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_run_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.run_job); +struct spsc_node *node; +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +mutex_lock (&q->sched.lock); + +if (drm_dep_queue_is_stopped (q)) + goto put_queue; + +if (!drm_dep_queue_is_ready (q)) + goto put_queue; + +/* Peek to check credits before committing to pop and dep resolution */ + node =spsc_queue_peek (&q->job.queue); +if (!node) + goto put_queue; + + job = container_of(node, struct drm_dep_job, queue_node); +if (!drm_dep_queue_has_credits (q, job)) + goto put_queue; + + job =drm_dep_queue_pop_job (q); +if (!job) + goto put_queue; + +drm_dep_queue_run_job (q, job); +drm_dep_queue_run_job_queue (q); + +put_queue: +mutex_unlock (&q->sched.lock); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/* +* drm_dep_queue_remove_job() - unlink a job from the pending list and reset TDR +* @q: dep queue owning @job +* @job: job to remove +* +* Splices @job out of @q->job.pending, cancels any pending TDR delayed work, +* and arms the timeout for the new list head (if any). +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_remove_job (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->job.lock); + +list_del_init (&job->pending_link); +cancel_delayed_work (&q->sched.tdr); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_get_finished_job() - dequeue a finished job +* @q: dep queue +* +* Under @q->job.lock checks the head of the pending list for a +* finished dep fence. If found, removes the job from the list, +* cancels the TDR, and re-arms it for the new head. +* +* Context: Process context (workqueue). DMA fence signaling path. +* Return: the finished &drm_dep_job, or NULL if none is ready. +*/ +static struct drm_dep_job * +drm_dep_queue_get_finished_job (struct drm_dep_queue *q) +{ +struct drm_dep_job *job; + +guard (spinlock_irq)(&q->job.lock); + + job = list_first_entry_or_null(&q->job.pending, struct drm_dep_job, + pending_link); +if (job &&drm_dep_fence_is_finished (job->dfence)) +drm_dep_queue_remove_job (q, job); +else + job =NULL ; + +return job; +} + +/** +* drm_dep_queue_put_job_work() - put-job worker +* @work: work item embedded in the dep queue +* +* Drains all finished jobs by calling drm_dep_job_put() in a loop, +* then kicks the run-job worker. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because workqueue is shared with other items in the fence signaling path. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_put_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.put_job); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +while ((job =drm_dep_queue_get_finished_job (q))) +drm_dep_job_put (job); + +drm_dep_queue_run_job_queue (q); + +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_queue_tdr_work() - TDR worker +* @work: work item embedded in the delayed TDR work +* +* Removes the head job from the pending list under @q->job.lock, +* asserts @q->ops->timedout_job is non-NULL, calls it outside the lock, +* requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the +* queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and always +* restarts the TDR timer after handling the job (unless @q is stopping). +* Any other return value triggers a WARN. +* +* The TDR is never armed when @q->ops->timedout_job is NULL, so firing +* this worker without a timedout_job callback is a driver bug. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because timedout_job() is expected to signal the guilty job's fence as part +* of reset. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_tdr_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.tdr.work); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + + scoped_guard(spinlock_irq, &q->job.lock) { + job =list_first_entry_or_null (&q->job.pending, + struct drm_dep_job, + pending_link); +if (job) +/* +* Remove from pending so it cannot be freed +* concurrently by drm_dep_queue_get_finished_job() or +* .drm_dep_job_done(). +*/ +list_del_init (&job->pending_link); + } + +if (job) { +enum drm_dep_timedout_stat status; + +if (WARN_ON (!q->ops->timedout_job)) { +drm_dep_job_put (job); + goto out; + } + + status = q->ops->timedout_job (job); + +switch (status) { +case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB : + scoped_guard(spinlock_irq, &q->job.lock) +list_add (&job->pending_link, &q->job.pending); +drm_dep_queue_put_job_queue (q); +break ; +case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED : +drm_dep_job_put (job); +break ; +default : +WARN_ON ("invalid drm_dep_timedout_stat" ); +break ; + } + } + +out: +drm_queue_start_timeout_unlocked (q); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue +* @name: name for the workqueue +* @flags: DRM_DEP_QUEUE_FLAGS_* flags +* +* Allocates an ordered workqueue for job submission with %WQ_MEM_RECLAIM and +* %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use from +* memory reclaim context and properly annotated for lockdep taint tracking. +* Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When +* CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct * +drm_dep_alloc_submit_wq (const char *name,enum drm_dep_queue_flags flags) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +if (flags &DRM_DEP_QUEUE_FLAGS_HIGHPRI ) + wq_flags |=WQ_HIGHPRI ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_submit_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue +* @name: name for the workqueue +* +* Allocates an ordered workqueue for timeout detection and recovery with +* %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring consistent taint +* annotation with the submit workqueue. When CONFIG_LOCKDEP is enabled, uses +* a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct *drm_dep_alloc_timeout_wq (const char *name) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_timeout_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_queue_init() - initialize a dep queue +* @q: dep queue to initialize +* @args: initialization arguments +* +* Initializes all fields of @q from @args. If @args->submit_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds one kref +* reference and drm_dep_queue_put() must be called to drop this reference +* (i.e., drivers cannot directly free the queue). +* +* When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against the +* fs_reclaim pseudo-lock so that lockdep can detect any lock ordering +* inversion between @sched.lock and memory reclaim. +* +* Return: 0 on success, %-EINVAL when @args->credit_limit is zero, @args->ops +* is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when +* @args->submit_wq or @args->timeout_wq is non-NULL but was not allocated with +* %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails. +* +* Context: Process context. May allocate memory and create workqueues. +*/ +int drm_dep_queue_init (struct drm_dep_queue *q, +const struct drm_dep_queue_init_args *args) +{ +if (!args->credit_limit || !args->drm || !args->ops || + !args->ops->run_job) +return -EINVAL ; + +if (args->submit_wq && !workqueue_is_reclaim_annotated (args->submit_wq)) +return -EINVAL ; + +if (args->timeout_wq && + !workqueue_is_reclaim_annotated (args->timeout_wq)) +return -EINVAL ; + +memset (q,0 ,sizeof (*q)); + + q->name = args->name; + q->drm = args->drm; + q->credit.limit = args->credit_limit; + q->job.timeout = args->timeout ? args->timeout :MAX_SCHEDULE_TIMEOUT ; + +init_rcu_head (&q->rcu); +INIT_LIST_HEAD (&q->job.pending); +spin_lock_init (&q->job.lock); +spsc_queue_init (&q->job.queue); + +mutex_init (&q->sched.lock); +if (IS_ENABLED (CONFIG_LOCKDEP )) { +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&q->sched.lock); +fs_reclaim_release (GFP_KERNEL ); + } + +if (args->submit_wq) { + q->sched.submit_wq = args->submit_wq; + }else { + q->sched.submit_wq =drm_dep_alloc_submit_wq (args->name ?:"drm_dep" , + args->flags); +if (!q->sched.submit_wq) +return -ENOMEM ; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ; + } + +if (args->timeout_wq) { + q->sched.timeout_wq = args->timeout_wq; + }else { + q->sched.timeout_wq =drm_dep_alloc_timeout_wq (args->name ?:"drm_dep" ); +if (!q->sched.timeout_wq) + goto err_submit_wq; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ; + } + + q->sched.flags |= args->flags & + ~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ | +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ); + +INIT_DELAYED_WORK (&q->sched.tdr, drm_dep_queue_tdr_work); +INIT_WORK (&q->sched.run_job, drm_dep_queue_run_job_work); +INIT_WORK (&q->sched.put_job, drm_dep_queue_put_job_work); + + q->fence.context =dma_fence_context_alloc (1 ); + +kref_init (&q->refcount); + q->ops = args->ops; +drm_dev_get (q->drm); + +return 0 ; + +err_submit_wq: +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); +mutex_destroy (&q->sched.lock); + +return -ENOMEM ; +} +EXPORT_SYMBOL (drm_dep_queue_init); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* drm_dep_queue_push_job_begin() - mark the start of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the start of drm_dep_job_arm() and warns if the push context is +* already owned by another task, which would indicate concurrent arm/push on +* the same queue. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner); + q->job.push.owner = current; +} + +/** +* drm_dep_queue_push_job_end() - mark the end of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the end of drm_dep_job_push() and warns if the push context is not +* owned by the current task, which would indicate a mismatched begin/end pair +* or a push from the wrong thread. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner != current); + q->job.push.owner =NULL ; +} +#endif + +/** +* drm_dep_queue_assert_teardown_invariants() - assert teardown invariants +* @q: dep queue being torn down +* +* Warns if the pending-job list, the SPSC submission queue, or the credit +* counter is non-zero when called, or if the queue still has a non-zero +* reference count. +* +* Context: Any context. +*/ +static void drm_dep_queue_assert_teardown_invariants (struct drm_dep_queue *q) +{ +WARN_ON (!list_empty (&q->job.pending)); +WARN_ON (spsc_queue_count (&q->job.queue)); +WARN_ON (atomic_read (&q->credit.count)); +WARN_ON (drm_dep_queue_refcount (q)); +} + +/** +* drm_dep_queue_release() - final internal cleanup of a dep queue +* @q: dep queue to clean up +* +* Asserts teardown invariants and destroys internal resources allocated by +* drm_dep_queue_init() that cannot be torn down earlier in the teardown +* sequence. Currently this destroys @q->sched.lock. +* +* Drivers that implement &drm_dep_queue_ops.release **must** call this +* function after removing @q from any internal bookkeeping (e.g. lookup +* tables or lists) but before freeing the memory that contains @q. When +* &drm_dep_queue_ops.release is NULL, drm_dep follows the default teardown +* path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_release (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); +mutex_destroy (&q->sched.lock); +} +EXPORT_SYMBOL (drm_dep_queue_release); + +/** +* drm_dep_queue_free() - final cleanup of a dep queue +* @q: dep queue to free +* +* Invokes &drm_dep_queue_ops.release if set, in which case the driver is +* responsible for calling drm_dep_queue_release() and freeing @q itself. +* If &drm_dep_queue_ops.release is NULL, calls drm_dep_queue_release() +* and then frees @q with kfree_rcu(). +* +* In either case, releases the drm_dev_get() reference taken at init time +* via drm_dev_put(), allowing the owning &drm_device to be unloaded once +* all queues have been freed. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free (struct drm_dep_queue *q) +{ +struct drm_device *drm = q->drm; + +if (q->ops->release) { + q->ops->release (q); + }else { +drm_dep_queue_release (q); +kfree_rcu (q, rcu); + } +drm_dev_put (drm); +} + +/** +* drm_dep_queue_free_work() - deferred queue teardown worker +* @work: free_work item embedded in the dep queue +* +* Runs on dep_free_wq. Disables all work items synchronously +* (preventing re-queue and waiting for in-flight instances), +* destroys any owned workqueues, then calls drm_dep_queue_free(). +* Running on dep_free_wq ensures destroy_workqueue() is never +* called from within one of the queue's own workers (deadlock) +* and disable_*_sync() cannot deadlock either. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, free_work); + +drm_dep_queue_assert_teardown_invariants (q); + +disable_delayed_work_sync (&q->sched.tdr); +disable_work_sync (&q->sched.run_job); +disable_work_sync (&q->sched.put_job); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ) +destroy_workqueue (q->sched.timeout_wq); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); + +drm_dep_queue_free (q); +} + +/** +* drm_dep_queue_fini() - tear down a dep queue +* @q: dep queue to tear down +* +* Asserts teardown invariants and nitiates teardown of @q by queuing the +* deferred free work onto tht module-private dep_free_wq workqueue. The work +* item disables any pending TDR and run/put-job work synchronously, destroys +* any workqueues that were allocated by drm_dep_queue_init(), and then releases +* the queue memory. +* +* Running teardown from dep_free_wq ensures that destroy_workqueue() is never +* called from within one of the queue's own workers (e.g. via +* drm_dep_queue_put()), which would deadlock. +* +* Drivers can wait for all outstanding deferred work to complete by waiting +* for the last drm_dev_put() reference on their &drm_device, which is +* released as the final step of each queue's teardown. +* +* Drivers that implement &drm_dep_queue_ops.fini **must** call this +* function after removing @q from any device bookkeeping but before freeing the +* memory that contains @q. When &drm_dep_queue_ops.fini is NULL, drm_dep +* follows the default teardown path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_fini (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); + +INIT_WORK (&q->free_work, drm_dep_queue_free_work); +queue_work (dep_free_wq, &q->free_work); +} +EXPORT_SYMBOL (drm_dep_queue_fini); + +/** +* drm_dep_queue_get() - acquire a reference to a dep queue +* @q: dep queue to acquire a reference on, or NULL +* +* Return: @q with an additional reference held, or NULL if @q is NULL. +* +* Context: Any context. +*/ +struct drm_dep_queue *drm_dep_queue_get (struct drm_dep_queue *q) +{ +if (q) +kref_get (&q->refcount); +return q; +} +EXPORT_SYMBOL (drm_dep_queue_get); + +/** +* __drm_dep_queue_release() - kref release callback for a dep queue +* @kref: kref embedded in the dep queue +* +* Calls &drm_dep_queue_ops.fini if set, otherwise calls +* drm_dep_queue_fini() to initiate deferred teardown. +* +* Context: Any context. +*/ +static void __drm_dep_queue_release (struct kref *kref) +{ +struct drm_dep_queue *q = + container_of(kref, struct drm_dep_queue, refcount); + +if (q->ops->fini) + q->ops->fini (q); +else +drm_dep_queue_fini (q); +} + +/** +* drm_dep_queue_put() - release a reference to a dep queue +* @q: dep queue to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_queue_ops.fini if set, +* otherwise calls drm_dep_queue_fini(). Final memory release is handled by +* &drm_dep_queue_ops.release (which must call drm_dep_queue_release()) if set, +* or drm_dep_queue_release() followed by kfree_rcu() otherwise. +* Does nothing if @q is NULL. +* +* Context: Any context. +*/ +void drm_dep_queue_put (struct drm_dep_queue *q) +{ +if (q) +kref_put (&q->refcount, __drm_dep_queue_release); +} +EXPORT_SYMBOL (drm_dep_queue_put); + +/** +* drm_dep_queue_stop() - stop a dep queue from processing new jobs +* @q: dep queue to stop +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from finished +* fenced signaling context. Then cancels any in-flight run_job and put_job work +* items. Once stopped, the bypass path and the submit workqueue will not +* dispatch further jobs nor will any jobs be removed from the pending list. +* Call drm_dep_queue_start() to resume processing. +* +* Context: Process context. Waits for in-flight workers to complete. +*/ +void drm_dep_queue_stop (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +cancel_work_sync (&q->sched.run_job); +cancel_work_sync (&q->sched.put_job); +} +EXPORT_SYMBOL (drm_dep_queue_stop); + +/** +* drm_dep_queue_start() - resume a stopped dep queue +* @q: dep queue to start +* +* Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from IRQ +* context. Then re-queues the run_job and put_job work items so that any jobs +* pending since the queue was stopped are processed. Must only be called after +* drm_dep_queue_stop(). +* +* Context: Process context. +*/ +void drm_dep_queue_start (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_clear (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +drm_dep_queue_run_job_queue (q); +drm_dep_queue_put_job_queue (q); +} +EXPORT_SYMBOL (drm_dep_queue_start); + +/** +* drm_dep_queue_trigger_timeout() - trigger the TDR immediately for +* all pending jobs +* @q: dep queue to trigger timeout on +* +* Sets @q->job.timeout to 1 and arms the TDR delayed work with a one-jiffy +* delay, causing it to fire almost immediately without hot-spinning at zero +* delay. This is used to force-expire any pendind jobs on the queue, for +* example when the device is being torn down or has encountered an +* unrecoverable error. +* +* It is suggested that when this function is used, the first timedout_job call +* causes the driver to kick the queue off the hardware and signal all pending +* job fences. Subsequent calls continue to signal all pending job fences. +* +* Has no effect if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_trigger_timeout (struct drm_dep_queue *q) +{ +guard (spinlock_irqsave)(&q->job.lock); + q->job.timeout =1 ; +drm_queue_start_timeout (q); +} +EXPORT_SYMBOL (drm_dep_queue_trigger_timeout); + +/** +* drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait +* for it to finish +* @q: dep queue whose TDR to cancel +* +* Cancels the TDR delayed work item if it has not yet started, and waits for +* it to complete if it is already running. After this call returns, the TDR +* worker is guaranteed not to be executing and will not fire again until +* explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by a new +* job being submitted). +* +* Useful during error recovery or queue teardown when the caller needs to +* know that no timeout handling races with its own reset logic. +* +* Context: Process context. May sleep waiting for the TDR worker to finish. +*/ +void drm_dep_queue_cancel_tdr_sync (struct drm_dep_queue *q) +{ +cancel_delayed_work_sync (&q->sched.tdr); +} +EXPORT_SYMBOL (drm_dep_queue_cancel_tdr_sync); + +/** +* drm_dep_queue_resume_timeout() - restart the TDR timer with the +* configured timeout +* @q: dep queue to resume the timeout for +* +* Restarts the TDR delayed work using @q->job.timeout. Called after device +* recovery to give pending jobs a fresh full timeout window. Has no effect +* if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_resume_timeout (struct drm_dep_queue *q) +{ +drm_queue_start_timeout_unlocked (q); +} +EXPORT_SYMBOL (drm_dep_queue_resume_timeout); + +/** +* drm_dep_queue_is_stopped() - check whether a dep queue is stopped +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_stopped (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_STOPPED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_stopped); + +/** +* drm_dep_queue_kill() - kill a dep queue and flush all pending jobs +* @q: dep queue to kill +* +* Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock. If a +* dependency fence is currently being waited on, its callback is removed and +* the run-job worker is kicked immediately so that the blocked job drains +* without waiting. +* +* Once killed, drm_dep_queue_job_dependency() returns NULL for all jobs, +* bypassing dependency waits so that every queued job drains through +* &drm_dep_queue_ops.run_job without blocking. +* +* The &drm_dep_queue_ops.run_job callback is guaranteed to be called for every +* job that was pushed before or after drm_dep_queue_kill(), even during queue +* teardown. Drivers should use this guarantee to perform any necessary +* bookkeeping cleanup without executing the actual backend operation when the +* queue is killed. +* +* Unlike drm_dep_queue_stop(), killing is one-way: there is no corresponding +* start function. +* +* **Driver safety requirement** +* +* drm_dep_queue_kill() must only be called once the driver can guarantee that +* no job in the queue will touch memory associated with any of its fences +* (i.e., the queue has been removed from the device and will never be put back +* on). +* +* Context: Process context. +*/ +void drm_dep_queue_kill (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { +struct dma_fence *fence; + +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_KILLED ); + +/* +* Holding &q->sched.lock guarantees that the run-job work item +* cannot drop its reference to q->dep.fence concurrently, so +* reading q->dep.fence here is safe. +*/ + fence =READ_ONCE (q->dep.fence); +if (fence &&dma_fence_remove_callback (fence, &q->dep.cb)) +drm_dep_queue_remove_dependency (q, fence); + } +} +EXPORT_SYMBOL (drm_dep_queue_kill); + +/** +* drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Drivers may use this to queue their own work items alongside the queue's +* internal run-job and put-job workers — for example to process incoming +* messages in the same serialisation domain. +* +* Prefer drm_dep_queue_work_enqueue() when the only need is to enqueue a +* work item, as it additionally checks the stopped state. Use this accessor +* when the workqueue itself is required (e.g. for alloc_ordered_workqueue +* replacement or drain_workqueue calls). +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for job submission. +*/ +struct workqueue_struct *drm_dep_queue_submit_wq (struct drm_dep_queue *q) +{ +return q->sched.submit_wq; +} +EXPORT_SYMBOL (drm_dep_queue_submit_wq); + +/** +* drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Returns the workqueue used by @q to run TDR (timeout detection and recovery) +* work. Drivers may use this to queue their own timeout-domain work items, or +* to call drain_workqueue() when tearing down and needing to ensure all pending +* timeout callbacks have completed before proceeding. +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for TDR work. +*/ +struct workqueue_struct *drm_dep_queue_timeout_wq (struct drm_dep_queue *q) +{ +return q->sched.timeout_wq; +} +EXPORT_SYMBOL (drm_dep_queue_timeout_wq); + +/** +* drm_dep_queue_work_enqueue() - queue work on the dep queue's submit workqueue +* @q: dep queue to enqueue work on +* @work: work item to enqueue +* +* Queues @work on @q->sched.submit_wq if the queue is not stopped. This +* allows drivers to schedule custom work items that run serialised with the +* queue's own run-job and put-job workers. +* +* Return: true if the work was queued, false if the queue is stopped or the +* work item was already pending. +* +* Context: Any context. +*/ +bool drm_dep_queue_work_enqueue (struct drm_dep_queue *q, +struct work_struct *work) +{ +if (drm_dep_queue_is_stopped (q)) +return false; + +return queue_work (q->sched.submit_wq, work); +} +EXPORT_SYMBOL (drm_dep_queue_work_enqueue); + +/** +* drm_dep_queue_can_job_bypass() - test whether a job can skip the SPSC queue +* @q: dep queue +* @job: job to test +* +* A job may bypass the submit workqueue and run inline on the calling thread +* if all of the following hold: +* +* - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue +* - the queue is not stopped +* - the SPSC submission queue is empty (no other jobs waiting) +* - the queue has enough credits for @job +* - @job has no unresolved dependency fences +* +* Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). +* Return: true if the job may be run inline, false otherwise. +*/ +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->sched.lock); + +return q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED && + !drm_dep_queue_is_stopped (q) && + !spsc_queue_count (&q->job.queue) && +drm_dep_queue_has_credits (q, job) && +xa_empty (&job->dependencies); +} + +/** +* drm_dep_job_done() - mark a job as complete +* @job: the job that finished +* @result: error code to propagate, or 0 for success +* +* Subtracts @job->credits from the queue credit counter, then signals the +* job's dep fence with @result. +* +* When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), a +* temporary extra reference is taken on @job before signalling the fence. +* This prevents a concurrent put-job worker — which may be woken by timeouts or +* queue starting — from freeing the job while this function still holds a +* pointer to it. The extra reference is released at the end of the function. +* +* After signalling, the IRQ-safe path removes the job from the pending list +* under @q->job.lock, provided the queue is not stopped. Removal is skipped +* when the queue is stopped so that drm_dep_queue_for_each_pending_job() can +* iterate the list without racing with the completion path. On successful +* removal, kicks the run-job worker so the next queued job can be dispatched +* immediately, then drops the job reference. If the job was already removed +* by TDR, or removal was skipped because the queue is stopped, kicks the +* put-job worker instead to allow the deferred put to complete. +* +* Context: Any context. +*/ +static void drm_dep_job_done (struct drm_dep_job *job,int result) +{ +struct drm_dep_queue *q = job->q; +bool irq_safe =drm_dep_queue_is_job_put_irq_safe (q), removed = false; + +/* +* Local ref to ensure the put worker—which may be woken by external +* forces (TDR, driver-side queue starting)—doesn't free the job behind +* this function's back after drm_dep_fence_done() while it is still on +* the pending list. +*/ +if (irq_safe) +drm_dep_job_get (job); + +atomic_sub (job->credits, &q->credit.count); +drm_dep_fence_done (job->dfence, result); + +/* Only safe to touch job after fence signal if we have a local ref. */ + +if (irq_safe) { + scoped_guard(spinlock_irqsave, &q->job.lock) { + removed = !list_empty (&job->pending_link) && + !drm_dep_queue_is_stopped (q); + +/* Guard against TDR operating on job */ +if (removed) +drm_dep_queue_remove_job (q, job); + } + } + +if (removed) { +drm_dep_queue_run_job_queue (q); +drm_dep_job_put (job); + }else { +drm_dep_queue_put_job_queue (q); + } + +if (irq_safe) +drm_dep_job_put (job); +} + +/** +* drm_dep_job_done_cb() - dma_fence callback to complete a job +* @f: the hardware fence that signalled +* @cb: fence callback embedded in the dep job +* +* Extracts the job from @cb and calls drm_dep_job_done() with +* @f->error as the result. +* +* Context: Any context, but with IRQ disabled. May not sleep. +*/ +static void drm_dep_job_done_cb (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_job *job = container_of(cb, struct drm_dep_job, cb); + +drm_dep_job_done (job, f->error); +} + +/** +* drm_dep_queue_run_job() - submit a job to hardware and set up +* completion tracking +* @q: dep queue +* @job: job to run +* +* Accounts @job->credits against the queue, appends the job to the pending +* list, then calls @q->ops->run_job(). The TDR timer is started only when +* @job is the first entry on the pending list; subsequent jobs added while +* a TDR is already in flight do not reset the timer (which would otherwise +* extend the deadline for the already-running head job). Stores the returned +* hardware fence as the parent of the job's dep fence, then installs +* drm_dep_job_done_cb() on it. If the hardware fence is already signalled +* (%-ENOENT from dma_fence_add_callback()) or run_job() returns NULL/error, +* the job is completed immediately. Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). DMA fence +* signaling path. +*/ +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +struct dma_fence *fence; +int r; + +lockdep_assert_held (&q->sched.lock); + +drm_dep_job_get (job); +atomic_add (job->credits, &q->credit.count); + + scoped_guard(spinlock_irq, &q->job.lock) { +bool first =list_empty (&q->job.pending); + +list_add_tail (&job->pending_link, &q->job.pending); +if (first) +drm_queue_start_timeout (q); + } + + fence = q->ops->run_job (job); +drm_dep_fence_set_parent (job->dfence, fence); + +if (!IS_ERR_OR_NULL (fence)) { + r =dma_fence_add_callback (fence, &job->cb, + drm_dep_job_done_cb); +if (r == -ENOENT ) +drm_dep_job_done (job, fence->error); +else if (r) +drm_err (q->drm,"fence add callback failed (%d)\n" , r); +dma_fence_put (fence);drivers/gpu/drm/dep/drm_dep_queue.c line 1587
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ }else { +drm_dep_job_done (job,IS_ERR (fence) ?PTR_ERR (fence) :0 ); + } + +/* +* Drop all input dependency fences now, in process context, before the +* final job put. Once the job is on the pending list its last reference +* may be dropped from a dma_fence callback (IRQ context), where calling +* xa_destroy() would be unsafe. +*/ I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Quoted 201 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +60
-0+drm_dep_job_drop_dependencies (job); +drm_dep_job_put (job); +} + +/** +* drm_dep_queue_push_job() - enqueue a job on the SPSC submission queue +* @q: dep queue +* @job: job to push +* +* Pushes @job onto the SPSC queue. If the queue was previously empty +* (i.e. this is the first pending job), kicks the run_job worker so it +* processes the job promptly without waiting for the next wakeup. +* May be called with or without @q->sched.lock held. +* +* Context: Any context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +/* +* spsc_queue_push() returns true if the queue was previously empty, +* i.e. this is the first pending job. Kick the run_job worker so it +* picks it up without waiting for the next wakeup. +*/ +if (spsc_queue_push (&q->job.queue, &job->queue_node)) +drm_dep_queue_run_job_queue (q); +} + +/** +* drm_dep_init() - module initialiser +* +* Allocates the module-private dep_free_wq unbound workqueue used for +* deferred queue teardown. +* +* Return: 0 on success, %-ENOMEM if workqueue allocation fails. +*/ +static int __init drm_dep_init (void ) +{ + dep_free_wq =alloc_workqueue ("drm_dep_free" ,WQ_UNBOUND ,0 ); +if (!dep_free_wq) +return -ENOMEM ; + +return 0 ; +} + +/** +* drm_dep_exit() - module exit +* +* Destroys the module-private dep_free_wq workqueue. +*/ +static void __exit drm_dep_exit(void) +{ +destroy_workqueue (dep_free_wq); + dep_free_wq =NULL ; +} + +module_init (drm_dep_init); +module_exit (drm_dep_exit); + +MODULE_DESCRIPTION ("DRM dependency queue" ); +MODULE_LICENSE ("Dual MIT/GPL" );drivers/gpu/drm/dep/drm_dep_queue.h new file +31
-0@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_QUEUE_H_ +#define _DRM_DEP_QUEUE_H_ + +#include <linux/types.h> + +struct drm_dep_job ; +struct drm_dep_queue ; + +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job); +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job); +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q); +void drm_dep_queue_push_job_end (struct drm_dep_queue *q); +#else +static inline void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +} +static inline void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +} +#endif + +#endif /* _DRM_DEP_QUEUE_H_ */ include/drm/drm_dep.h new file +98
-0@@ -0,0 +1,597 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_H_ +#define _DRM_DEP_H_ + +#include <drm/spsc_queue.h> +#include <linux/dma-fence.h> +#include <linux/xarray.h> +#include <linux/workqueue.h> + +enum dma_resv_usage ; +struct dma_resv ; +struct drm_dep_fence ; +struct drm_dep_job ; +struct drm_dep_queue ; +struct drm_file ; +struct drm_gem_object ; + +/** +* enum drm_dep_timedout_stat - return value of &drm_dep_queue_ops.timedout_job +* @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's finished +* fence during reset; drm_dep may safely drop its reference to the job. +* @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; reinsert the +* job at the head of the pending list so it can complete normally. +*/ +enum drm_dep_timedout_stat { +DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED , +DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB , +}; + +/** +* struct drm_dep_queue_ops - driver callbacks for a dep queue +*/ +struct drm_dep_queue_ops { +/** +* @run_job: submit the job to hardware. Returns the hardware completion +* fence (with a reference held for the scheduler), or NULL/ERR_PTR on +* synchronous completion or error. +*/ +struct dma_fence *(*run_job)(struct drm_dep_job *job); + +/** +* @timedout_job: called when the TDR fires for the head job. Must stop +* the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if the +* job's fence was signalled during reset, or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or +* signalling was otherwise delayed, and the job should be re-inserted +* at the head of the pending list. Any other value triggers a WARN. +*/ +enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job); + +/** +* @release: called when the last kref on the queue is dropped and +* drm_dep_queue_fini() has completed. The driver is responsible for +* removing @q from any internal bookkeeping, calling +* drm_dep_queue_release(), and then freeing the memory containing @q +* (e.g. via kfree_rcu() using @q->rcu). If NULL, drm_dep calls +* drm_dep_queue_release() and frees @q automatically via kfree_rcu(). +* Use this when the queue is embedded in a larger structure. +*/ +void (*release)(struct drm_dep_queue *q); + +/** +* @fini: if set, called instead of drm_dep_queue_fini() when the last +* kref is dropped. The driver is responsible for calling +* drm_dep_queue_fini() itself after it is done with the queue. Use this +* when additional teardown logic must run before fini (e.g., cleanup +* firmware resources associated with the queue). +*/ +void (*fini)(struct drm_dep_queue *q); +}; + +/** +* enum drm_dep_queue_flags - flags for &drm_dep_queue and +* &drm_dep_queue_init_args include/drm/drm_dep.h line 108
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* +* Flags are divided into three categories: +* +* - **Private static**: set internally at init time and never changed. +* Drivers must not read or write these. +* %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ, +* %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ. +* +* - **Public dynamic**: toggled at runtime by drivers via accessors. +* Any modification must be performed under &drm_dep_queue.sched.lock. Can’t enforce that in C.
include/drm/drm_dep.h line 115
Quoted 8 lines
include/drm/drm_dep.h new file +7
-0+* Accessor functions provide unstable reads. +* %DRM_DEP_QUEUE_FLAGS_STOPPED, +* %DRM_DEP_QUEUE_FLAGS_KILLED. +* +* - **Public static**: supplied by the driver in +* &drm_dep_queue_init_args.flags at queue creation time and not modified +* thereafter. Same here.
Quoted 47 lines
+ * %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED, + * %DRM_DEP_QUEUE_FLAGS_HIGHPRI, + * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE.+ *
+ * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit workqueue was
+ * allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout workqueue
+ * was allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is stopped and+ * will not dispatch new jobs or remove jobs from the pending list, dropping + * the drm_dep-owned reference. Set by drm_dep_queue_stop(), cleared by + * drm_dep_queue_start(). + * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been killed + * via drm_dep_queue_kill(). Any active dependency wait is cancelled + * immediately. Jobs continue to flow through run_job for bookkeeping + * cleanup, but dependency waiting is skipped so that queued work drains + * as quickly as possible. + * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue supports + * the bypass path where eligible jobs skip the SPSC queue and run inline. + * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit workqueue owned + * by the queue is created with %WQ_HIGHPRI, causing run-job and put-job + * workers to execute at elevated priority. Only privileged clients (e.g. + * drivers managing time-critical or real-time GPU contexts) should request + * this flag; granting it to unprivileged userspace would allow priority + * inversion attacks. + * @drm_dep_queue_init_args.submit_wq is provided. + * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set, + * drm_dep_job_done() may be called from hardirq context (e.g. from a + * hardware-signalled dma_fence callback). drm_dep_job_done() will directly + * dequeue the job and call drm_dep_job_put() without deferring to a + * workqueue. The driver's &drm_dep_job_ops.release callback must therefore + * be safe to invoke from IRQ context.+ */
include/drm/drm_dep.h new file +11
-0+enum drm_dep_queue_flags { +DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ =BIT (0 ), +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ =BIT (1 ), +DRM_DEP_QUEUE_FLAGS_STOPPED =BIT (2 ), +DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED =BIT (3 ), +DRM_DEP_QUEUE_FLAGS_HIGHPRI =BIT (4 ), +DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE =BIT (5 ), +DRM_DEP_QUEUE_FLAGS_KILLED =BIT (6 ), +}; + +/** include/drm/drm_dep.h line 171
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* struct drm_dep_queue - a dependency-tracked GPU submission queue +* +* Combines the role of &drm_gpu_scheduler and &drm_sched_entity into a single +* object. Each queue owns a submit workqueue (or borrows one), a timeout +* workqueue, an SPSC submission queue, and a pending-job list used for TDR. +* +* Initialise with drm_dep_queue_init(), tear down with drm_dep_queue_fini(). +* Reference counted via drm_dep_queue_get() / drm_dep_queue_put(). +* +* All fields are **opaque to drivers**. Do not read or write any field Can’t enforce this in C.
Quoted 23 lines
+ * directly; use the provided helper functions instead. The sole exception
+ * is @rcu, which drivers may pass tokfree_rcu()when the queue is embedded
+ * inside a larger driver-managed structure and the &drm_dep_queue_ops.release
+ * vfunc performs an RCU-deferred free.+ */
+structdrm_dep_queue{
+ /** @ops: driver callbacks, set at init time. */
+ const structdrm_dep_queue_ops*ops;
+ /** @name: human-readable name used for workqueue and fence naming. */
+ const char *name;
+ /** @drm: owning DRM device; adrm_dev_get()reference is held for the
+ * lifetime of the queue to prevent module unload while queues are live.
+ */
+ structdrm_device*drm;
+ /** @refcount: reference count; use drm_dep_queue_get/put(). */
+ struct kref refcount;
+ /**
+ * @free_work: deferred teardown work queued unconditionally by
+ *drm_dep_queue_fini()onto the module-privatedep_free_wq. The work
+ * item disables pending workers synchronously and destroys any owned
+ * workqueues before releasing the queue memory and dropping the
+ *drm_dev_get()reference. Running ondep_free_wqensuresQuoted 8 lines
+ *
destroy_workqueue()is never called from within one of the queue's
+ * own workers.
+ */
+ structwork_structfree_work;
+ /**
+ * @rcu: RCU head for deferred freeing.
+ *
+ * This is the **only** field drivers may access directly. When theWe can enforce this in Rust at compile time.
Quoted 88 lines
+ * queue is embedded in a larger structure, implement
+ * &drm_dep_queue_ops.release, calldrm_dep_queue_release()to destroy
+ * internal resources, then pass this field tokfree_rcu()so that any
+ * in-flight RCU readers referencing the queue'sdma_fencetimeline name
+ * complete before the memory is returned. All other fields must be
+ * accessed through the provided helpers.
+ */
+ structrcu_headrcu;include/drm/drm_dep.h new file +80
-0+ +/** @sched: scheduling and workqueue state. */ +struct { +/** @sched.submit_wq: ordered workqueue for run/put-job work. */ +struct workqueue_struct *submit_wq; +/** @sched.timeout_wq: workqueue for the TDR delayed work. */ +struct workqueue_struct *timeout_wq; +/** +* @sched.run_job: work item that dispatches the next queued +* job. +*/ +struct work_struct run_job; +/** @sched.put_job: work item that frees finished jobs. */ +struct work_struct put_job; +/** @sched.tdr: delayed work item for timeout/reset (TDR). */ +struct delayed_work tdr; +/** +* @sched.lock: mutex serialising job dispatch, bypass +* decisions, stop/start, and flag updates. +*/ +struct mutex lock; +/** +* @sched.flags: bitmask of &enum drm_dep_queue_flags. +* Any modification after drm_dep_queue_init() must be +* performed under @sched.lock. +*/ +enum drm_dep_queue_flags flags; + } sched; + +/** @job: pending-job tracking state. */ +struct { +/** +* @job.pending: list of jobs that have been dispatched to +* hardware and not yet freed. Protected by @job.lock. +*/ +struct list_head pending; +/** +* @job.queue: SPSC queue of jobs waiting to be dispatched. +* Producers push via drm_dep_queue_push_job(); the run_job +* work item pops from the consumer side. +*/ +struct spsc_queue queue; +/** +* @job.lock: spinlock protecting @job.pending, TDR start, and +* the %DRM_DEP_QUEUE_FLAGS_STOPPED flag. Always acquired with +* irqsave (spin_lock_irqsave / spin_unlock_irqrestore) to +* support %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE queues where +* drm_dep_job_done() may run from hardirq context. +*/ +spinlock_t lock; +/** Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
include/drm/drm_dep.h line 270
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+* @job.timeout: per-job TDR timeout in jiffies. +* %MAX_SCHEDULE_TIMEOUT means no timeout. +*/ +long timeout; +#if IS_ENABLED(CONFIG_PROVE_LOCKING) + /** + * @job.push: lockdep annotation tracking the arm-to-push + * critical section. + */ + struct {We can enforce this in Rust automatically.
include/drm/drm_dep.h
Quoted 73 lines
+ */
+ structdma_fence*removed_fence;
+ /** @dep.cb: callback installed on @dep.fence. */
+ structdma_fence_cbcb;
+ } dep;include/drm/drm_dep.h new file +68
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
And if they don’t?
By the way, we can also enforce this in Rust.
include/drm/drm_dep.h
Quoted 88 lines
+ *
drm_dep_job_put()will callkfree()on the job directly.
+ */
+ void (*release)(structdrm_dep_job*job);include/drm/drm_dep.h new file +85
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
So if you’re not careful, you have just introduced a race :/
include/drm/drm_dep.h
Quoted 89 lines
+ *
+ * **Constraint: only fromsubmit_wqworker context**
+ *
+ * This guard must only be acquired from a work item running on the queue's
+ * submit workqueue (@q->sched.submit_wq) by drivers.
+ *
+ * Context: Process context only; must be called fromsubmit_wqwork by
+ * drivers.
+ */include/drm/drm_dep.h new file +80
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Not enforceable in C.
Quoted 51 lines
include/drm/drm_dep.h new file +48
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1By the way:
I invite you to have a look at this implementation [0]. It currently works in real hardware i.e.: our downstream "Tyr" driver for Arm Mali is using that at the moment. It is a mere prototype that we’ve put together to test different approaches, so it’s not meant to be a “solution” at all. It’s a mere data point for further discussion.
Philip Stanner is working on this “Job Queue” concept too, but from an upstream perspective.
[0]: https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61
Part of the diff is not shown here. View the raw message.
Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 5 lines · Matthew Brost
You can do RAII in C - see
cleanup.h. Clear object lifetimes and ownership are what is important. Disciplined coding is the only to do this regardless of language. RAII doesn't help with help with bad object models / ownership / lifetime models either.
"Ownership", "lifetimes" and being "disciplined" *is* what Rust helps with. That is the whole point (even if there are other advantages).
Yes, the cleanup attribute is nice, but even the whole CLASS thing is meant to simplify code. Simplifying code does reduce bugs in general, but it doesn't solve anything fundamental. Even if we had C++ and full-fledged smart pointers and so on, it doesn't improve meaningfully the situation -- one can still mess things up very easily with them.
And yes, sanitizers and lockdep and runtime solutions that require to trigger paths are amazing, but not anywhere close to enforcing something statically.
The fact that unsafe exists doesn't mean "Rust doesn't solve anything". Quite the opposite: the goal is to provide safe abstractions where possible, i.e. we minimize the need for unsafe. And for the cases where there is no other way around it, the toolchain will force you to write an explanation for your unsafe usage. Then maintainers and reviewers will have to agree with your argument for it.
In particular, it is not something that gets routinely (and implicitly) used every second line like we do in C.
Cheers,
Miguel
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 7 lines · Miguel Ojeda
Quoted 5 lines · Matthew Brost
You can do RAII in C - see
cleanup.h. Clear object lifetimes and ownership are what is important. Disciplined coding is the only to do this regardless of language. RAII doesn't help with help with bad object models / ownership / lifetime models either.
I hate cut off in thteads.
Quoted 3 lines
"Ownership", "lifetimes" and being "disciplined" *is* what Rust helps with. That is the whole point (even if there are other advantages).
I get it — you’re a Rust zealot. You can do this in C and enforce the rules quite well.
RAII cannot describe ownership transfers of refs, nor can it express who owns what in multi-threaded components, as far as I know. Ref-tracking and ownership need to be explicit.
I’m not going to reply to Rust vs C comments in this thread. If you want to talk about ownership, lifetimes, dma-fence enforcement, and teardown guarantees, sure.
If you want to build on top of a component that’s been tested on a production driver, great — please join in. If you want to figure out all the pitfalls yourself, well… have fun.
Matt
Quoted 24 lines
Yes, the cleanup attribute is nice, but even the whole
CLASSthing is meant to simplify code. Simplifying code does reduce bugs in general, but it doesn't solve anything fundamental. Even if we had C++ and full-fledged smart pointers and so on, it doesn't improve meaningfully the situation -- one can still mess things up very easily with them.And yes, sanitizers and lockdep and runtime solutions that require to trigger paths are amazing, but not anywhere close to enforcing something statically.
The fact that
unsafeexists doesn't mean "Rust doesn't solve anything". Quite the opposite: the goal is to provide safe abstractions where possible, i.e. we minimize the need forunsafe. And for the cases where there is no other way around it, the toolchain will force you to write an explanation for yourunsafeusage. Then maintainers and reviewers will have to agree with your argument for it.In particular, it is not something that gets routinely (and implicitly) used every second line like we do in C.
Cheers,
Miguel
Daniel Almeida
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Matthew,
Quoted 11 lines
I get it — you’re a Rust zealot. You can do this in C and enforce the rules quite well.
RAII cannot describe ownership transfers of refs, nor can it express who owns what in multi-threaded components, as far as I know. Ref-tracking and ownership need to be explicit.
I’m not going to reply to Rust vs C comments in this thread. If you want to talk about ownership, lifetimes, dma-fence enforcement, and teardown guarantees, sure.
Quoted 6 lines
If you want to build on top of a component that’s been tested on a production driver, great — please join in. If you want to figure out all the pitfalls yourself, well… have fun.
Matt
It is not about being a Rust zealot. I pointed out that your code has issues. Every time you access the queue you have to use a special function because the queue might be gone, how is this not a problem?
+ * However, there is a secondary hazard: a worker can be queued while the
+ * queue is in a "zombie" state — refcount has already reached zero and async
+ * teardown is in flight, but the work item has not yet been disabled by
+ * free_work. To guard against this every worker uses
+ * drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero
+ * the worker bails immediately without touching the queue state.
At various points you document requirements that are simply comments. Resource management is scattered all over the place, and it’s sometimes even shared with drivers, whom you have no control over.
+ * Drivers that set %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED and wish to
+ * serialise their own submit work against the bypass path must acquire this
+ * guard. Without it, a concurrent caller of drm_dep_job_push() could take
+ * the bypass path and call ops->run_job() inline between the driver's
+ * eligibility check and its corresponding action, producing a race.
How is this not a problem? Again, you’re not in control of driver code.
+ * If set, the driver is responsible for freeing the job. If NULL,
Same here.
Even if we take Rust out of the equation, how do you plan to solve these things? Or do you consider them solved as is?
I worry that we will find ourselves again at XDC in yet another scheduler workshop to address the issues that will invariably come up with your new design in a few years.
Quoted 3 lines
If you want to build on top of a component that’s been tested on a production driver, great — please join in. If you want to figure out all the pitfalls yourself, well… have fun.
Note that I didn’t show up with a low-effort “hey, how about we rewrite this in Rust?”. Instead, I linked to an actual Rust implementation that I spent weeks painstakingly debugging, not to mention the time it took to write it. Again, I suggest that you guys have a look, like I did with your code. You might find things you end up liking there.
— Daniel
Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Matthew BrostQuoted 4 lines · Matthew Brost
I hate cut off in thteads.
I get it — you’re a Rust zealot.
Cut off? Zealot?
Look, I got the email in my inbox, so I skimmed it to understand why I got it and why the Rust list was Cc'd. I happened to notice your (quite surprising) claims about Rust, so I decided to reply to a couple of those, since I proposed Rust for the kernel.
How is that a cut off and how does that make a maintainer a zealot?
Anyway, my understanding is that we agreed that the cleanup attribute in C doesn't enforce much of anything. We also agreed that it is important to think about ownership and lifetimes and to enforce the rules and to be disciplined. All good so far.
Now, what I said is simply that Rust fundamentally improves the situation -- C "RAII" not doing so is not comparable. For instance, that statically enforcing things is a meaningful improvement over runtime approaches (which generally require to trigger an issue, and which in some cases are not suitable for production settings).
Really, I just said Rust would help with things you already stated you care about. And nobody claims "Rust solves everything" as you stated. So I don't see zealots here, and insulting others doesn't help your argument.
Cheers,
Miguel
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 8 lines · Miguel Ojeda
Quoted 4 lines · Matthew Brost
I hate cut off in thteads.
I get it — you’re a Rust zealot.
Cut off? Zealot?
I appologize here I shouldn't type when I get annoyed. This the 2nd comment that pointing out difference between C and Rust which really wasn't direction I have hoping this thread would take.
Quoted 5 lines
Look, I got the email in my inbox, so I skimmed it to understand why I got it and why the Rust list was Cc'd. I happened to notice your (quite surprising) claims about Rust, so I decided to reply to a couple of those, since I proposed Rust for the kernel.
Again my mistake.
Quoted 7 lines
How is that a cut off and how does that make a maintainer a zealot?
Anyway, my understanding is that we agreed that the cleanup attribute in C doesn't enforce much of anything. We also agreed that it is important to think about ownership and lifetimes and to enforce the rules and to be disciplined. All good so far.
Quoted 6 lines
Now, what I said is simply that Rust fundamentally improves the situation -- C "RAII" not doing so is not comparable. For instance, that statically enforcing things is a meaningful improvement over runtime approaches (which generally require to trigger an issue, and which in some cases are not suitable for production settings).
I agree the static checking in Rust is a very nice feature.
Quoted 4 lines
Really, I just said Rust would help with things you already stated you care about. And nobody claims "Rust solves everything" as you stated. So I don't see zealots here, and insulting others doesn't help your argument.
I know, appologize.
Matt
Quoted 3 lines
Cheers,
Miguel
Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 4 lines · Matthew Brost
I appologize here I shouldn't type when I get annoyed. This the 2nd comment that pointing out difference between C and Rust which really wasn't direction I have hoping this thread would take.
No worries, it happens to everyone from time to time.
Thanks!
Cheers,
Miguel
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Matthew BrostQuoted 133 lines · Matthew Brost
Quoted 8 lines · Daniel Almeida
(+cc a few other people + Rust-for-Linux ML)
Hi Matthew,
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Holy war? See my reply to Danilo — I’ll write this in Rust if needed, but it’s not my first choice since I’m not yet a native speaker.
Quoted 21 lines
Quoted 21 lines · Matthew Brost
Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that
drm_gpu_scheduleris no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendereddrm_gpu_schedulerunmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.Add
drm_dep, a lightweight GPU submission queue intended as a replacement fordrm_gpu_schedulerfor firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlikedrm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination,drm_depmerges both roles into a single structdrm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.Quoted 5 lines
Quoted 3 lines
Unlike
drm_gpu_scheduler, which relies on external locking and lifetime management by the driver,drm_depuses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queueIn a domain that has been plagued by lifetime issues, we really should be
Yes, drm sched is a mess. I’ve been suggesting we fix it for years and have met pushback. This, however (drm dep), isn’t plagued by lifetime issues — that’s the primary focus here.
Quoted 2 lines
enforcing RAII for resource management instead of manual calls.
You can do RAII in C - see
cleanup.h. Clear object lifetimes and ownership are what is important. Disciplined coding is the only to do this regardless of language. RAII doesn't help with help with bad object models / ownership / lifetime models either.I don't buy the Rust solves everything argument but again non-native speaker.
Quoted 8 lines
Quoted 4 lines
reference from init until its last put, and the queue holds a job reference from dispatch until the
put_jobworker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.It makes use-after-free impossible _if_ you’re careful. It is not a property of the type system, and incorrect code will compile just fine.
Sure. If a driver puts a
drm_depobject reference on a resource thatdrm_depowns, it will explode. That’s effectively putting a reference on a resource the driver doesn’t own. A driver can write to any physical memory and crash the system anyway, so I’m not really sure what we’re talking about here. Rust doesn’t solve anything in this scenario — you can always use an unsafe block and put a reference on a resource you don’t own.Object model, ownership, and lifetimes are what is important and that is what drm dep is built around.
Quoted 26 lines
Quoted 18 lines
The core objects are:
struct
drm_dep_queue- a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.struct
drm_dep_job- a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of inputdma_fencedependencies and produce adrm_dep_fenceas their finished fence.struct
drm_dep_fence- adma_fencesubclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).Quoted 7 lines
Job lifecycle:
1.drm_dep_job_init()- allocate and initialise; job acquires a queue reference.
2.drm_dep_job_add_dependency()and friends - register input fences; duplicates from the same context are deduplicated.
3.drm_dep_job_arm()- assign sequence number, obtain finished fence.
4.drm_dep_job_push()- submit to queue.Quoted 4 lines
You cannot enforce this sequence easily in C code. Once again, we are trusting drivers that it is followed, but in Rust, you can simply reject code that does not follow this order at compile time.
I don’t know Rust, but yes — you can enforce this in C. It’s called lockdep and annotations. It’s not compile-time, but all of this is strictly enforced. e.g., write some code that doesn't follow this and report back if the kernel doesn't explode. It will, if doesn't I'll fix it to complain.
Quoted 11 lines
Quoted 11 lines
Submission paths under queue lock:
- Bypass path: ifDRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTEDis set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and therun_jobworker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before callingops->run_job().Quoted 9 lines
Quoted 3 lines
Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.
Why can’t we design an API where the driver can refuse jobs in
ops->run_job()if there are no resources to run it? This would do away with the credit system that has been in place for quite a while. Has this approach been tried in the past?
Quoted 7 lines
That seems possible if this is the preferred option.
-EAGAINis the way to do this. I’m open to the idea, but we also need to weigh the cost of converting drivers against the number of changes required.Partial - reply with catch up the rest later.
Appreciate the feedback.
Picking up replies.
Quoted 24 lines
Matt
Quoted 21 lines
Quoted 7 lines
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.Quoted 8 lines
IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.In rust, when you queue work, you have to pass a reference-counted pointer (Arc<T>). We simply never have this problem in a Rust design. If there is work queued, the queue is alive.
By the way, why can’t we simply require synchronous teardowns?
Consider the case where the DRM dep queue’s refcount drops to zero, but the device firmware still holds references to the associated queue. These are resources that must be torn down asynchronously. In Xe, I need to send two asynchronous firmware commands before I can safely remove the memory associated with the queue (faulting on this kind of global memory will take down the device) and recycle the firmware ID tied to the queue. These async commands are issued on the driver side, on the DRM dep queue’s workqueue as well.
Now consider a scenario where something goes wrong and those firmware commands never complete, and a device reset is required to recover. The driver’s per-queue tracking logic stops all queues (including zombie ones), determines which commands were lost, cleans up the side effects of that lost state, and then restarts all queues. That is how we would end up in this work item with a zombie queue. The restart logic could probably be made smart enough to avoid queueing work for zombie queues, but in my opinion it’s safe enough to use kref_get_unless_zero() in the work items.
It should also be clear that a DRM dep queue is primarily intended to be embedded inside the driver’s own queue object, even though it is valid to use it as a standalone object. The async teardown flows are also optional features.
Let’s also consider a case where you do not need the async firmware flows described above, but the DRM dep queue is still embedded in a driver-side object that owns memory via dma-resv. The final queue put may occur in IRQ context (DRM dep avoids kicking a worker just to drop a refi as opt in), or in the reclaim path (any scheduler workqueue is the reclaim path). In either case, you cannot free memory there taking a dma-resv lock, which is why all DRM dep queues ultimately free their resources in a work item outside of reclaim. Many drivers already follow this pattern, but in DRM dep this behavior is built-in.
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Quoted 103 lines
Quoted 103 lines
Quoted 103 lines
Teardown is always deferred to a module-private workqueue (
dep_free_wq) so thatdestroy_workqueue()is never called from within one of the queue's own workers. Each queue holds adrm_dev_get()reference on its owning structdrm_device, released as the final step of teardown viadrm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.Cc Boris Brezillon <[email protected]>Cc Tvrtko Ursulin <[email protected]>Cc Rodrigo Vivi <[email protected]>Cc Thomas Hellström <[email protected]>Cc Christian König <[email protected]>Cc Danilo Krummrich <[email protected]>Cc David Airlie <[email protected]>Cc Maarten Lankhorst <[email protected]>Cc Maxime Ripard <[email protected]>Cc Philipp Stanner <[email protected]>Cc Simona Vetter <[email protected]>Cc Sumit Semwal <[email protected]>Cc Thomas Zimmermann <[email protected]>Signed-off-by Matthew Brost <[email protected]>Assisted-by GitHub Copilot:claude-sonnet-4.6---
drivers/gpu/drm/Kconfig | 4 + drivers/gpu/drm/Makefile | 1 + drivers/gpu/drm/dep/Makefile | 5 + drivers/gpu/drm/dep/drm_dep_fence.c | 406 +++++++ drivers/gpu/drm/dep/drm_dep_fence.h | 25 + drivers/gpu/drm/dep/drm_dep_job.c | 675 +++++++++++ drivers/gpu/drm/dep/drm_dep_job.h | 13 + drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++ drivers/gpu/drm/dep/drm_dep_queue.h | 31 + include/drm/drm_dep.h | 597 ++++++++++ 10 files changed, 3404 insertions(+)create mode 100644
drivers/gpu/drm/dep/Makefilecreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_fence.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_job.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_job.hcreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.ccreate mode 100644drivers/gpu/drm/dep/drm_dep_queue.hcreate mode 100644include/drm/drm_dep.hdrivers/gpu/drm/Kconfig +0
-0@@ -276,6 +276,10 @@ config DRM_SCHEDtristate
depends on DRMdrivers/gpu/drm/Kconfig +4
-0+config DRM_DEP + tristate + depends on DRM +# Separate option as not all DRM drivers use it
configDRM_PANEL_BACKLIGHT_QUIRKS
tristatedrivers/gpu/drm/Makefile +0
-0@@ -173,6 +173,7 @@ obj-y += clients/obj-y += display/
obj-$(CONFIG_DRM_TTM) += ttm/
obj-$(CONFIG_DRM_SCHED) += scheduler/
+obj-$(CONFIG_DRM_DEP) += dep/
obj-$(CONFIG_DRM_RADEON)+= radeon/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/drivers/gpu/drm/dep/Makefile new file +5
-0@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o + +obj-$(CONFIG_DRM_DEP) += drm_dep.odrivers/gpu/drm/dep/drm_dep_fence.c new file +12
-0@@ -0,0 +1,406 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency fence +* +* Each struct drm_dep_job has an associated struct drm_dep_fence that +* provides a single dma_fence (@finished) signalled when the hardware +* completes the job. +*
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_fence.c line 16
Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +4
-0+* The hardware fence returned by &drm_dep_queue_ops.run_job is stored as +* @parent. @finished is chained to @parent via drm_dep_job_done_cb() and +* is signalled once @parent signals (or immediately if run_job() returns +* NULL or an error). I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here. One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
Another case is late arming of hardware fences in run_job (which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available at run_job, though.
Quoted 141 lines
Quoted 141 lines
Quoted 141 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +141
-0+* +* Drivers should expose @finished as the out-fence for GPU work since it is +* valid from the moment drm_dep_job_arm() returns, whereas the hardware fence +* could be a compound fence, which is disallowed when installed into +* drm_syncobjs or dma-resv. +* +* The fence uses the kernel's inline spinlock (NULL passed to dma_fence_init()) +* so no separate lock allocation is required. +* +* Deadline propagation is supported: if a consumer sets a deadline via +* dma_fence_set_deadline(), it is forwarded to @parent when @parent is set. +* If @parent has not been set yet the deadline is stored in @deadline and +* forwarded at that point. +* +* Memory management: drm_dep_fence objects are allocated with kzalloc() and +* freed via kfree_rcu() once the fence is released, ensuring safety with +* RCU-protected fence accesses. +*/ + +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include "drm_dep_fence.h" + +/** +* DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has been set +* +* Set by the deadline callback on the finished fence to indicate a deadline +* has been set which may need to be propagated to the parent hardware fence. +*/ +#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT (DMA_FENCE_FLAG_USER_BITS + 1) + +/** +* struct drm_dep_fence - fence tracking the completion of a dep job +* +* Contains a single dma_fence (@finished) that is signalled when the +* hardware completes the job. The fence uses the kernel's inline_lock +* (no external spinlock required). +* +* This struct is private to the drm_dep module; external code interacts +* through the accessor functions declared in drm_dep_fence.h. +*/ +struct drm_dep_fence { +/** +* @finished: signalled when the job completes on hardware. +* +* Drivers should use this fence as the out-fence for a job since it +* is available immediately upon drm_dep_job_arm(). +*/ +struct dma_fence finished; + +/** +* @deadline: deadline set on @finished which potentially needs to be +* propagated to @parent. +*/ +ktime_t deadline; + +/** +* @parent: The hardware fence returned by &drm_dep_queue_ops.run_job. +* +* @finished is signaled once @parent is signaled. The initial store is +* performed via smp_store_release to synchronize with deadline handling. +* +* All readers must access this under the fence lock and take a reference to +* it, as @parent is set to NULL under the fence lock when the drm_dep_fence +* signals, and this drop also releases its internal reference. +*/ +struct dma_fence *parent; + +/** +* @q: the queue this fence belongs to. +*/ +struct drm_dep_queue *q; +}; + +static const struct dma_fence_ops drm_dep_fence_ops; + +/** +* to_drm_dep_fence() - cast a dma_fence to its enclosing drm_dep_fence +* @f: dma_fence to cast +* +* Context: No context requirements (inline helper). +* Return: pointer to the enclosing &drm_dep_fence. +*/ +static struct drm_dep_fence *to_drm_dep_fence (struct dma_fence *f) +{ + return container_of(f, struct drm_dep_fence, finished); +} + +/** +* drm_dep_fence_set_parent() - store the hardware fence and propagate +* any deadline +* @dfence: dep fence +* @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or NULL/error +* +* Stores @parent on @dfence under smp_store_release() so that a concurrent +* drm_dep_fence_set_deadline() call sees the parent before checking the +* deadline bit. If a deadline has already been set on @dfence->finished it is +* forwarded to @parent immediately. Does nothing if @parent is NULL or an +* error pointer. +* +* Context: Any context. +*/ +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent) +{ +if (IS_ERR_OR_NULL (parent)) +return ; + +/* +* smp_store_release() to ensure a thread racing us in +* drm_dep_fence_set_deadline() sees the parent set before +* it calls test_bit(HAS_DEADLINE_BIT). +*/ +smp_store_release (&dfence->parent,dma_fence_get (parent)); +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , + &dfence->finished.flags)) +dma_fence_set_deadline (parent, dfence->deadline); +} + +/** +* drm_dep_fence_finished() - signal the finished fence with a result +* @dfence: dep fence to signal +* @result: error code to set, or 0 for success +* +* Sets the fence error to @result if non-zero, then signals +* @dfence->finished. Also removes parent visibility under the fence lock +* and drops the parent reference. Dropping the parent here allows the +* DRM dep fence to be completely decoupled from the DRM dep module. +* +* Context: Any context. +*/ +static void drm_dep_fence_finished (struct drm_dep_fence *dfence,int result) +{ +struct dma_fence *parent; +unsigned long flags; + +dma_fence_lock_irqsave (&dfence->finished, flags); +if (result) +dma_fence_set_error (&dfence->finished, result); +dma_fence_signal_locked (&dfence->finished); + parent = dfence->parent;
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_fence.c line 162
Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +5
-0+ dfence->parent =NULL ; +dma_fence_unlock_irqrestore (&dfence->finished, flags); + +dma_fence_put (parent); +}We should really try to move away from manual locks and unlocks.
I agree. Let's see if we can get dma_fence scoped guard in.
Quoted 241 lines
Quoted 241 lines
Quoted 241 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +239
-0+ +static const char *drm_dep_fence_get_driver_name (struct dma_fence *fence) +{ +return "drm_dep" ; +} + +static const char *drm_dep_fence_get_timeline_name (struct dma_fence *f) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); + +return dfence->q->name; +} + +/** +* drm_dep_fence_get_parent() - get a reference to the parent hardware fence +* @dfence: dep fence to query +* +* Returns a new reference to @dfence->parent, or NULL if the parent has +* already been cleared (i.e. @dfence->finished has signalled and the parent +* reference was dropped under the fence lock). +* +* Uses smp_load_acquire() to pair with the smp_store_release() in +* drm_dep_fence_set_parent(), ensuring that if we race a concurrent +* drm_dep_fence_set_parent() call we observe the parent pointer only after +* the store is fully visible — before set_parent() tests +* %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT. +* +* Caller must hold the fence lock on @dfence->finished. +* +* Context: Any context, fence lock on @dfence->finished must be held. +* Return: a new reference to the parent fence, or NULL. +*/ +static struct dma_fence *drm_dep_fence_get_parent (struct drm_dep_fence *dfence) +{ +dma_fence_assert_held (&dfence->finished); + +return dma_fence_get (smp_load_acquire (&dfence->parent)); +} + +/** +* drm_dep_fence_set_deadline() - dma_fence_ops deadline callback +* @f: fence on which the deadline is being set +* @deadline: the deadline hint to apply +* +* Stores the earliest deadline under the fence lock, then propagates +* it to the parent hardware fence via smp_load_acquire() to race +* safely with drm_dep_fence_set_parent(). +* +* Context: Any context. +*/ +static void drm_dep_fence_set_deadline (struct dma_fence *f,ktime_t deadline) +{ +struct drm_dep_fence *dfence =to_drm_dep_fence (f); +struct dma_fence *parent; +unsigned long flags; + +dma_fence_lock_irqsave (f, flags); + +/* If we already have an earlier deadline, keep it: */ +if (test_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags) && +ktime_before (dfence->deadline, deadline)) { +dma_fence_unlock_irqrestore (f, flags); +return ; + } + + dfence->deadline = deadline; +set_bit (DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags); + + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (f, flags); + +if (parent) +dma_fence_set_deadline (parent, deadline); + +dma_fence_put (parent); +} + +static const struct dma_fence_ops drm_dep_fence_ops = { + .get_driver_name = drm_dep_fence_get_driver_name, + .get_timeline_name = drm_dep_fence_get_timeline_name, + .set_deadline = drm_dep_fence_set_deadline, +}; + +/** +* drm_dep_fence_alloc() - allocate a dep fence +* +* Allocates a &drm_dep_fence with kzalloc() without initialising the +* dma_fence. Call drm_dep_fence_init() to fully initialise it. +* +* Context: Process context. +* Return: new &drm_dep_fence on success, NULL on allocation failure. +*/ +struct drm_dep_fence *drm_dep_fence_alloc (void ) +{ + return kzalloc_obj(struct drm_dep_fence); +} + +/** +* drm_dep_fence_init() - initialise the dma_fence inside a dep fence +* @dfence: dep fence to initialise +* @q: queue the owning job belongs to +* +* Initialises @dfence->finished using the context and sequence number from @q. +* Passes NULL as the lock so the fence uses its inline spinlock. +* +* Context: Any context. +*/ +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q) +{ +u32 seq = ++q->fence.seqno; + +/* +* XXX: Inline fence hazard: currently all expected users of DRM dep +* hardware fences have a unique lockdep class. If that ever changes, +* we will need to assign a unique lockdep class here so lockdep knows +* this fence is allowed to nest with driver hardware fences. +*/ + + dfence->q = q; +dma_fence_init (&dfence->finished, &drm_dep_fence_ops, +NULL , q->fence.context, seq); +} + +/** +* drm_dep_fence_cleanup() - release a dep fence at job teardown +* @dfence: dep fence to clean up +* +* Called from drm_dep_job_fini(). If the dep fence was armed (refcount > 0) +* it is released via dma_fence_put() and will be freed by the RCU release +* callback once all waiters have dropped their references. If it was never +* armed it is freed directly with kfree(). +* +* Context: Any context. +*/ +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence) +{ +if (drm_dep_fence_is_armed (dfence)) +dma_fence_put (&dfence->finished); +else +kfree (dfence); +} + +/** +* drm_dep_fence_is_armed() - check whether the fence has been armed +* @dfence: dep fence to check +* +* Returns true if drm_dep_job_arm() has been called, i.e. @dfence->finished +* has been initialised and its reference count is non-zero. Used by +* assertions to enforce correct job lifecycle ordering (arm before push, +* add_dependency before arm). +* +* Context: Any context. +* Return: true if the fence is armed, false otherwise. +*/ +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence) +{ +return !!kref_read (&dfence->finished.refcount); +} + +/** +* drm_dep_fence_is_finished() - test whether the finished fence has signalled +* @dfence: dep fence to check +* +* Uses dma_fence_test_signaled_flag() to read %DMA_FENCE_FLAG_SIGNALED_BIT +* directly without invoking the fence's ->signaled() callback or triggering +* any signalling side-effects. +* +* Context: Any context. +* Return: true if @dfence->finished has been signalled, false otherwise. +*/ +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence) +{ +return dma_fence_test_signaled_flag (&dfence->finished); +} + +/** +* drm_dep_fence_is_complete() - test whether the job has completed +* @dfence: dep fence to check +* +* Takes the fence lock on @dfence->finished and calls +* drm_dep_fence_get_parent() to safely obtain a reference to the parent +* hardware fence — or NULL if the parent has already been cleared after +* signalling. Calls dma_fence_is_signaled() on @parent outside the lock, +* which may invoke the fence's ->signaled() callback and trigger signalling +* side-effects if the fence has completed but the signalled flag has not yet +* been set. The finished fence is tested via dma_fence_test_signaled_flag(), +* without side-effects. +* +* May only be called on a stopped queue (see drm_dep_queue_is_stopped()). +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is complete, false otherwise. +*/ +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence) +{ +struct dma_fence *parent; +unsigned long flags; +bool complete; + +dma_fence_lock_irqsave (&dfence->finished, flags); + parent =drm_dep_fence_get_parent (dfence); +dma_fence_unlock_irqrestore (&dfence->finished, flags); + + complete = (parent &&dma_fence_is_signaled (parent)) || +dma_fence_test_signaled_flag (&dfence->finished); + +dma_fence_put (parent); + +return complete; +} + +/** +* drm_dep_fence_to_dma() - return the finished dma_fence for a dep fence +* @dfence: dep fence to query +* +* No reference is taken; the caller must hold its own reference to the owning +* &drm_dep_job for the duration of the access. +* +* Context: Any context. +* Return: the finished &dma_fence. +*/ +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence) +{ +return &dfence->finished; +} + +/** +* drm_dep_fence_done() - signal the finished fence on job completion +* @dfence: dep fence to signal +* @result: job error code, or 0 on success +* +* Gets a temporary reference to @dfence->finished to guard against a racing +* last-put, signals the fence with @result, then drops the temporary +* reference. Called from drm_dep_job_done() in the queue core when a +* hardware completion callback fires or when run_job() returns immediately. +* +* Context: Any context. +*/ +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result)
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_fence.c line 406
Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +5
-0+{ +dma_fence_get (&dfence->finished); +drm_dep_fence_finished (dfence, result); +dma_fence_put (&dfence->finished); +}Proper refcounting is automated (and enforced) in Rust.
That is a nice feature.
Quoted 116 lines
Quoted 116 lines
Quoted 116 lines
drivers/gpu/drm/dep/drm_dep_fence.h new file +25
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_FENCE_H_ +#define _DRM_DEP_FENCE_H_ + +#include <linux/dma-fence.h> + +struct drm_dep_fence ; +struct drm_dep_queue ; + +struct drm_dep_fence *drm_dep_fence_alloc (void ); +void drm_dep_fence_init (struct drm_dep_fence *dfence,struct drm_dep_queue *q); +void drm_dep_fence_cleanup (struct drm_dep_fence *dfence); +void drm_dep_fence_set_parent (struct drm_dep_fence *dfence, +struct dma_fence *parent); +void drm_dep_fence_done (struct drm_dep_fence *dfence,int result); +bool drm_dep_fence_is_armed (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_finished (struct drm_dep_fence *dfence); +bool drm_dep_fence_is_complete (struct drm_dep_fence *dfence); +struct dma_fence *drm_dep_fence_to_dma (struct drm_dep_fence *dfence); + +#endif /* _DRM_DEP_FENCE_H_ */ drivers/gpu/drm/dep/drm_dep_job.c new file +79
-0@@ -0,0 +1,675 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency job +* +* A struct drm_dep_job represents a single unit of GPU work associated with +* a struct drm_dep_queue. The lifecycle of a job is: +* +* 1. **Allocation**: the driver allocates memory for the job (typically by +* embedding struct drm_dep_job in a larger structure) and calls +* drm_dep_job_init() to initialise it. On success the job holds one +* kref reference and a reference to its queue. +* +* 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(), +* drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(), +* or drm_dep_job_add_implicit_dependencies() to register dma_fence objects +* that must be signalled before the job can run. Duplicate fences from the +* same fence context are deduplicated automatically. +* +* 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence, +* consuming a sequence number from the queue. After arming, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* 4. **Submission**: drm_dep_job_push() submits the job to the queue. The +* queue takes a reference that it holds until the job's finished fence +* signals and the job is freed by the put_job worker. +* +* 5. **Completion**: when the job's hardware work finishes its finished fence +* is signalled and drm_dep_job_put() is called by the queue. The driver +* must release any driver-private resources in &drm_dep_job_ops.release. +* +* Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The +* internal drm_dep_job_fini() tears down the dependency xarray and fence +* objects before the driver's release callback is invoked. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/slab.h> +#include <drm/drm_dep.h> +#include <drm/drm_file.h> +#include <drm/drm_gem.h> +#include <drm/drm_syncobj.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/** +* drm_dep_job_init() - initialise a dep job +* @job: dep job to initialise +* @args: initialisation arguments +* +* Initialises @job with the queue, ops and credit count from @args. Acquires +* a reference to @args->q via drm_dep_queue_get(); this reference is held for +* the lifetime of the job and released by drm_dep_job_release() when the last
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_job.c line 84
Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +5
-0+* job reference is dropped. +* +* Resources are released automatically when the last reference is dropped +* via drm_dep_job_put(), which must be called to release the job; drivers +* must not free the job directly. Again, can’t enforce that in C.
I agree. A driver could just kfree() the job after init… but in this design the driver unload would hang.
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +53
-0+* +* Context: Process context. Allocates memory with GFP_KERNEL. +* Return: 0 on success, -%EINVAL if credits is 0, +* -%ENOMEM on fence allocation failure. +*/ +int drm_dep_job_init (struct drm_dep_job *job, +const struct drm_dep_job_init_args *args) +{ +if (unlikely (!args->credits)) { +pr_err ("drm_dep: %s: credits cannot be 0\n" , __func__); +return -EINVAL ; + } + +memset (job,0 ,sizeof (*job)); + + job->dfence =drm_dep_fence_alloc (); +if (!job->dfence) +return -ENOMEM ; + + job->ops = args->ops; + job->q =drm_dep_queue_get (args->q); + job->credits = args->credits; + +kref_init (&job->refcount); +xa_init_flags (&job->dependencies,XA_FLAGS_ALLOC ); +INIT_LIST_HEAD (&job->pending_link); + +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_init); + +/** +* drm_dep_job_drop_dependencies() - release all input dependency fences +* @job: dep job whose dependency xarray to drain +* +* Walks @job->dependencies, puts each fence, and destroys the xarray. +* Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel — +* i.e. slots that were pre-allocated but never replaced — are silently +* skipped; the sentinel carries no reference. Called from +* drm_dep_queue_run_job() in process context immediately after +* @ops->run_job() returns, before the final drm_dep_job_put(). Releasing +* dependencies here — while still in process context — avoids calling +* xa_destroy() from IRQ context if the job's last reference is later +* dropped from a dma_fence callback. +* +* Context: Process context. +*/ +void drm_dep_job_drop_dependencies (struct drm_dep_job *job) +{ +struct dma_fence *fence; +unsigned long index; + +xa_for_each (&job->dependencies, index, fence) {
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_job.c line 143
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +6
-0+if (unlikely (fence ==DRM_DEP_JOB_FENCE_PREALLOC )) +continue ; +dma_fence_put (fence); + } +xa_destroy (&job->dependencies); +}This is automated in Rust. You also can’t “forget” to call this.
Driver code can’t call this function—note the lack of an export. DRM dep owns this call, and it always invokes it. But as discussed, a driver could still kfree() the job or forget to drop its creation reference.
Quoted 364 lines
Quoted 364 lines
Quoted 26 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +25
-0+ +/** +* drm_dep_job_fini() - clean up a dep job +* @job: dep job to clean up +* +* Cleans up the dep fence and drops the queue reference held by @job. +* +* If the job was never armed (e.g. init failed before drm_dep_job_arm()), +* the dependency xarray is also released here. For armed jobs the xarray +* has already been drained by drm_dep_job_drop_dependencies() in process +* context immediately after run_job(), so it is left untouched to avoid +* calling xa_destroy() from IRQ context. +* +* Warns if @job is still linked on the queue's pending list, which would +* indicate a bug in the teardown ordering. +* +* Context: Any context. +*/ +static void drm_dep_job_fini (struct drm_dep_job *job) +{ +bool armed =drm_dep_fence_is_armed (job->dfence); + +WARN_ON (!list_empty (&job->pending_link)); + +drm_dep_fence_cleanup (job->dfence);drivers/gpu/drm/dep/drm_dep_job.c line 178
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ job->dfence =NULL ; + +/* +* Armed jobs have their dependencies drained by +* drm_dep_job_drop_dependencies() in process context after run_job(). +* Skip here to avoid calling xa_destroy() from IRQ context. +*/ +if (!armed) +drm_dep_job_drop_dependencies (job); +}Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +6
-0+ +/** +* drm_dep_job_get() - acquire a reference to a dep job +* @job: dep job to acquire a reference on, or NULL +* +* Context: Any context. drivers/gpu/drm/dep/drm_dep_job.c line 194
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* Return: @job with an additional reference held, or NULL if @job is NULL. +*/ +struct drm_dep_job *drm_dep_job_get (struct drm_dep_job *job) +{ +if (job) +kref_get (&job->refcount); +return job; +} +EXPORT_SYMBOL (drm_dep_job_get); +Same here.
Quoted 19 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +19
-0+/** +* drm_dep_job_release() - kref release callback for a dep job +* @kref: kref embedded in the dep job +* +* Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Finally, releases the queue reference +* that was acquired by drm_dep_job_init() via drm_dep_queue_put(). The +* queue put is performed last to ensure no queue state is accessed after +* the job memory is freed. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the +* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +static void drm_dep_job_release (struct kref *kref) +{ +struct drm_dep_job *job = + container_of(kref, struct drm_dep_job, refcount); +struct drm_dep_queue *q = job->q;drivers/gpu/drm/dep/drm_dep_job.c line 223
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+ +drm_dep_job_fini (job); + +if (job->ops && job->ops->release) + job->ops->release (job); +else +kfree (job); + +drm_dep_queue_put (q); +}Same here.
Quoted 9 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +9
-0+ +/** +* drm_dep_job_put() - release a reference to a dep job +* @job: dep job to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_job_ops.release if set, +* otherwise frees @job with kfree(). Does nothing if @job is NULL. +* +* Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the drivers/gpu/drm/dep/drm_dep_job.c line 242
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+* job's queue; otherwise process context only, as the release callback may +* sleep. +*/ +void drm_dep_job_put (struct drm_dep_job *job) +{ +if (job) +kref_put (&job->refcount, drm_dep_job_release); +} +EXPORT_SYMBOL (drm_dep_job_put); +Same here.
Quoted 65 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +65
-0+/** +* drm_dep_job_arm() - arm a dep job for submission +* @job: dep job to arm +* +* Initialises the finished fence on @job->dfence, assigning +* it a sequence number from the job's queue. Must be called after +* drm_dep_job_init() and before drm_dep_job_push(). Once armed, +* drm_dep_job_finished_fence() returns a valid fence that may be passed to +* userspace or used as a dependency by other jobs. +* +* Begins the DMA fence signalling path via dma_fence_begin_signalling(). +* After this point, memory allocations that could trigger reclaim are +* forbidden; lockdep enforces this. arm() must always be paired with +* drm_dep_job_push(); lockdep also enforces this pairing. +* +* Warns if the job has already been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_arm (struct drm_dep_job *job) +{ +drm_dep_queue_push_job_begin (job->q); +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +drm_dep_fence_init (job->dfence, job->q); + job->signalling_cookie =dma_fence_begin_signalling (); +} +EXPORT_SYMBOL (drm_dep_job_arm); + +/** +* drm_dep_job_push() - submit a job to its queue for execution +* @job: dep job to push +* +* Submits @job to the queue it was initialised with. Must be called after +* drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue, +* held until the queue is fully done with it. The reference is released +* directly in the finished-fence dma_fence callback for queues with +* %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run +* from hardirq context), or via the put_job work item on the submit +* workqueue otherwise. +* +* Ends the DMA fence signalling path begun by drm_dep_job_arm() via +* dma_fence_end_signalling(). This must be paired with arm(); lockdep +* enforces the pairing. +* +* Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for +* @job exactly once, even if the queue is killed or torn down before the +* job reaches the head of the queue. Drivers can use this guarantee to +* perform bookkeeping cleanup; the actual backend operation should be +* skipped when drm_dep_queue_is_killed() returns true. +* +* If the queue does not support the bypass path, the job is pushed directly +* onto the SPSC submission queue via drm_dep_queue_push_job() without holding +* @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either +* run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or +* enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item. +* +* Warns if the job has not been armed. +* +* Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set +* (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling +* path. +*/ +void drm_dep_job_push (struct drm_dep_job *job)drivers/gpu/drm/dep/drm_dep_job.c line 317
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+{ +struct drm_dep_queue *q = job->q; + +WARN_ON (!drm_dep_fence_is_armed (job->dfence)); + +drm_dep_job_get (job); + +if (!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED )) { +drm_dep_queue_push_job (q, job); +dma_fence_end_signalling (job->signalling_cookie);Signaling is enforced in a more thorough way in Rust. I’ll expand on this later in this patch.
Quoted 174 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +173
-0+drm_dep_queue_push_job_end (job->q); +return ; + } + + scoped_guard(mutex, &q->sched.lock) { +if (drm_dep_queue_can_job_bypass (q, job)) +drm_dep_queue_run_job (q, job); +else +drm_dep_queue_push_job (q, job); + } + +dma_fence_end_signalling (job->signalling_cookie); +drm_dep_queue_push_job_end (job->q); +} +EXPORT_SYMBOL (drm_dep_job_push); + +/** +* drm_dep_job_add_dependency() - adds the fence as a job dependency +* @job: dep job to add the dependencies to +* @fence: the dma_fence to add to the list of dependencies, or +* %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later. +* +* Note that @fence is consumed in both the success and error cases (except +* when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference). +* +* Signalled fences and fences belonging to the same queue as @job (i.e. where +* fence->context matches the queue's finished fence context) are silently +* dropped; the job need not wait on its own queue's output. +* +* Warns if the job has already been armed (dependencies must be added before +* drm_dep_job_arm()). +* +* **Pre-allocation pattern** +* +* When multiple jobs across different queues must be prepared and submitted +* together in a single atomic commit — for example, where job A's finished +* fence is an input dependency of job B — all jobs must be armed and pushed +* within a single dma_fence_begin_signalling() / dma_fence_end_signalling() +* region. Once that region has started no memory allocation is permitted. +* +* To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation +* phase (before arming any job, while GFP_KERNEL allocation is still allowed) +* to pre-allocate a slot in @job->dependencies. The slot index assigned by +* the underlying xarray must be tracked by the caller separately (e.g. it is +* always index 0 when the dependency array is empty, as Xe relies on). +* After all jobs have been armed and the finished fences are available, call +* drm_dep_job_replace_dependency() with that index and the real fence. +* drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be +* called from atomic or signalling context. +* +* The sentinel slot is never skipped by the signalled-fence fast-path, +* ensuring a slot is always allocated even when the real fence is not yet +* known. +* +* **Example: bind job feeding TLB invalidation jobs** +* +* Consider a GPU with separate queues for page-table bind operations and for +* TLB invalidation. A single atomic commit must: +* +* 1. Run a bind job that modifies page tables. +* 2. Run one TLB-invalidation job per MMU that depends on the bind +* completing, so stale translations are flushed before the engines +* continue. +* +* Because all jobs must be armed and pushed inside a signalling region (where +* GFP_KERNEL is forbidden), pre-allocate slots before entering the region:: +* +* // Phase 1 — process context, GFP_KERNEL allowed +* drm_dep_job_init(bind_job, bind_queue, ops); +* for_each_mmu(mmu) { +* drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops); +* // Pre-allocate slot at index 0; real fence not available yet +* drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC); +* } +* +* // Phase 2 — inside signalling region, no GFP_KERNEL +* dma_fence_begin_signalling(); +* drm_dep_job_arm(bind_job); +* for_each_mmu(mmu) { +* // Swap sentinel for bind job's finished fence +* drm_dep_job_replace_dependency(tlb_job[mmu], 0, +* dma_fence_get(bind_job->finished)); +* drm_dep_job_arm(tlb_job[mmu]); +* } +* drm_dep_job_push(bind_job); +* for_each_mmu(mmu) +* drm_dep_job_push(tlb_job[mmu]); +* dma_fence_end_signalling(); +* +* Context: Process context. May allocate memory with GFP_KERNEL. +* Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on +* success, else 0 on success, or a negative error code. +*/ +int drm_dep_job_add_dependency (struct drm_dep_job *job,struct dma_fence *fence) +{ +struct drm_dep_queue *q = job->q; +struct dma_fence *entry; +unsigned long index; +u32 id =0 ; +int ret; + +WARN_ON (drm_dep_fence_is_armed (job->dfence)); +might_alloc (GFP_KERNEL ); + +if (!fence) +return 0 ; + +if (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) + goto add_fence; + +/* +* Ignore signalled fences or fences from our own queue — finished +* fences use q->fence.context. +*/ +if (dma_fence_test_signaled_flag (fence) || + fence->context == q->fence.context) { +dma_fence_put (fence); +return 0 ; + } + +/* Deduplicate if we already depend on a fence from the same context. +* This lets the size of the array of deps scale with the number of +* engines involved, rather than the number of BOs. +*/ +xa_for_each (&job->dependencies, index, entry) { +if (entry ==DRM_DEP_JOB_FENCE_PREALLOC || + entry->context != fence->context) +continue ; + +if (dma_fence_is_later (fence, entry)) { +dma_fence_put (entry); +xa_store (&job->dependencies, index, fence,GFP_KERNEL ); + }else { +dma_fence_put (fence); + } +return 0 ; + } + +add_fence: + ret =xa_alloc (&job->dependencies, &id, fence, xa_limit_32b, +GFP_KERNEL ); +if (ret !=0 ) { +if (fence !=DRM_DEP_JOB_FENCE_PREALLOC ) +dma_fence_put (fence); +return ret; + } + +return (fence ==DRM_DEP_JOB_FENCE_PREALLOC ) ? id :0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_dependency); + +/** +* drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot +* @job: dep job to update +* @index: xarray index of the slot to replace, as returned when the sentinel +* was originally inserted via drm_dep_job_add_dependency() +* @fence: the real dma_fence to store; its reference is always consumed +* +* Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in +* @job->dependencies with @fence. The slot must have been pre-allocated by +* passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the +* existing entry is asserted to be the sentinel. +* +* This is the second half of the pre-allocation pattern described in +* drm_dep_job_add_dependency(). It is intended to be called inside a +* dma_fence_begin_signalling() / dma_fence_end_signalling() region where +* memory allocation with GFP_KERNEL is forbidden. It uses GFP_NOWAIT +* internally so it is safe to call from atomic or signalling context, but +* since the slot has been pre-allocated no actual memory allocation occurs. +* +* If @fence is already signalled the slot is erased rather than storing a +* redundant dependency. The successful store is asserted — if the store +* fails it indicates a programming error (slot index out of range or
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
+ * concurrent modification).
+ *
+ * Must be called beforedrm_dep_job_arm(). @fence is consumed in all cases.Can’t enforce this in C. Also, how is the fence “consumed” ? You can’t enforce that the user can’t access the fence anymore after this function returns, like we can do at compile time in Rust.
I agree—you can’t enforce correct usage at compile time. The best you can do is document the rules and annotate them. DRM dep will complain when those rules are violated.
Quoted 293 lines
Quoted 293 lines
Quoted 163 lines
+ *
+ * Context: Any context. DMA fence signaling path.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +160
-0+void drm_dep_job_replace_dependency (struct drm_dep_job *job,u32 index, +struct dma_fence *fence) +{ +WARN_ON (xa_load (&job->dependencies, index) != +DRM_DEP_JOB_FENCE_PREALLOC ); + +if (dma_fence_test_signaled_flag (fence)) { +xa_erase (&job->dependencies, index); +dma_fence_put (fence); +return ; + } + +if (WARN_ON (xa_is_err (xa_store (&job->dependencies, index, fence, +GFP_NOWAIT )))) { +dma_fence_put (fence); +return ; + } +} +EXPORT_SYMBOL (drm_dep_job_replace_dependency); + +/** +* drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a +* job dependency +* @job: dep job to add the dependencies to +* @file: drm file private pointer +* @handle: syncobj handle to lookup +* @point: timeline point +* +* This adds the fence matching the given syncobj to @job. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_syncobj_dependency (struct drm_dep_job *job, +struct drm_file *file,u32 handle, +u32 point) +{ +struct dma_fence *fence; +int ret; + + ret =drm_syncobj_find_fence (file, handle, point,0 , &fence); +if (ret) +return ret; + +return drm_dep_job_add_dependency (job, fence); +} +EXPORT_SYMBOL (drm_dep_job_add_syncobj_dependency); + +/** +* drm_dep_job_add_resv_dependencies() - add all fences from the resv to the job +* @job: dep job to add the dependencies to +* @resv: the dma_resv object to get the fences from +* @usage: the dma_resv_usage to use to filter the fences +* +* This adds all fences matching the given usage from @resv to @job. +* Must be called with the @resv lock held. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_resv_dependencies (struct drm_dep_job *job, +struct dma_resv *resv, +enum dma_resv_usage usage) +{ +struct dma_resv_iter cursor; +struct dma_fence *fence; +int ret; + +dma_resv_assert_held (resv); + +dma_resv_for_each_fence (&cursor, resv, usage, fence) { +/* +* As drm_dep_job_add_dependency always consumes the fence +* reference (even when it fails), and dma_resv_for_each_fence +* is not obtaining one, we need to grab one before calling. +*/ + ret =drm_dep_job_add_dependency (job,dma_fence_get (fence)); +if (ret) +return ret; + } +return 0 ; +} +EXPORT_SYMBOL (drm_dep_job_add_resv_dependencies); + +/** +* drm_dep_job_add_implicit_dependencies() - adds implicit dependencies +* as job dependencies +* @job: dep job to add the dependencies to +* @obj: the gem object to add new dependencies from. +* @write: whether the job might write the object (so we need to depend on +* shared fences in the reservation object). +* +* This should be called after drm_gem_lock_reservations() on your array of +* GEM objects used in the job but before updating the reservations with your +* own fences. +* +* Context: Process context. +* Return: 0 on success, or a negative error code. +*/ +int drm_dep_job_add_implicit_dependencies (struct drm_dep_job *job, +struct drm_gem_object *obj, +bool write) +{ +return drm_dep_job_add_resv_dependencies (job, obj->resv, +dma_resv_usage_rw (write)); +} +EXPORT_SYMBOL (drm_dep_job_add_implicit_dependencies); + +/** +* drm_dep_job_is_signaled() - check whether a dep job has completed +* @job: dep job to check +* +* Determines whether @job has signalled. The queue should be stopped before +* calling this to obtain a stable snapshot of state. Both the parent hardware +* fence and the finished software fence are checked. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if the job is signalled, false otherwise. +*/ +bool drm_dep_job_is_signaled (struct drm_dep_job *job) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q)); +return drm_dep_fence_is_complete (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_signaled); + +/** +* drm_dep_job_is_finished() - test whether a dep job's finished fence has signalled +* @job: dep job to check +* +* Tests whether the job's software finished fence has been signalled, using +* dma_fence_test_signaled_flag() to avoid any signalling side-effects. Unlike +* drm_dep_job_is_signaled(), this does not require the queue to be stopped and +* does not check the parent hardware fence — it is a lightweight test of the +* finished fence only. +* +* Context: Any context. +* Return: true if the job's finished fence has been signalled, false otherwise. +*/ +bool drm_dep_job_is_finished (struct drm_dep_job *job) +{ +return drm_dep_fence_is_finished (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_is_finished); + +/** +* drm_dep_job_invalidate_job() - increment the invalidation count for a job +* @job: dep job to invalidate +* @threshold: threshold above which the job is considered invalidated +* +* Increments @job->invalidate_count and returns true if it exceeds @threshold, +* indicating the job should be considered hung and discarded. The queue must +* be stopped before calling this function. +* +* Context: Process context. The queue must be stopped before calling this. +* Return: true if @job->invalidate_count exceeds @threshold, false otherwise. +*/ +bool drm_dep_job_invalidate_job (struct drm_dep_job *job,int threshold) +{ +WARN_ON (!drm_dep_queue_is_stopped (job->q));drivers/gpu/drm/dep/drm_dep_job.c line 666
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10
-0+return ++job->invalidate_count > threshold; +} +EXPORT_SYMBOL (drm_dep_job_invalidate_job); + +/** +* drm_dep_job_finished_fence() - return the finished fence for a job +* @job: dep job to query +* +* No reference is taken on the returned fence; the caller must hold its own +* reference to @job for the duration of any access. Can’t enforce this in C.
Quoted 117 lines
+ *
+ * Context: Any context.
+ * Return: the finished&dma_fencefor @job.
+ */drivers/gpu/drm/dep/drm_dep_job.c new file +5
-0+struct dma_fence *drm_dep_job_finished_fence (struct drm_dep_job *job) +{ +return drm_dep_fence_to_dma (job->dfence); +} +EXPORT_SYMBOL (drm_dep_job_finished_fence);drivers/gpu/drm/dep/drm_dep_job.h new file +13
-0@@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_JOB_H_ +#define _DRM_DEP_JOB_H_ + +struct drm_dep_queue ; + +void drm_dep_job_drop_dependencies (struct drm_dep_job *job); + +#endif /* _DRM_DEP_JOB_H_ */ drivers/gpu/drm/dep/drm_dep_queue.c new file +83
-0@@ -0,0 +1,1647 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +/** +* DOC: DRM dependency queue +* +* The drm_dep subsystem provides a lightweight GPU submission queue that +* combines the roles of drm_gpu_scheduler and drm_sched_entity into a +* single object (struct drm_dep_queue). Each queue owns its own ordered +* submit workqueue, timeout workqueue, and TDR delayed-work. +* +* **Job lifecycle** +* +* 1. Allocate and initialise a job with drm_dep_job_init(). +* 2. Add dependency fences with drm_dep_job_add_dependency() and friends. +* 3. Arm the job with drm_dep_job_arm() to obtain its out-fences. +* 4. Submit with drm_dep_job_push(). +* +* **Submission paths** +* +* drm_dep_job_push() decides between two paths under @q->sched.lock: +* +* - **Bypass path** (drm_dep_queue_can_job_bypass()): if +* %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not stopped, +* the SPSC queue is empty, the job has no dependency fences, and credits +* are available, the job is submitted inline on the calling thread without +* touching the submit workqueue. +* +* - **Queued path** (drm_dep_queue_push_job()): the job is pushed onto an +* SPSC queue and the run_job worker is kicked. The run_job worker pops the +* job, resolves any remaining dependency fences (installing wakeup +* callbacks for unresolved ones), and calls drm_dep_queue_run_job(). +* +* **Running a job** +* +* drm_dep_queue_run_job() accounts credits, appends the job to the pending +* list (starting the TDR timer only when the list was previously empty), +* calls @ops->run_job(), stores the returned hardware fence as the parent +* of the job's dep fence, then installs a callback on it. When the hardware +* fence fires (or the job completes synchronously), drm_dep_job_done() +* signals the finished fence, returns credits, and kicks the put_job worker +* to free the job. +* +* **Timeout detection and recovery (TDR)** +* +* A delayed work item fires when a job on the pending list takes longer than +* @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on the +* returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB). +* drm_dep_queue_trigger_timeout() forces the timer to fire immediately (without +* changing the stored timeout), for example during device teardown. +* +* **Reference counting** +* +* Jobs and queues are both reference counted. +* +* A job holds a reference to its queue from drm_dep_job_init() until +* drm_dep_job_put() drops the job's last reference and its release callback +* runs. This ensures the queue remains valid for the entire lifetime of any +* job that was submitted to it. +*
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_queue.c line 87
Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +4
-0+* The queue holds its own reference to a job for as long as the job is +* internally tracked: from the moment the job is added to the pending list +* in drm_dep_queue_run_job() until drm_dep_job_done() kicks the put_job +* worker, which calls drm_dep_job_put() to release that reference. Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context) so we avoid switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +17
-0+* +* **Hazard: use-after-free from within a worker** +* +* Because a job holds a queue reference, drm_dep_job_put() dropping the last +* job reference will also drop a queue reference via the job's release path. +* If that happens to be the last queue reference, drm_dep_queue_fini() can be +* called, which queues @q->free_work on dep_free_wq and returns immediately. +* free_work calls disable_work_sync() / disable_delayed_work_sync() on the +* queue's own workers before destroying its workqueues, so in practice a +* running worker always completes before the queue memory is freed. +* +* However, there is a secondary hazard: a worker can be queued while the +* queue is in a "zombie" state — refcount has already reached zero and async +* teardown is in flight, but the work item has not yet been disabled by +* free_work. To guard against this every worker uses +* drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero +* the worker bails immediately without touching the queue state.
Quoted 2 lines
Quoted 2 lines
Again, this problem is gone in Rust.
I answer this one above.
Quoted 41 lines
Quoted 41 lines
Quoted 41 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +41
-0+* +* Because all actual teardown (disable_*_sync, destroy_workqueue) runs on +* dep_free_wq — which is independent of the queue's own submit/timeout +* workqueues — there is no deadlock risk. Each queue holds a drm_dev_get() +* reference on its owning &drm_device, which is released as the last step of +* teardown. This ensures the driver module cannot be unloaded while any queue +* is still alive. +*/ + +#include <linux/dma-resv.h> +#include <linux/kref.h> +#include <linux/module.h> +#include <linux/overflow.h> +#include <linux/slab.h> +#include <linux/wait.h> +#include <linux/workqueue.h> +#include <drm/drm_dep.h> +#include <drm/drm_drv.h> +#include <drm/drm_print.h> +#include "drm_dep_fence.h" +#include "drm_dep_job.h" +#include "drm_dep_queue.h" + +/* +* Dedicated workqueue for deferred drm_dep_queue teardown. Using a +* module-private WQ instead of system_percpu_wq keeps teardown isolated +* from unrelated kernel subsystems. +*/ +static struct workqueue_struct *dep_free_wq; + +/** +* drm_dep_queue_flags_set() - set a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to set (one of &enum drm_dep_queue_flags) +* +* Sets @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_set(struct drm_dep_queue *q,
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_queue.c line 148
Quoted 3 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +3
-0+ enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock);We can enforce this in Rust at compile-time. The code does not compile if the lock is not taken. Same here and everywhere else where the sched lock has to be taken.
I do understand part of Rust and agree it is a nice feature.
Quoted 371 lines
Quoted 371 lines
Quoted 125 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +124
-0+ q->sched.flags |= flag; +} + +/** +* drm_dep_queue_flags_clear() - clear a flag on the queue under sched.lock +* @q: dep queue +* @flag: flag to clear (one of &enum drm_dep_queue_flags) +* +* Clears @flag in @q->sched.flags. Must be called with @q->sched.lock +* held; the lockdep assertion enforces this. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_flags_clear (struct drm_dep_queue *q, +enum drm_dep_queue_flags flag) +{ +lockdep_assert_held (&q->sched.lock); + q->sched.flags &= ~flag; +} + +/** +* drm_dep_queue_has_credits() - check whether the queue has enough credits +* @q: dep queue +* @job: job requesting credits +* +* Checks whether the queue has enough available credits to dispatch +* @job. If @job->credits exceeds the queue's credit limit, it is +* clamped with a WARN. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if available credits >= @job->credits, false otherwise. +*/ +static bool drm_dep_queue_has_credits (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +u32 available; + +lockdep_assert_held (&q->sched.lock); + +if (job->credits > q->credit.limit) { +drm_warn (q->drm, +"Jobs may not exceed the credit limit, truncate.\n" ); + job->credits = q->credit.limit; + } + +WARN_ON (check_sub_overflow (q->credit.limit, +atomic_read (&q->credit.count), + &available)); + +return available >= job->credits; +} + +/** +* drm_dep_queue_run_job_queue() - kick the run-job worker +* @q: dep queue +* +* Queues @q->sched.run_job on @q->sched.submit_wq unless the queue is stopped +* or the job queue is empty. The empty-queue check avoids queueing a work item +* that would immediately return with nothing to do. +* +* Context: Any context. +*/ +static void drm_dep_queue_run_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q) &&spsc_queue_count (&q->job.queue)) +queue_work (q->sched.submit_wq, &q->sched.run_job); +} + +/** +* drm_dep_queue_put_job_queue() - kick the put-job worker +* @q: dep queue +* +* Queues @q->sched.put_job on @q->sched.submit_wq unless the queue +* is stopped. +* +* Context: Any context. +*/ +static void drm_dep_queue_put_job_queue (struct drm_dep_queue *q) +{ +if (!drm_dep_queue_is_stopped (q)) +queue_work (q->sched.submit_wq, &q->sched.put_job); +} + +/** +* drm_queue_start_timeout() - arm or re-arm the TDR delayed work +* @q: dep queue +* +* Arms the TDR delayed work with @q->job.timeout. No-op if +* @q->ops->timedout_job is NULL, the timeout is MAX_SCHEDULE_TIMEOUT, +* or the pending list is empty. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_queue_start_timeout (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->job.lock); + +if (!q->ops->timedout_job || + q->job.timeout ==MAX_SCHEDULE_TIMEOUT || +list_empty (&q->job.pending)) +return ; + +mod_delayed_work (q->sched.timeout_wq, &q->sched.tdr, q->job.timeout); +} + +/** +* drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock +* @q: dep queue +* +* Acquires @q->job.lock with irq and calls +* drm_queue_start_timeout(). +* +* Context: Process context (workqueue). +*/ +static void drm_queue_start_timeout_unlocked (struct drm_dep_queue *q) +{ +guard (spinlock_irq)(&q->job.lock); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_remove_dependency() - clear the active dependency and wake +* the run-job worker +* @q: dep queue drivers/gpu/drm/dep/drm_dep_queue.c line 282
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @f: the dependency fence being removed +* +* Stores @f into @q->dep.removed_fence via smp_store_release() so that the +* run-job worker can drop the reference to it in drm_dep_queue_is_ready(), +* paired with smp_load_acquire(). Clears @q->dep.fence and kicks the +* run-job worker. +* +* The fence reference is not dropped here; it is deferred to the run-job +* worker via @q->dep.removed_fence to keep this path suitable dma_fence +* callback removal in drm_dep_queue_kill(). This is a comment in C, but in Rust this is encoded directly in the type system.
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_remove_dependency (struct drm_dep_queue *q, +struct dma_fence *f) +{ +/* removed_fence must be visible to the reader before &q->dep.fence */ +smp_store_release (&q->dep.removed_fence, f); + +WRITE_ONCE (q->dep.fence,NULL ); +drm_dep_queue_run_job_queue (q); +}drivers/gpu/drm/dep/drm_dep_queue.c line 304
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +/** +* drm_dep_queue_wakeup() - dma_fence callback to wake the run-job worker +* @f: the signalled dependency fence +* @cb: callback embedded in the dep queue +* +* Called from dma_fence_signal() when the active dependency fence signals. +* Delegates to drm_dep_queue_remove_dependency() to clear @q->dep.fence and +* kick the run-job worker. The fence reference is not dropped here; it is +* deferred to the run-job worker via @q->dep.removed_fence. Same here.
Quoted 6 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +6
-0+* +* Context: Any context. +*/ +static void drm_dep_queue_wakeup (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_queue *q =drivers/gpu/drm/dep/drm_dep_queue.c line 320
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ container_of(cb, struct drm_dep_queue, dep.cb); + +drm_dep_queue_remove_dependency (q, f); +} + +/** +* drm_dep_queue_is_ready() - check whether the queue has a dispatchable job +* @q: dep queue +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. Can’t call this in Rust if the lock is not taken.
Quoted 158 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +158
-0+* Return: true if SPSC queue non-empty and no dep fence pending, +* false otherwise. +*/ +static bool drm_dep_queue_is_ready (struct drm_dep_queue *q) +{ +lockdep_assert_held (&q->sched.lock); + +if (!spsc_queue_count (&q->job.queue)) +return false; + +if (READ_ONCE (q->dep.fence)) +return false; + +/* Paired with smp_store_release in drm_dep_queue_remove_dependency() */ +dma_fence_put (smp_load_acquire (&q->dep.removed_fence)); + + q->dep.removed_fence =NULL ; + +return true; +} + +/** +* drm_dep_queue_is_killed() - check whether a dep queue has been killed +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_killed (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_KILLED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_killed); + +/** +* drm_dep_queue_is_initialized() - check whether a dep queue has been initialized +* @q: dep queue to check +* +* A queue is considered initialized once its ops pointer has been set by a +* successful call to drm_dep_queue_init(). Drivers that embed a +* &drm_dep_queue inside a larger structure may call this before attempting any +* other queue operation to confirm that initialization has taken place. +* drm_dep_queue_put() must be called if this function returns true to drop the +* initialization reference from drm_dep_queue_init(). +* +* Return: true if @q has been initialized, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_initialized (struct drm_dep_queue *q) +{ +return !!q->ops; +} +EXPORT_SYMBOL (drm_dep_queue_is_initialized); + +/** +* drm_dep_queue_set_stopped() - pre-mark a queue as stopped before first use +* @q: dep queue to mark +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going through the +* normal drm_dep_queue_stop() path. This is only valid during the driver-side +* queue initialisation sequence — i.e. after drm_dep_queue_init() returns but +* before the queue is made visible to other threads (e.g. before it is added +* to any lookup structures). Using this after the queue is live is a driver +* bug; use drm_dep_queue_stop() instead. +* +* Context: Process context, queue not yet visible to other threads. +*/ +void drm_dep_queue_set_stopped (struct drm_dep_queue *q) +{ + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_STOPPED ; +} +EXPORT_SYMBOL (drm_dep_queue_set_stopped); + +/** +* drm_dep_queue_refcount() - read the current reference count of a queue +* @q: dep queue to query +* +* Returns the instantaneous kref value. The count may change immediately +* after this call; callers must not make safety decisions based solely on +* the returned value. Intended for diagnostic snapshots and debugfs output. +* +* Context: Any context. +* Return: current reference count. +*/ +unsigned int drm_dep_queue_refcount (const struct drm_dep_queue *q) +{ +return kref_read (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_refcount); + +/** +* drm_dep_queue_timeout() - read the per-job TDR timeout for a queue +* @q: dep queue to query +* +* Returns the per-job timeout in jiffies as set at init time. +* %MAX_SCHEDULE_TIMEOUT means no timeout is configured. +* +* Context: Any context. +* Return: timeout in jiffies. +*/ +long drm_dep_queue_timeout (const struct drm_dep_queue *q) +{ +return q->job.timeout; +} +EXPORT_SYMBOL (drm_dep_queue_timeout); + +/** +* drm_dep_queue_is_job_put_irq_safe() - test whether job-put from IRQ is allowed +* @q: dep queue +* +* Context: Any context. +* Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set, +* false otherwise. +*/ +static bool drm_dep_queue_is_job_put_irq_safe (const struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE ); +} + +/** +* drm_dep_queue_job_dependency() - get next unresolved dep fence +* @q: dep queue +* @job: job whose dependencies to advance +* +* Returns NULL immediately if the queue has been killed via +* drm_dep_queue_kill(), bypassing all dependency waits so that jobs +* drain through run_job as quickly as possible. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next unresolved &dma_fence with a new reference, or NULL +* when all dependencies have been consumed (or the queue is killed). +*/ +static struct dma_fence * +drm_dep_queue_job_dependency (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *f; + +lockdep_assert_held (&q->sched.lock); + +if (drm_dep_queue_is_killed (q)) +return NULL ; + + f =xa_load (&job->dependencies, job->last_dependency); +if (f) { + job->last_dependency++; +if (WARN_ON (DRM_DEP_JOB_FENCE_PREALLOC == f)) +return dma_fence_get_stub (); +return dma_fence_get (f); + } + +return NULL ; +} + +/** +* drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence drivers/gpu/drm/dep/drm_dep_queue.c line 488
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+* @q: dep queue +* @job: job whose dependency fence is stored in @q->dep.fence +* +* Installs a wakeup callback on @q->dep.fence. Returns true if the +* callback was installed (the queue must wait), false if the fence is +* already signalled or is a self-fence from the same queue context. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: true if callback installed, false if fence already done. +*/ In Rust, we can encode the signaling paths with a “token type”. So any sections that are part of the signaling path can simply take this token as an argument. This type also enforces that
end_signaling()is called automatically when it goes out of scope.By the way, we can easily offer an irq handler type where we enforce this:
fn
handle_threaded_irq(&self, device: &Device<Bound>)-> IrqReturn {
let_annotation=DmaFenceSignallingAnnotation::new(); // Callsbegin_signaling()self.driver.handle_threaded_irq(device)//
end_signaling()is called here automatically.
}Same for workqueues:
fn
work_fn(&self, device: &Device<Bound>){
let_annotation=DmaFenceSignallingAnnotation::new(); // Callsbegin_signaling()self.driver.work_fn(device)
Quoted 5 lines
Quoted 5 lines
//
end_signaling()is called here automatically.
}This is not Rust-specific, of course, but it is more ergonomic to write in Rust.
Yes, I agree this is a nice feature, and properly annotating C code requires discipline.
Quoted 68 lines
Quoted 68 lines
Quoted 12 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +12
-0+static bool drm_dep_queue_add_dep_cb (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +struct dma_fence *fence = q->dep.fence; + +lockdep_assert_held (&q->sched.lock); + +if (WARN_ON (fence->context == q->fence.context)) { +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; +return false; + }drivers/gpu/drm/dep/drm_dep_queue.c line 510
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ +if (!dma_fence_add_callback (q->dep.fence, &q->dep.cb, + drm_dep_queue_wakeup)) +return true; + +dma_fence_put (q->dep.fence); + q->dep.fence =NULL ; + +return false; +}In rust we can enforce that all callbacks take a reference to the fence automatically. If the callback is “forgotten” in a buggy path, it is automatically removed, and the fence is automatically signaled with
-ECANCELED.Quoted 41 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +41
-0+ +/** +* drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC queue +* @q: dep queue +* +* Peeks at the head of the SPSC queue and drains all resolved +* dependencies. If a dependency is still pending, installs a wakeup +* callback and returns NULL. On success pops the job and returns it. +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +* Return: next dispatchable job, or NULL if a dep is still pending. +*/ +static struct drm_dep_job *drm_dep_queue_pop_job (struct drm_dep_queue *q) +{ +struct spsc_node *node; +struct drm_dep_job *job; + +lockdep_assert_held (&q->sched.lock); + + node =spsc_queue_peek (&q->job.queue); +if (!node) +return NULL ; + + job = container_of(node, struct drm_dep_job, queue_node); + +while ((q->dep.fence =drm_dep_queue_job_dependency (q, job))) { +if (drm_dep_queue_add_dep_cb (q, job)) +return NULL ; + } + +spsc_queue_pop (&q->job.queue); + +return job; +} + +/* +* drm_dep_queue_get_unless_zero() - try to acquire a queue reference +* +* Workers use this instead of drm_dep_queue_get() to guard against the zombie +* state: the queue's refcount has already reached zero (async teardown is in +* flight) but a work item was queued before free_work had a chance to cancel
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/dep/drm_dep_queue.c line 556
Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +5
-0+* it. If kref_get_unless_zero() fails the caller must bail immediately. +* +* Context: Any context. +* Returns true if the reference was acquired, false if the queue is zombie. +*/ Again, this function is totally gone in Rust.
See above. I don't think it is given the async teardown flow design.
Quoted 1,035 lines
Quoted 1,035 lines
Quoted 1,021 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +1021
-0+bool drm_dep_queue_get_unless_zero (struct drm_dep_queue *q) +{ +return kref_get_unless_zero (&q->refcount); +} +EXPORT_SYMBOL (drm_dep_queue_get_unless_zero); + +/** +* drm_dep_queue_run_job_work() - run-job worker +* @work: work item embedded in the dep queue +* +* Acquires @q->sched.lock, checks stopped state, queue readiness and +* available credits, pops the next job via drm_dep_queue_pop_job(), +* dispatches it via drm_dep_queue_run_job(), then re-kicks itself. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_run_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.run_job); +struct spsc_node *node; +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +mutex_lock (&q->sched.lock); + +if (drm_dep_queue_is_stopped (q)) + goto put_queue; + +if (!drm_dep_queue_is_ready (q)) + goto put_queue; + +/* Peek to check credits before committing to pop and dep resolution */ + node =spsc_queue_peek (&q->job.queue); +if (!node) + goto put_queue; + + job = container_of(node, struct drm_dep_job, queue_node); +if (!drm_dep_queue_has_credits (q, job)) + goto put_queue; + + job =drm_dep_queue_pop_job (q); +if (!job) + goto put_queue; + +drm_dep_queue_run_job (q, job); +drm_dep_queue_run_job_queue (q); + +put_queue: +mutex_unlock (&q->sched.lock); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/* +* drm_dep_queue_remove_job() - unlink a job from the pending list and reset TDR +* @q: dep queue owning @job +* @job: job to remove +* +* Splices @job out of @q->job.pending, cancels any pending TDR delayed work, +* and arms the timeout for the new list head (if any). +* +* Context: Process context. Must hold @q->sched.lock. DMA fence signaling path. +*/ +static void drm_dep_queue_remove_job (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->job.lock); + +list_del_init (&job->pending_link); +cancel_delayed_work (&q->sched.tdr); +drm_queue_start_timeout (q); +} + +/** +* drm_dep_queue_get_finished_job() - dequeue a finished job +* @q: dep queue +* +* Under @q->job.lock checks the head of the pending list for a +* finished dep fence. If found, removes the job from the list, +* cancels the TDR, and re-arms it for the new head. +* +* Context: Process context (workqueue). DMA fence signaling path. +* Return: the finished &drm_dep_job, or NULL if none is ready. +*/ +static struct drm_dep_job * +drm_dep_queue_get_finished_job (struct drm_dep_queue *q) +{ +struct drm_dep_job *job; + +guard (spinlock_irq)(&q->job.lock); + + job = list_first_entry_or_null(&q->job.pending, struct drm_dep_job, + pending_link); +if (job &&drm_dep_fence_is_finished (job->dfence)) +drm_dep_queue_remove_job (q, job); +else + job =NULL ; + +return job; +} + +/** +* drm_dep_queue_put_job_work() - put-job worker +* @work: work item embedded in the dep queue +* +* Drains all finished jobs by calling drm_dep_job_put() in a loop, +* then kicks the run-job worker. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because workqueue is shared with other items in the fence signaling path. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_put_job_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.put_job); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + +while ((job =drm_dep_queue_get_finished_job (q))) +drm_dep_job_put (job); + +drm_dep_queue_run_job_queue (q); + +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_queue_tdr_work() - TDR worker +* @work: work item embedded in the delayed TDR work +* +* Removes the head job from the pending list under @q->job.lock, +* asserts @q->ops->timedout_job is non-NULL, calls it outside the lock, +* requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the +* queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and always +* restarts the TDR timer after handling the job (unless @q is stopping). +* Any other return value triggers a WARN. +* +* The TDR is never armed when @q->ops->timedout_job is NULL, so firing +* this worker without a timedout_job callback is a driver bug. +* +* Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the +* queue is in zombie state (refcount already zero, async teardown in flight). +* +* Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling() +* because timedout_job() is expected to signal the guilty job's fence as part +* of reset. +* +* Context: Process context (workqueue). DMA fence signaling path. +*/ +static void drm_dep_queue_tdr_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, sched.tdr.work); +struct drm_dep_job *job; +bool cookie =dma_fence_begin_signalling (); + +/* Bail if queue is zombie (refcount already zero, teardown in flight). */ +if (!drm_dep_queue_get_unless_zero (q)) { +dma_fence_end_signalling (cookie); +return ; + } + + scoped_guard(spinlock_irq, &q->job.lock) { + job =list_first_entry_or_null (&q->job.pending, + struct drm_dep_job, + pending_link); +if (job) +/* +* Remove from pending so it cannot be freed +* concurrently by drm_dep_queue_get_finished_job() or +* .drm_dep_job_done(). +*/ +list_del_init (&job->pending_link); + } + +if (job) { +enum drm_dep_timedout_stat status; + +if (WARN_ON (!q->ops->timedout_job)) { +drm_dep_job_put (job); + goto out; + } + + status = q->ops->timedout_job (job); + +switch (status) { +case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB : + scoped_guard(spinlock_irq, &q->job.lock) +list_add (&job->pending_link, &q->job.pending); +drm_dep_queue_put_job_queue (q); +break ; +case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED : +drm_dep_job_put (job); +break ; +default : +WARN_ON ("invalid drm_dep_timedout_stat" ); +break ; + } + } + +out: +drm_queue_start_timeout_unlocked (q); +drm_dep_queue_put (q); +dma_fence_end_signalling (cookie); +} + +/** +* drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue +* @name: name for the workqueue +* @flags: DRM_DEP_QUEUE_FLAGS_* flags +* +* Allocates an ordered workqueue for job submission with %WQ_MEM_RECLAIM and +* %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use from +* memory reclaim context and properly annotated for lockdep taint tracking. +* Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When +* CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct * +drm_dep_alloc_submit_wq (const char *name,enum drm_dep_queue_flags flags) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +if (flags &DRM_DEP_QUEUE_FLAGS_HIGHPRI ) + wq_flags |=WQ_HIGHPRI ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_submit_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue +* @name: name for the workqueue +* +* Allocates an ordered workqueue for timeout detection and recovery with +* %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring consistent taint +* annotation with the submit workqueue. When CONFIG_LOCKDEP is enabled, uses +* a dedicated lockdep map for annotation. +* +* Context: Process context. +* Return: the new &workqueue_struct, or NULL on failure. +*/ +static struct workqueue_struct *drm_dep_alloc_timeout_wq (const char *name) +{ +unsigned int wq_flags =WQ_MEM_RECLAIM |WQ_MEM_WARN_ON_RECLAIM ; + +#if IS_ENABLED (CONFIG_LOCKDEP ) +static struct lockdep_map map = { + .name ="drm_dep_timeout_lockdep_map" + }; +return alloc_ordered_workqueue_lockdep_map (name, wq_flags, &map); +#else +return alloc_ordered_workqueue (name, wq_flags); +#endif +} + +/** +* drm_dep_queue_init() - initialize a dep queue +* @q: dep queue to initialize +* @args: initialization arguments +* +* Initializes all fields of @q from @args. If @args->submit_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL an +* ordered workqueue is allocated and owned by the queue +* (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds one kref +* reference and drm_dep_queue_put() must be called to drop this reference +* (i.e., drivers cannot directly free the queue). +* +* When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against the +* fs_reclaim pseudo-lock so that lockdep can detect any lock ordering +* inversion between @sched.lock and memory reclaim. +* +* Return: 0 on success, %-EINVAL when @args->credit_limit is zero, @args->ops +* is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when +* @args->submit_wq or @args->timeout_wq is non-NULL but was not allocated with +* %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails. +* +* Context: Process context. May allocate memory and create workqueues. +*/ +int drm_dep_queue_init (struct drm_dep_queue *q, +const struct drm_dep_queue_init_args *args) +{ +if (!args->credit_limit || !args->drm || !args->ops || + !args->ops->run_job) +return -EINVAL ; + +if (args->submit_wq && !workqueue_is_reclaim_annotated (args->submit_wq)) +return -EINVAL ; + +if (args->timeout_wq && + !workqueue_is_reclaim_annotated (args->timeout_wq)) +return -EINVAL ; + +memset (q,0 ,sizeof (*q)); + + q->name = args->name; + q->drm = args->drm; + q->credit.limit = args->credit_limit; + q->job.timeout = args->timeout ? args->timeout :MAX_SCHEDULE_TIMEOUT ; + +init_rcu_head (&q->rcu); +INIT_LIST_HEAD (&q->job.pending); +spin_lock_init (&q->job.lock); +spsc_queue_init (&q->job.queue); + +mutex_init (&q->sched.lock); +if (IS_ENABLED (CONFIG_LOCKDEP )) { +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&q->sched.lock); +fs_reclaim_release (GFP_KERNEL ); + } + +if (args->submit_wq) { + q->sched.submit_wq = args->submit_wq; + }else { + q->sched.submit_wq =drm_dep_alloc_submit_wq (args->name ?:"drm_dep" , + args->flags); +if (!q->sched.submit_wq) +return -ENOMEM ; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ; + } + +if (args->timeout_wq) { + q->sched.timeout_wq = args->timeout_wq; + }else { + q->sched.timeout_wq =drm_dep_alloc_timeout_wq (args->name ?:"drm_dep" ); +if (!q->sched.timeout_wq) + goto err_submit_wq; + + q->sched.flags |=DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ; + } + + q->sched.flags |= args->flags & + ~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ | +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ); + +INIT_DELAYED_WORK (&q->sched.tdr, drm_dep_queue_tdr_work); +INIT_WORK (&q->sched.run_job, drm_dep_queue_run_job_work); +INIT_WORK (&q->sched.put_job, drm_dep_queue_put_job_work); + + q->fence.context =dma_fence_context_alloc (1 ); + +kref_init (&q->refcount); + q->ops = args->ops; +drm_dev_get (q->drm); + +return 0 ; + +err_submit_wq: +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); +mutex_destroy (&q->sched.lock); + +return -ENOMEM ; +} +EXPORT_SYMBOL (drm_dep_queue_init); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* drm_dep_queue_push_job_begin() - mark the start of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the start of drm_dep_job_arm() and warns if the push context is +* already owned by another task, which would indicate concurrent arm/push on +* the same queue. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner); + q->job.push.owner = current; +} + +/** +* drm_dep_queue_push_job_end() - mark the end of an arm/push critical section +* @q: dep queue the job belongs to +* +* Called at the end of drm_dep_job_push() and warns if the push context is not +* owned by the current task, which would indicate a mismatched begin/end pair +* or a push from the wrong thread. +* +* No-op when CONFIG_PROVE_LOCKING is disabled. +* +* Context: Process context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +WARN_ON (q->job.push.owner != current); + q->job.push.owner =NULL ; +} +#endif + +/** +* drm_dep_queue_assert_teardown_invariants() - assert teardown invariants +* @q: dep queue being torn down +* +* Warns if the pending-job list, the SPSC submission queue, or the credit +* counter is non-zero when called, or if the queue still has a non-zero +* reference count. +* +* Context: Any context. +*/ +static void drm_dep_queue_assert_teardown_invariants (struct drm_dep_queue *q) +{ +WARN_ON (!list_empty (&q->job.pending)); +WARN_ON (spsc_queue_count (&q->job.queue)); +WARN_ON (atomic_read (&q->credit.count)); +WARN_ON (drm_dep_queue_refcount (q)); +} + +/** +* drm_dep_queue_release() - final internal cleanup of a dep queue +* @q: dep queue to clean up +* +* Asserts teardown invariants and destroys internal resources allocated by +* drm_dep_queue_init() that cannot be torn down earlier in the teardown +* sequence. Currently this destroys @q->sched.lock. +* +* Drivers that implement &drm_dep_queue_ops.release **must** call this +* function after removing @q from any internal bookkeeping (e.g. lookup +* tables or lists) but before freeing the memory that contains @q. When +* &drm_dep_queue_ops.release is NULL, drm_dep follows the default teardown +* path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_release (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); +mutex_destroy (&q->sched.lock); +} +EXPORT_SYMBOL (drm_dep_queue_release); + +/** +* drm_dep_queue_free() - final cleanup of a dep queue +* @q: dep queue to free +* +* Invokes &drm_dep_queue_ops.release if set, in which case the driver is +* responsible for calling drm_dep_queue_release() and freeing @q itself. +* If &drm_dep_queue_ops.release is NULL, calls drm_dep_queue_release() +* and then frees @q with kfree_rcu(). +* +* In either case, releases the drm_dev_get() reference taken at init time +* via drm_dev_put(), allowing the owning &drm_device to be unloaded once +* all queues have been freed. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free (struct drm_dep_queue *q) +{ +struct drm_device *drm = q->drm; + +if (q->ops->release) { + q->ops->release (q); + }else { +drm_dep_queue_release (q); +kfree_rcu (q, rcu); + } +drm_dev_put (drm); +} + +/** +* drm_dep_queue_free_work() - deferred queue teardown worker +* @work: free_work item embedded in the dep queue +* +* Runs on dep_free_wq. Disables all work items synchronously +* (preventing re-queue and waiting for in-flight instances), +* destroys any owned workqueues, then calls drm_dep_queue_free(). +* Running on dep_free_wq ensures destroy_workqueue() is never +* called from within one of the queue's own workers (deadlock) +* and disable_*_sync() cannot deadlock either. +* +* Context: Process context (workqueue), reclaim safe. +*/ +static void drm_dep_queue_free_work (struct work_struct *work) +{ +struct drm_dep_queue *q = + container_of(work, struct drm_dep_queue, free_work); + +drm_dep_queue_assert_teardown_invariants (q); + +disable_delayed_work_sync (&q->sched.tdr); +disable_work_sync (&q->sched.run_job); +disable_work_sync (&q->sched.put_job); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ ) +destroy_workqueue (q->sched.timeout_wq); + +if (q->sched.flags &DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ ) +destroy_workqueue (q->sched.submit_wq); + +drm_dep_queue_free (q); +} + +/** +* drm_dep_queue_fini() - tear down a dep queue +* @q: dep queue to tear down +* +* Asserts teardown invariants and nitiates teardown of @q by queuing the +* deferred free work onto tht module-private dep_free_wq workqueue. The work +* item disables any pending TDR and run/put-job work synchronously, destroys +* any workqueues that were allocated by drm_dep_queue_init(), and then releases +* the queue memory. +* +* Running teardown from dep_free_wq ensures that destroy_workqueue() is never +* called from within one of the queue's own workers (e.g. via +* drm_dep_queue_put()), which would deadlock. +* +* Drivers can wait for all outstanding deferred work to complete by waiting +* for the last drm_dev_put() reference on their &drm_device, which is +* released as the final step of each queue's teardown. +* +* Drivers that implement &drm_dep_queue_ops.fini **must** call this +* function after removing @q from any device bookkeeping but before freeing the +* memory that contains @q. When &drm_dep_queue_ops.fini is NULL, drm_dep +* follows the default teardown path and calls this function automatically. +* +* Context: Any context. +*/ +void drm_dep_queue_fini (struct drm_dep_queue *q) +{ +drm_dep_queue_assert_teardown_invariants (q); + +INIT_WORK (&q->free_work, drm_dep_queue_free_work); +queue_work (dep_free_wq, &q->free_work); +} +EXPORT_SYMBOL (drm_dep_queue_fini); + +/** +* drm_dep_queue_get() - acquire a reference to a dep queue +* @q: dep queue to acquire a reference on, or NULL +* +* Return: @q with an additional reference held, or NULL if @q is NULL. +* +* Context: Any context. +*/ +struct drm_dep_queue *drm_dep_queue_get (struct drm_dep_queue *q) +{ +if (q) +kref_get (&q->refcount); +return q; +} +EXPORT_SYMBOL (drm_dep_queue_get); + +/** +* __drm_dep_queue_release() - kref release callback for a dep queue +* @kref: kref embedded in the dep queue +* +* Calls &drm_dep_queue_ops.fini if set, otherwise calls +* drm_dep_queue_fini() to initiate deferred teardown. +* +* Context: Any context. +*/ +static void __drm_dep_queue_release (struct kref *kref) +{ +struct drm_dep_queue *q = + container_of(kref, struct drm_dep_queue, refcount); + +if (q->ops->fini) + q->ops->fini (q); +else +drm_dep_queue_fini (q); +} + +/** +* drm_dep_queue_put() - release a reference to a dep queue +* @q: dep queue to release a reference on, or NULL +* +* When the last reference is dropped, calls &drm_dep_queue_ops.fini if set, +* otherwise calls drm_dep_queue_fini(). Final memory release is handled by +* &drm_dep_queue_ops.release (which must call drm_dep_queue_release()) if set, +* or drm_dep_queue_release() followed by kfree_rcu() otherwise. +* Does nothing if @q is NULL. +* +* Context: Any context. +*/ +void drm_dep_queue_put (struct drm_dep_queue *q) +{ +if (q) +kref_put (&q->refcount, __drm_dep_queue_release); +} +EXPORT_SYMBOL (drm_dep_queue_put); + +/** +* drm_dep_queue_stop() - stop a dep queue from processing new jobs +* @q: dep queue to stop +* +* Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from finished +* fenced signaling context. Then cancels any in-flight run_job and put_job work +* items. Once stopped, the bypass path and the submit workqueue will not +* dispatch further jobs nor will any jobs be removed from the pending list. +* Call drm_dep_queue_start() to resume processing. +* +* Context: Process context. Waits for in-flight workers to complete. +*/ +void drm_dep_queue_stop (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +cancel_work_sync (&q->sched.run_job); +cancel_work_sync (&q->sched.put_job); +} +EXPORT_SYMBOL (drm_dep_queue_stop); + +/** +* drm_dep_queue_start() - resume a stopped dep queue +* @q: dep queue to start +* +* Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex) +* and @q->job.lock (spinlock_irq), making the flag safe to test from IRQ +* context. Then re-queues the run_job and put_job work items so that any jobs +* pending since the queue was stopped are processed. Must only be called after +* drm_dep_queue_stop(). +* +* Context: Process context. +*/ +void drm_dep_queue_start (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { + scoped_guard(spinlock_irq, &q->job.lock) +drm_dep_queue_flags_clear (q,DRM_DEP_QUEUE_FLAGS_STOPPED ); + } +drm_dep_queue_run_job_queue (q); +drm_dep_queue_put_job_queue (q); +} +EXPORT_SYMBOL (drm_dep_queue_start); + +/** +* drm_dep_queue_trigger_timeout() - trigger the TDR immediately for +* all pending jobs +* @q: dep queue to trigger timeout on +* +* Sets @q->job.timeout to 1 and arms the TDR delayed work with a one-jiffy +* delay, causing it to fire almost immediately without hot-spinning at zero +* delay. This is used to force-expire any pendind jobs on the queue, for +* example when the device is being torn down or has encountered an +* unrecoverable error. +* +* It is suggested that when this function is used, the first timedout_job call +* causes the driver to kick the queue off the hardware and signal all pending +* job fences. Subsequent calls continue to signal all pending job fences. +* +* Has no effect if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_trigger_timeout (struct drm_dep_queue *q) +{ +guard (spinlock_irqsave)(&q->job.lock); + q->job.timeout =1 ; +drm_queue_start_timeout (q); +} +EXPORT_SYMBOL (drm_dep_queue_trigger_timeout); + +/** +* drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait +* for it to finish +* @q: dep queue whose TDR to cancel +* +* Cancels the TDR delayed work item if it has not yet started, and waits for +* it to complete if it is already running. After this call returns, the TDR +* worker is guaranteed not to be executing and will not fire again until +* explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by a new +* job being submitted). +* +* Useful during error recovery or queue teardown when the caller needs to +* know that no timeout handling races with its own reset logic. +* +* Context: Process context. May sleep waiting for the TDR worker to finish. +*/ +void drm_dep_queue_cancel_tdr_sync (struct drm_dep_queue *q) +{ +cancel_delayed_work_sync (&q->sched.tdr); +} +EXPORT_SYMBOL (drm_dep_queue_cancel_tdr_sync); + +/** +* drm_dep_queue_resume_timeout() - restart the TDR timer with the +* configured timeout +* @q: dep queue to resume the timeout for +* +* Restarts the TDR delayed work using @q->job.timeout. Called after device +* recovery to give pending jobs a fresh full timeout window. Has no effect +* if the pending list is empty. +* +* Context: Any context. +*/ +void drm_dep_queue_resume_timeout (struct drm_dep_queue *q) +{ +drm_queue_start_timeout_unlocked (q); +} +EXPORT_SYMBOL (drm_dep_queue_resume_timeout); + +/** +* drm_dep_queue_is_stopped() - check whether a dep queue is stopped +* @q: dep queue to check +* +* Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false otherwise. +* +* Context: Any context. +*/ +bool drm_dep_queue_is_stopped (struct drm_dep_queue *q) +{ +return !!(q->sched.flags &DRM_DEP_QUEUE_FLAGS_STOPPED ); +} +EXPORT_SYMBOL (drm_dep_queue_is_stopped); + +/** +* drm_dep_queue_kill() - kill a dep queue and flush all pending jobs +* @q: dep queue to kill +* +* Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock. If a +* dependency fence is currently being waited on, its callback is removed and +* the run-job worker is kicked immediately so that the blocked job drains +* without waiting. +* +* Once killed, drm_dep_queue_job_dependency() returns NULL for all jobs, +* bypassing dependency waits so that every queued job drains through +* &drm_dep_queue_ops.run_job without blocking. +* +* The &drm_dep_queue_ops.run_job callback is guaranteed to be called for every +* job that was pushed before or after drm_dep_queue_kill(), even during queue +* teardown. Drivers should use this guarantee to perform any necessary +* bookkeeping cleanup without executing the actual backend operation when the +* queue is killed. +* +* Unlike drm_dep_queue_stop(), killing is one-way: there is no corresponding +* start function. +* +* **Driver safety requirement** +* +* drm_dep_queue_kill() must only be called once the driver can guarantee that +* no job in the queue will touch memory associated with any of its fences +* (i.e., the queue has been removed from the device and will never be put back +* on). +* +* Context: Process context. +*/ +void drm_dep_queue_kill (struct drm_dep_queue *q) +{ + scoped_guard(mutex, &q->sched.lock) { +struct dma_fence *fence; + +drm_dep_queue_flags_set (q,DRM_DEP_QUEUE_FLAGS_KILLED ); + +/* +* Holding &q->sched.lock guarantees that the run-job work item +* cannot drop its reference to q->dep.fence concurrently, so +* reading q->dep.fence here is safe. +*/ + fence =READ_ONCE (q->dep.fence); +if (fence &&dma_fence_remove_callback (fence, &q->dep.cb)) +drm_dep_queue_remove_dependency (q, fence); + } +} +EXPORT_SYMBOL (drm_dep_queue_kill); + +/** +* drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Drivers may use this to queue their own work items alongside the queue's +* internal run-job and put-job workers — for example to process incoming +* messages in the same serialisation domain. +* +* Prefer drm_dep_queue_work_enqueue() when the only need is to enqueue a +* work item, as it additionally checks the stopped state. Use this accessor +* when the workqueue itself is required (e.g. for alloc_ordered_workqueue +* replacement or drain_workqueue calls). +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for job submission. +*/ +struct workqueue_struct *drm_dep_queue_submit_wq (struct drm_dep_queue *q) +{ +return q->sched.submit_wq; +} +EXPORT_SYMBOL (drm_dep_queue_submit_wq); + +/** +* drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a dep queue +* @q: dep queue whose workqueue to retrieve +* +* Returns the workqueue used by @q to run TDR (timeout detection and recovery) +* work. Drivers may use this to queue their own timeout-domain work items, or +* to call drain_workqueue() when tearing down and needing to ensure all pending +* timeout callbacks have completed before proceeding. +* +* Context: Any context. +* Return: the &workqueue_struct used by @q for TDR work. +*/ +struct workqueue_struct *drm_dep_queue_timeout_wq (struct drm_dep_queue *q) +{ +return q->sched.timeout_wq; +} +EXPORT_SYMBOL (drm_dep_queue_timeout_wq); + +/** +* drm_dep_queue_work_enqueue() - queue work on the dep queue's submit workqueue +* @q: dep queue to enqueue work on +* @work: work item to enqueue +* +* Queues @work on @q->sched.submit_wq if the queue is not stopped. This +* allows drivers to schedule custom work items that run serialised with the +* queue's own run-job and put-job workers. +* +* Return: true if the work was queued, false if the queue is stopped or the +* work item was already pending. +* +* Context: Any context. +*/ +bool drm_dep_queue_work_enqueue (struct drm_dep_queue *q, +struct work_struct *work) +{ +if (drm_dep_queue_is_stopped (q)) +return false; + +return queue_work (q->sched.submit_wq, work); +} +EXPORT_SYMBOL (drm_dep_queue_work_enqueue); + +/** +* drm_dep_queue_can_job_bypass() - test whether a job can skip the SPSC queue +* @q: dep queue +* @job: job to test +* +* A job may bypass the submit workqueue and run inline on the calling thread +* if all of the following hold: +* +* - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue +* - the queue is not stopped +* - the SPSC submission queue is empty (no other jobs waiting) +* - the queue has enough credits for @job +* - @job has no unresolved dependency fences +* +* Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). +* Return: true if the job may be run inline, false otherwise. +*/ +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job) +{ +lockdep_assert_held (&q->sched.lock); + +return q->sched.flags &DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED && + !drm_dep_queue_is_stopped (q) && + !spsc_queue_count (&q->job.queue) && +drm_dep_queue_has_credits (q, job) && +xa_empty (&job->dependencies); +} + +/** +* drm_dep_job_done() - mark a job as complete +* @job: the job that finished +* @result: error code to propagate, or 0 for success +* +* Subtracts @job->credits from the queue credit counter, then signals the +* job's dep fence with @result. +* +* When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), a +* temporary extra reference is taken on @job before signalling the fence. +* This prevents a concurrent put-job worker — which may be woken by timeouts or +* queue starting — from freeing the job while this function still holds a +* pointer to it. The extra reference is released at the end of the function. +* +* After signalling, the IRQ-safe path removes the job from the pending list +* under @q->job.lock, provided the queue is not stopped. Removal is skipped +* when the queue is stopped so that drm_dep_queue_for_each_pending_job() can +* iterate the list without racing with the completion path. On successful +* removal, kicks the run-job worker so the next queued job can be dispatched +* immediately, then drops the job reference. If the job was already removed +* by TDR, or removal was skipped because the queue is stopped, kicks the +* put-job worker instead to allow the deferred put to complete. +* +* Context: Any context. +*/ +static void drm_dep_job_done (struct drm_dep_job *job,int result) +{ +struct drm_dep_queue *q = job->q; +bool irq_safe =drm_dep_queue_is_job_put_irq_safe (q), removed = false; + +/* +* Local ref to ensure the put worker—which may be woken by external +* forces (TDR, driver-side queue starting)—doesn't free the job behind +* this function's back after drm_dep_fence_done() while it is still on +* the pending list. +*/ +if (irq_safe) +drm_dep_job_get (job); + +atomic_sub (job->credits, &q->credit.count); +drm_dep_fence_done (job->dfence, result); + +/* Only safe to touch job after fence signal if we have a local ref. */ + +if (irq_safe) { + scoped_guard(spinlock_irqsave, &q->job.lock) { + removed = !list_empty (&job->pending_link) && + !drm_dep_queue_is_stopped (q); + +/* Guard against TDR operating on job */ +if (removed) +drm_dep_queue_remove_job (q, job); + } + } + +if (removed) { +drm_dep_queue_run_job_queue (q); +drm_dep_job_put (job); + }else { +drm_dep_queue_put_job_queue (q); + } + +if (irq_safe) +drm_dep_job_put (job); +} + +/** +* drm_dep_job_done_cb() - dma_fence callback to complete a job +* @f: the hardware fence that signalled +* @cb: fence callback embedded in the dep job +* +* Extracts the job from @cb and calls drm_dep_job_done() with +* @f->error as the result. +* +* Context: Any context, but with IRQ disabled. May not sleep. +*/ +static void drm_dep_job_done_cb (struct dma_fence *f,struct dma_fence_cb *cb) +{ + struct drm_dep_job *job = container_of(cb, struct drm_dep_job, cb); + +drm_dep_job_done (job, f->error); +} + +/** +* drm_dep_queue_run_job() - submit a job to hardware and set up +* completion tracking +* @q: dep queue +* @job: job to run +* +* Accounts @job->credits against the queue, appends the job to the pending +* list, then calls @q->ops->run_job(). The TDR timer is started only when +* @job is the first entry on the pending list; subsequent jobs added while +* a TDR is already in flight do not reset the timer (which would otherwise +* extend the deadline for the already-running head job). Stores the returned +* hardware fence as the parent of the job's dep fence, then installs +* drm_dep_job_done_cb() on it. If the hardware fence is already signalled +* (%-ENOENT from dma_fence_add_callback()) or run_job() returns NULL/error, +* the job is completed immediately. Must be called under @q->sched.lock. +* +* Context: Process context. Must hold @q->sched.lock (a mutex). DMA fence +* signaling path. +*/ +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +struct dma_fence *fence; +int r; + +lockdep_assert_held (&q->sched.lock); + +drm_dep_job_get (job); +atomic_add (job->credits, &q->credit.count); + + scoped_guard(spinlock_irq, &q->job.lock) { +bool first =list_empty (&q->job.pending); + +list_add_tail (&job->pending_link, &q->job.pending); +if (first) +drm_queue_start_timeout (q); + } + + fence = q->ops->run_job (job); +drm_dep_fence_set_parent (job->dfence, fence); + +if (!IS_ERR_OR_NULL (fence)) { + r =dma_fence_add_callback (fence, &job->cb, + drm_dep_job_done_cb); +if (r == -ENOENT ) +drm_dep_job_done (job, fence->error); +else if (r) +drm_err (q->drm,"fence add callback failed (%d)\n" , r); +dma_fence_put (fence);drivers/gpu/drm/dep/drm_dep_queue.c line 1587
Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +10
-0+ }else { +drm_dep_job_done (job,IS_ERR (fence) ?PTR_ERR (fence) :0 ); + } + +/* +* Drop all input dependency fences now, in process context, before the +* final job put. Once the job is on the pending list its last reference +* may be dropped from a dma_fence callback (IRQ context), where calling +* xa_destroy() would be unsafe. +*/ I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?
Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
I could drop this, but conceptually it still feels like the right approach.
Quoted 206 lines
Quoted 206 lines
Quoted 206 lines
drivers/gpu/drm/dep/drm_dep_queue.c new file +60
-0+drm_dep_job_drop_dependencies (job); +drm_dep_job_put (job); +} + +/** +* drm_dep_queue_push_job() - enqueue a job on the SPSC submission queue +* @q: dep queue +* @job: job to push +* +* Pushes @job onto the SPSC queue. If the queue was previously empty +* (i.e. this is the first pending job), kicks the run_job worker so it +* processes the job promptly without waiting for the next wakeup. +* May be called with or without @q->sched.lock held. +* +* Context: Any context. DMA fence signaling path. +*/ +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job) +{ +/* +* spsc_queue_push() returns true if the queue was previously empty, +* i.e. this is the first pending job. Kick the run_job worker so it +* picks it up without waiting for the next wakeup. +*/ +if (spsc_queue_push (&q->job.queue, &job->queue_node)) +drm_dep_queue_run_job_queue (q); +} + +/** +* drm_dep_init() - module initialiser +* +* Allocates the module-private dep_free_wq unbound workqueue used for +* deferred queue teardown. +* +* Return: 0 on success, %-ENOMEM if workqueue allocation fails. +*/ +static int __init drm_dep_init (void ) +{ + dep_free_wq =alloc_workqueue ("drm_dep_free" ,WQ_UNBOUND ,0 ); +if (!dep_free_wq) +return -ENOMEM ; + +return 0 ; +} + +/** +* drm_dep_exit() - module exit +* +* Destroys the module-private dep_free_wq workqueue. +*/ +static void __exit drm_dep_exit(void) +{ +destroy_workqueue (dep_free_wq); + dep_free_wq =NULL ; +} + +module_init (drm_dep_init); +module_exit (drm_dep_exit); + +MODULE_DESCRIPTION ("DRM dependency queue" ); +MODULE_LICENSE ("Dual MIT/GPL" );drivers/gpu/drm/dep/drm_dep_queue.h new file +31
-0@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_QUEUE_H_ +#define _DRM_DEP_QUEUE_H_ + +#include <linux/types.h> + +struct drm_dep_job ; +struct drm_dep_queue ; + +bool drm_dep_queue_can_job_bypass (struct drm_dep_queue *q, +struct drm_dep_job *job); +void drm_dep_queue_run_job (struct drm_dep_queue *q,struct drm_dep_job *job); +void drm_dep_queue_push_job (struct drm_dep_queue *q,struct drm_dep_job *job); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void drm_dep_queue_push_job_begin (struct drm_dep_queue *q); +void drm_dep_queue_push_job_end (struct drm_dep_queue *q); +#else +static inline void drm_dep_queue_push_job_begin (struct drm_dep_queue *q) +{ +} +static inline void drm_dep_queue_push_job_end (struct drm_dep_queue *q) +{ +} +#endif + +#endif /* _DRM_DEP_QUEUE_H_ */ include/drm/drm_dep.h new file +103
-0@@ -0,0 +1,597 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright 2015 Advanced Micro Devices, Inc. +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice shall be included in +* all copies or substantial portions of the Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR +* OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +* ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +* OTHER DEALINGS IN THE SOFTWARE. +* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _DRM_DEP_H_ +#define _DRM_DEP_H_ + +#include <drm/spsc_queue.h> +#include <linux/dma-fence.h> +#include <linux/xarray.h> +#include <linux/workqueue.h> + +enum dma_resv_usage ; +struct dma_resv ; +struct drm_dep_fence ; +struct drm_dep_job ; +struct drm_dep_queue ; +struct drm_file ; +struct drm_gem_object ; + +/** +* enum drm_dep_timedout_stat - return value of &drm_dep_queue_ops.timedout_job +* @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's finished +* fence during reset; drm_dep may safely drop its reference to the job. +* @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; reinsert the +* job at the head of the pending list so it can complete normally. +*/ +enum drm_dep_timedout_stat { +DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED , +DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB , +}; + +/** +* struct drm_dep_queue_ops - driver callbacks for a dep queue +*/ +struct drm_dep_queue_ops { +/** +* @run_job: submit the job to hardware. Returns the hardware completion +* fence (with a reference held for the scheduler), or NULL/ERR_PTR on +* synchronous completion or error. +*/ +struct dma_fence *(*run_job)(struct drm_dep_job *job); + +/** +* @timedout_job: called when the TDR fires for the head job. Must stop +* the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if the +* job's fence was signalled during reset, or +* %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or +* signalling was otherwise delayed, and the job should be re-inserted +* at the head of the pending list. Any other value triggers a WARN. +*/ +enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job); + +/** +* @release: called when the last kref on the queue is dropped and +* drm_dep_queue_fini() has completed. The driver is responsible for +* removing @q from any internal bookkeeping, calling +* drm_dep_queue_release(), and then freeing the memory containing @q +* (e.g. via kfree_rcu() using @q->rcu). If NULL, drm_dep calls +* drm_dep_queue_release() and frees @q automatically via kfree_rcu(). +* Use this when the queue is embedded in a larger structure. +*/ +void (*release)(struct drm_dep_queue *q); + +/** +* @fini: if set, called instead of drm_dep_queue_fini() when the last +* kref is dropped. The driver is responsible for calling +* drm_dep_queue_fini() itself after it is done with the queue. Use this +* when additional teardown logic must run before fini (e.g., cleanup +* firmware resources associated with the queue). +*/ +void (*fini)(struct drm_dep_queue *q); +}; + +/** +* enum drm_dep_queue_flags - flags for &drm_dep_queue and +* &drm_dep_queue_init_args +* +* Flags are divided into three categories: +* +* - **Private static**: set internally at init time and never changed. +* Drivers must not read or write these.
Quoted 8 lines
Quoted 8 lines
include/drm/drm_dep.h line 108
Quoted 5 lines
include/drm/drm_dep.h new file +5
-0+* %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ, +* %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ. +* +* - **Public dynamic**: toggled at runtime by drivers via accessors. +* Any modification must be performed under &drm_dep_queue.sched.lock. Can’t enforce that in C.
I agree. There are no true “private” fields in C if the object lives in a shared header file. I’d love to make drm_dep_queue and drm_dep_job private (defined in a C file), but then you can’t embed these objects inside driver objects—which is the primary use case. The best we can do is simply refuse to accept drivers that touch fields they shouldn’t, so the code can remain maintainable.
I did, however, make drm_dep_fence a private object—notice it’s defined in drm_dep_fence.c, so no one can abuse it.
Quoted 63 lines
Quoted 63 lines
include/drm/drm_dep.h line 115
Quoted 8 lines
include/drm/drm_dep.h new file +7
-0+* Accessor functions provide unstable reads. +* %DRM_DEP_QUEUE_FLAGS_STOPPED, +* %DRM_DEP_QUEUE_FLAGS_KILLED. +* +* - **Public static**: supplied by the driver in +* &drm_dep_queue_init_args.flags at queue creation time and not modified +* thereafter. Same here.
Quoted 52 lines
+ * %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED, + * %DRM_DEP_QUEUE_FLAGS_HIGHPRI, + * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE.+ *
+ * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit workqueue was
+ * allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout workqueue
+ * was allocated bydrm_dep_queue_init()and will be destroyed by
+ *drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is stopped and+ * will not dispatch new jobs or remove jobs from the pending list, dropping + * the drm_dep-owned reference. Set by drm_dep_queue_stop(), cleared by + * drm_dep_queue_start(). + * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been killed + * via drm_dep_queue_kill(). Any active dependency wait is cancelled + * immediately. Jobs continue to flow through run_job for bookkeeping + * cleanup, but dependency waiting is skipped so that queued work drains + * as quickly as possible. + * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue supports + * the bypass path where eligible jobs skip the SPSC queue and run inline. + * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit workqueue owned + * by the queue is created with %WQ_HIGHPRI, causing run-job and put-job + * workers to execute at elevated priority. Only privileged clients (e.g. + * drivers managing time-critical or real-time GPU contexts) should request + * this flag; granting it to unprivileged userspace would allow priority + * inversion attacks. + * @drm_dep_queue_init_args.submit_wq is provided. + * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set, + * drm_dep_job_done() may be called from hardirq context (e.g. from a + * hardware-signalled dma_fence callback). drm_dep_job_done() will directly + * dequeue the job and call drm_dep_job_put() without deferring to a + * workqueue. The driver's &drm_dep_job_ops.release callback must therefore + * be safe to invoke from IRQ context.+ */
include/drm/drm_dep.h new file +16
-0+enum drm_dep_queue_flags { +DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ =BIT (0 ), +DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ =BIT (1 ), +DRM_DEP_QUEUE_FLAGS_STOPPED =BIT (2 ), +DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED =BIT (3 ), +DRM_DEP_QUEUE_FLAGS_HIGHPRI =BIT (4 ), +DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE =BIT (5 ), +DRM_DEP_QUEUE_FLAGS_KILLED =BIT (6 ), +}; + +/** +* struct drm_dep_queue - a dependency-tracked GPU submission queue +* +* Combines the role of &drm_gpu_scheduler and &drm_sched_entity into a single +* object. Each queue owns a submit workqueue (or borrows one), a timeout +* workqueue, an SPSC submission queue, and a pending-job list used for TDR.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
+ *
+ * Initialise withdrm_dep_queue_init(), tear down withdrm_dep_queue_fini().
+ * Reference counted viadrm_dep_queue_get()/drm_dep_queue_put().
+ *
+ * All fields are **opaque to drivers**. Do not read or write any fieldCan’t enforce this in C.
Just answer above, agree.
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
+ * directly; use the provided helper functions instead. The sole exception
+ * is @rcu, which drivers may pass tokfree_rcu()when the queue is embedded
+ * inside a larger driver-managed structure and the &drm_dep_queue_ops.release
+ * vfunc performs an RCU-deferred free.+ */
+structdrm_dep_queue{
+ /** @ops: driver callbacks, set at init time. */
+ const structdrm_dep_queue_ops*ops;
+ /** @name: human-readable name used for workqueue and fence naming. */
+ const char *name;
+ /** @drm: owning DRM device; adrm_dev_get()reference is held for the
+ * lifetime of the queue to prevent module unload while queues are live.
+ */
+ structdrm_device*drm;
+ /** @refcount: reference count; use drm_dep_queue_get/put(). */
+ struct kref refcount;
+ /**
+ * @free_work: deferred teardown work queued unconditionally by
+ *drm_dep_queue_fini()onto the module-privatedep_free_wq. The work
+ * item disables pending workers synchronously and destroys any owned
+ * workqueues before releasing the queue memory and dropping the
+ *drm_dev_get()reference. Running ondep_free_wqensures
+ *destroy_workqueue()is never called from within one of the queue's
+ * own workers.
+ */
+ structwork_structfree_work;
+ /**
+ * @rcu: RCU head for deferred freeing.
+ *
+ * This is the **only** field drivers may access directly. When the
Quoted 2 lines
Quoted 2 lines
We can enforce this in Rust at compile time.
That is nice.
Quoted 169 lines
Quoted 169 lines
Quoted 88 lines
+ * queue is embedded in a larger structure, implement
+ * &drm_dep_queue_ops.release, calldrm_dep_queue_release()to destroy
+ * internal resources, then pass this field tokfree_rcu()so that any
+ * in-flight RCU readers referencing the queue'sdma_fencetimeline name
+ * complete before the memory is returned. All other fields must be
+ * accessed through the provided helpers.
+ */
+ structrcu_headrcu;include/drm/drm_dep.h new file +80
-0+ +/** @sched: scheduling and workqueue state. */ +struct { +/** @sched.submit_wq: ordered workqueue for run/put-job work. */ +struct workqueue_struct *submit_wq; +/** @sched.timeout_wq: workqueue for the TDR delayed work. */ +struct workqueue_struct *timeout_wq; +/** +* @sched.run_job: work item that dispatches the next queued +* job. +*/ +struct work_struct run_job; +/** @sched.put_job: work item that frees finished jobs. */ +struct work_struct put_job; +/** @sched.tdr: delayed work item for timeout/reset (TDR). */ +struct delayed_work tdr; +/** +* @sched.lock: mutex serialising job dispatch, bypass +* decisions, stop/start, and flag updates. +*/ +struct mutex lock; +/** +* @sched.flags: bitmask of &enum drm_dep_queue_flags. +* Any modification after drm_dep_queue_init() must be +* performed under @sched.lock. +*/ +enum drm_dep_queue_flags flags; + } sched; + +/** @job: pending-job tracking state. */ +struct { +/** +* @job.pending: list of jobs that have been dispatched to +* hardware and not yet freed. Protected by @job.lock. +*/ +struct list_head pending; +/** +* @job.queue: SPSC queue of jobs waiting to be dispatched. +* Producers push via drm_dep_queue_push_job(); the run_job +* work item pops from the consumer side. +*/ +struct spsc_queue queue; +/** +* @job.lock: spinlock protecting @job.pending, TDR start, and +* the %DRM_DEP_QUEUE_FLAGS_STOPPED flag. Always acquired with +* irqsave (spin_lock_irqsave / spin_unlock_irqrestore) to +* support %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE queues where +* drm_dep_job_done() may run from hardirq context. +*/ +spinlock_t lock; +/** +* @job.timeout: per-job TDR timeout in jiffies. +* %MAX_SCHEDULE_TIMEOUT means no timeout. +*/ +long timeout; +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* @job.push: lockdep annotation tracking the arm-to-push +* critical section. Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
include/drm/drm_dep.h line 278
Quoted 10 lines
include/drm/drm_dep.h new file +10
-0+*/ +struct { +/* +* @job.push.owner: task that currently holds the push +* context, used to assert single-owner invariants. +* NULL when idle. +*/ +struct task_struct *owner; + } push; +#endif We can enforce this in Rust automatically.
Quoted 68 lines
+ */
+ structdma_fence*removed_fence;
+ /** @dep.cb: callback installed on @dep.fence. */
+ structdma_fence_cbcb;
+ } dep;include/drm/drm_dep.h new file +63
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
+struct
drm_dep_job_ops{
+ /**
+ * @release: called when the last reference to the job is dropped.
+ *
+ * If set, the driver is responsible for freeing the job. If NULL,And if they don’t?
They leak memory.
Quoted 85 lines
Quoted 85 lines
By the way, we can also enforce this in Rust.
Quoted 83 lines
+ *
drm_dep_job_put()will callkfree()on the job directly.
+ */
+ void (*release)(structdrm_dep_job*job);include/drm/drm_dep.h new file +80
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
+ * Drivers that set %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED and wish to
+ * serialise their own submit work against the bypass path must acquire this
+ * guard. Without it, a concurrent caller ofdrm_dep_job_push()could take
+ * the bypass path and callops->run_job()inline between the driver's
+ * eligibility check and its corresponding action, producing a race.So if you’re not careful, you have just introduced a race :/
Luckily I’m careful. The use case here is compositors, compute workloads, or servicing KMD page faults—none of which have input dependencies and all of which require very low latency and minimal jitter.
Quoted 147 lines
Quoted 147 lines
include/drm/drm_dep.h
Quoted 89 lines
+ *
+ * **Constraint: only fromsubmit_wqworker context**
+ *
+ * This guard must only be acquired from a work item running on the queue's
+ * submit workqueue (@q->sched.submit_wq) by drivers.
+ *
+ * Context: Process context only; must be called fromsubmit_wqwork by
+ * drivers.
+ */include/drm/drm_dep.h new file +80
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Not enforceable in C.
Quoted 51 lines
include/drm/drm_dep.h new file +48
-0Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.34.1By the way:
Quoted 5 lines
Quoted 5 lines
I invite you to have a look at this implementation [0]. It currently works in real hardware i.e.: our downstream "Tyr" driver for Arm Mali is using that at the moment. It is a mere prototype that we’ve put together to test different approaches, so it’s not meant to be a “solution” at all. It’s a mere data point for further discussion.
I think some of the things I pointed out—async teardown, bypass paths, and dropping job refs in IRQ context—would still need to be added, though.
Quoted 5 lines
Quoted 5 lines
Philip Stanner is working on this “Job Queue” concept too, but from an upstream perspective.
[0]: https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61
I scanned [0], it looks signicantly better than the post upstream. Let me dig in a bit more.
Matt
Matt
Part of the diff is not shown here. View the raw message.
Daniel Almeida
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
I still need to digest your answers above, there's quite a bit of information. Thanks for that. I'll do a pass on it tomorrow.
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
By the way:
I invite you to have a look at this implementation [0]. It currently works in real hardware i.e.: our downstream "Tyr" driver for Arm Mali is using that at the moment. It is a mere prototype that we’ve put together to test different approaches, so it’s not meant to be a “solution” at all. It’s a mere data point for further discussion.
I think some of the things I pointed out—async teardown, bypass paths, and dropping job refs in IRQ context—would still need to be added, though.
That’s ok, I suppose we can find a way to add these things if they’re needed in order to support other FW scheduling GPUs (i.e.: other than Mali, which is the only thing I tested on, and Nova, which I assume has very similar requirements).
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Philip Stanner is working on this “Job Queue” concept too, but from an upstream perspective.
Quoted 7 lines
Quoted 1 line
Quoted 1 line
[0]: https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61
I scanned [0], it looks signicantly better than the post upstream. Let me dig in a bit more.
Matt
Matt
One thing that is missing is that, at the moment, submit() is fallible and there is no preallocation. This can be added to the current design rather easily (i.e. by splitting into two different steps, a fallible prepare() where rollback is possible, and an infallible commit(), or whatever names get chosen).
Perhaps we can also split this into two types too, AtomicJobQueue and JobQueue, where only the first one allows refs to be dropped in IRQ context; i.e.: since we do not need this in Tyr, and not allowing this makes the design of the "non-atomic" version much simpler. Or perhaps we can figure out a way to ensure that we don't drop the last ref in IRQ context. I am just brainstorming some ideas at this point, and again, I still need to go through your explanations above.
— Daniel
Boris Brezillon
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Matthew BrostHi Matthew,
Just a few drive-by comments.
Quoted 20 lines · On Tue, 17 Mar 2026 11:14:36 -0700 Matthew Brost <[email protected]>…
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.Quoted 5 lines
Quoted 5 lines
In rust, when you queue work, you have to pass a reference-counted pointer (Arc<T>). We simply never have this problem in a Rust design. If there is work queued, the queue is alive.
By the way, why can’t we simply require synchronous teardowns?
Quoted 8 lines
Consider the case where the DRM dep queue’s refcount drops to zero, but the device firmware still holds references to the associated queue. These are resources that must be torn down asynchronously. In Xe, I need to send two asynchronous firmware commands before I can safely remove the memory associated with the queue (faulting on this kind of global memory will take down the device) and recycle the firmware ID tied to the queue. These async commands are issued on the driver side, on the DRM dep queue’s workqueue as well.
Asynchronous teardown is okay, but I'm not too sure using the refcnt to know that the queue is no longer usable is the way to go. To me the refcnt is what determines when the SW object is no longer referenced by any other item in the code, and a work item acting on the queue counts as one owner of this queue. If you want to cancel the work in order to speed up the destruction of the queue, you can call {cancel,disable}_work[_sync](), and have the ref dropped if the cancel/disable was effective. Multi-step teardown is also an option, but again, the state of the queue shouldn't be determined from its refcnt IMHO.
Quoted 10 lines
Now consider a scenario where something goes wrong and those firmware commands never complete, and a device reset is required to recover. The driver’s per-queue tracking logic stops all queues (including zombie ones), determines which commands were lost, cleans up the side effects of that lost state, and then restarts all queues. That is how we would end up in this work item with a zombie queue. The restart logic could probably be made smart enough to avoid queueing work for zombie queues, but in my opinion it’s safe enough to use
kref_get_unless_zero()in the work items.
Well, that only works for single-step teardown, or when you enter the last step. At which point, I'm not too sure it's significantly better than encoding the state of the queue through a separate field, and have the job queue logic reject new jobs if the queue is no longer usable (shouldn't even be exposed to userland at this point though).
Quoted 6 lines
It should also be clear that a DRM dep queue is primarily intended to be embedded inside the driver’s own queue object, even though it is valid to use it as a standalone object. The async teardown flows are also optional features.
Quoted 9 lines
Let’s also consider a case where you do not need the async firmware flows described above, but the DRM dep queue is still embedded in a driver-side object that owns memory via dma-resv. The final queue put may occur in IRQ context (DRM dep avoids kicking a worker just to drop a refi as opt in), or in the reclaim path (any scheduler workqueue is the reclaim path). In either case, you cannot free memory there taking a dma-resv lock, which is why all DRM dep queues ultimately free their resources in a work item outside of reclaim. Many drivers already follow this pattern, but in DRM dep this behavior is built-in.
I agree deferred cleanup is the way to go.
Quoted 3 lines
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Rust won't magically defer the cleanup, nor will it dictate how you want to do the queue teardown, those are things you need to implement. But it should give visibility about object lifetimes, and guarantee that an object that's still visible to some owners is usable (the notion of usable is highly dependent on the object implementation).
Just a purely theoretical example of a multi-step queue teardown that might be possible to encode in rust:
- MyJobQueue<Usable>: The job queue is currently exposed and usable. There's a ::destroy() method consuming 'self' and returning a MyJobQueue<Destroyed> object
- MyJobQueue<Destroyed>: The user asked for the workqueue to be destroyed. No new job can be pushed. Existing jobs that didn't make it to the FW queue are cancelled, jobs that are in-flight are cancelled if they can, or are just waited upon if they can't. When the whole destruction step is done, ::destroyed() is called, it consumes 'self' and returns a MyJobQueue<Inactive> object.
- MyJobQueue<Inactive>: The queue is no longer active (HW doesn't have any resources on this queue). It's ready to be cleaned up. ::cleanup() (or just ::drop()) defers the cleanup of some inner object that has been passed around between the various MyJobQueue<State> wrappers.
Each of the state transition can happen asynchronously. A state transition consumes the object in one state, and returns a new object in its new state. None of the transition involves dropping a refcnt, ownership is just transferred. The final MyJobQueue<Inactive> object is the object we'll defer cleanup on.
It's a very high-level view of one way this can be implemented (I'm sure there are others, probably better than my suggestion) in order to make sure the object doesn't go away without the compiler enforcing proper state transitions.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each structdrm_dep_jobhas an associated structdrm_dep_fencethat
+ * provides a singledma_fence(@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent viadrm_dep_job_done_cb()and
+ * is signalled once @parent signals (or immediately ifrun_job()returns
+ * NULL or an error).
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here.
Hm, so that's a driver returning a dma_fence_array/chain through ::run_job()? Why would we not want to have them directly exposed and split up into singular fence objects at resv insertion time (I don't think syncobjs care, but I might be wrong). I mean, one of the point behind the container extraction is so fences coming from the same context/timeline can be detected and merged. If you insert the container through a proxy, you're defeating the whole fence merging optimization.
The second thing is that I'm not sure drivers were ever supposed to return fence containers in the first place, because the whole idea behind a fence context is that fences are emitted/signalled in seqno-order, and if the fence is encoding the state of multiple timelines that progress at their own pace, it becomes tricky to control that. I guess if it's always the same set of timelines that are combined, that would work.
Quoted 2 lines
One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
We do have that in Panthor, but that's all explicit: in a single SUBMIT, you can have multiple jobs targeting different queues, each of them having their own set of deps/signal ops. The combination of all the signal ops into a container is left to the UMD. It could be automated kernel side, but that would be a flag on the SIGNAL op leading to the creation of a fence_array containing fences from multiple submitted jobs, rather than the driver combining stuff in the fence it returns in ::run_job().
Quoted 7 lines
Another case is late arming of hardware fences in
run_job(which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available atrun_job, though.
Yep, I also think we can arm the driver fence early in the case of JobQueue. The reason it couldn't be done before is because the scheduler was in the middle, deciding which entity to pull the next job from, which was changing the seqno a job driver-fence would be assigned (you can't guess that at queue time in that case).
[...]
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue fromdrm_dep_job_init()until
+ *drm_dep_job_put()drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * indrm_dep_queue_run_job()untildrm_dep_job_done()kicks theput_job
+ * worker, which callsdrm_dep_job_put()to release that reference.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context)
Did you mean in !IRQ (or !atomic) context here? Feels weird to not defer the cleanup when you're in an IRQ/atomic context, but defer it when you're in a thread context.
Quoted 3 lines
so we avoid
switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.
Well, the cleanup path is probably not where latency matters the most. It's adding scheduling overhead, sure, but given all the stuff we defer already, I'm not too sure we're at saving a few cycles to get the cleanup done immediately. What's important to have is a way to signal fences in an atomic context, because this has an impact on latency.
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
+ /*
+ * Drop all input dependency fences now, in process context, before the
+ * final job put. Once the job is on the pending list its last reference
+ * may be dropped from adma_fencecallback (IRQ context), where calling
+ *xa_destroy()would be unsafe.
+ */I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
If it significantly hurts the perf, I'd like to understand why, because to me it looks like pure-cleanup (no signaling involved), and thus no other process waiting for us to do the cleanup. The only thing that might have an impact is how fast you release the resources, and given it's only a partial cleanup (xa_destroy() still has to be deferred), I'd like to understand which part of the immediate cleanup is causing a contention (basically which kind of resources the system is starving of)
Regards,
Boris
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 58 lines · Boris Brezillon
Hi Matthew,
Just a few drive-by comments.
Quoted 20 lines · On Tue, 17 Mar 2026 11:14:36 -0700 Matthew Brost <[email protected]>…
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds
q->job.timeoutjiffies, callingops->timedout_job().drm_dep_queue_trigger_timeout()forces immediate expiry for device teardown.IRQ-safe completion: queues flagged
DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFEallowdrm_dep_job_done()to be called from hardirq context (e.g. adma_fencecallback). Dependency cleanup is deferred to process context afterops->run_job()returns to avoid callingxa_destroy()from IRQ.Zombie-state guard: workers use
kref_get_unless_zero()on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.Quoted 5 lines
Quoted 5 lines
In rust, when you queue work, you have to pass a reference-counted pointer (Arc<T>). We simply never have this problem in a Rust design. If there is work queued, the queue is alive.
By the way, why can’t we simply require synchronous teardowns?
Quoted 8 lines
Consider the case where the DRM dep queue’s refcount drops to zero, but the device firmware still holds references to the associated queue. These are resources that must be torn down asynchronously. In Xe, I need to send two asynchronous firmware commands before I can safely remove the memory associated with the queue (faulting on this kind of global memory will take down the device) and recycle the firmware ID tied to the queue. These async commands are issued on the driver side, on the DRM dep queue’s workqueue as well.
Asynchronous teardown is okay, but I'm not too sure using the refcnt to know that the queue is no longer usable is the way to go. To me the refcnt is what determines when the SW object is no longer referenced by any other item in the code, and a work item acting on the queue counts as one owner of this queue. If you want to cancel the work in order to speed up the destruction of the queue, you can call {cancel,disable}_work[_sync](), and have the ref dropped if the cancel/disable was effective. Multi-step teardown is also an option, but again, the state of the queue shouldn't be determined from its refcnt IMHO.
Quoted 10 lines
Now consider a scenario where something goes wrong and those firmware commands never complete, and a device reset is required to recover. The driver’s per-queue tracking logic stops all queues (including zombie ones), determines which commands were lost, cleans up the side effects of that lost state, and then restarts all queues. That is how we would end up in this work item with a zombie queue. The restart logic could probably be made smart enough to avoid queueing work for zombie queues, but in my opinion it’s safe enough to use
kref_get_unless_zero()in the work items.
Quoted 6 lines
Well, that only works for single-step teardown, or when you enter the last step. At which point, I'm not too sure it's significantly better than encoding the state of the queue through a separate field, and have the job queue logic reject new jobs if the queue is no longer usable (shouldn't even be exposed to userland at this point though).
'shouldn't even be exposed to userland at this point though' - Yes.
The philosophy of ref counting design is roughly:
- When queue is created by userland call drm_dep_queue_init
- All jobs hold a ref to drm_dep_queue
- When userland close queue, remove it from the FD, call drm_dep_queue_put + initiatie teardown (I'd recommned just setting TDR to immediately fire, kick off queue from device in first fire + signal all fences).
- Queue refcount goes to zero optionality implement drm_dep_queue_ops.fini to keep the drm_dep_queue (and object it is embedded) around for a bit longer if additional firmware / device side resources are still around, call drm_dep_queue_fini when this part completes. If drm_dep_queue_ops.fini isn't implement the core implementation just calls drm_dep_queue_fini.
- Work item releases drm_dep_queue out dma-fence signaling path for safe memory release (e.g., take dma-resv locks).
Quoted 16 lines
Quoted 6 lines
It should also be clear that a DRM dep queue is primarily intended to be embedded inside the driver’s own queue object, even though it is valid to use it as a standalone object. The async teardown flows are also optional features.
Quoted 9 lines
Let’s also consider a case where you do not need the async firmware flows described above, but the DRM dep queue is still embedded in a driver-side object that owns memory via dma-resv. The final queue put may occur in IRQ context (DRM dep avoids kicking a worker just to drop a refi as opt in), or in the reclaim path (any scheduler workqueue is the reclaim path). In either case, you cannot free memory there taking a dma-resv lock, which is why all DRM dep queues ultimately free their resources in a work item outside of reclaim. Many drivers already follow this pattern, but in DRM dep this behavior is built-in.
Quoted 2 lines
I agree deferred cleanup is the way to go.
+1. Yes. I spot a bunch of drivers that open code this part driver side, Xe included.
Quoted 34 lines
Quoted 3 lines
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Rust won't magically defer the cleanup, nor will it dictate how you want to do the queue teardown, those are things you need to implement. But it should give visibility about object lifetimes, and guarantee that an object that's still visible to some owners is usable (the notion of usable is highly dependent on the object implementation).
Just a purely theoretical example of a multi-step queue teardown that might be possible to encode in rust:
- MyJobQueue<Usable>: The job queue is currently exposed and usable. There's a ::destroy() method consuming 'self' and returning a MyJobQueue<Destroyed> object
- MyJobQueue<Destroyed>: The user asked for the workqueue to be destroyed. No new job can be pushed. Existing jobs that didn't make it to the FW queue are cancelled, jobs that are in-flight are cancelled if they can, or are just waited upon if they can't. When the whole destruction step is done, ::destroyed() is called, it consumes 'self' and returns a MyJobQueue<Inactive> object.
- MyJobQueue<Inactive>: The queue is no longer active (HW doesn't have any resources on this queue). It's ready to be cleaned up. ::cleanup() (or just ::drop()) defers the cleanup of some inner object that has been passed around between the various MyJobQueue<State> wrappers.Each of the state transition can happen asynchronously. A state transition consumes the object in one state, and returns a new object in its new state. None of the transition involves dropping a refcnt, ownership is just transferred. The final MyJobQueue<Inactive> object is the object we'll defer cleanup on.
Quoted 5 lines
It's a very high-level view of one way this can be implemented (I'm sure there are others, probably better than my suggestion) in order to make sure the object doesn't go away without the compiler enforcing proper state transitions.
I'm sure Rust can implement this. My point about Rust is it doesn't magically solve hard software arch probles, but I will admit the ownership model, way it can enforce locking at compile time is pretty cool.
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each structdrm_dep_jobhas an associated structdrm_dep_fencethat
+ * provides a singledma_fence(@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent viadrm_dep_job_done_cb()and
+ * is signalled once @parent signals (or immediately ifrun_job()returns
+ * NULL or an error).Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here.
Quoted 4 lines
Hm, so that's a driver returning a dma_fence_array/chain through ::run_job()? Why would we not want to have them directly exposed and split up into singular fence objects at resv insertion time (I don't think syncobjs care, but I might be wrong). I mean, one of the point
You can stick dma-fence-arrays in syncobjs, but not chains.
Neither dma-fence-arrays/chain can go into dma-resv.
Hence why disconnecting a job's finished fence from hardware fence IMO is good idea to keep so gives drivers flexiblity on the hardware fences. e.g., If this design didn't have a job's finished fence, I'd have to open code one Xe side.
Quoted 4 lines
behind the container extraction is so fences coming from the same context/timeline can be detected and merged. If you insert the container through a proxy, you're defeating the whole fence merging optimization.
Right. Finished fences have single timeline too...
Quoted 8 lines
The second thing is that I'm not sure drivers were ever supposed to return fence containers in the first place, because the whole idea behind a fence context is that fences are emitted/signalled in seqno-order, and if the fence is encoding the state of multiple timelines that progress at their own pace, it becomes tricky to control that. I guess if it's always the same set of timelines that are combined, that would work.
Xe does this is definitely works. We submit to multiple rings, when all rings signal a seqno, a chain or array signals -> finished fence signals. The queues used in this manor can only submit multiple ring jobs so the finished fence timeline stays intact. If you could a multiple rings followed by a single ring submission on the same queue, yes this could break.
Quoted 4 lines
Quoted 3 lines
One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
Quoted 8 lines
We do have that in Panthor, but that's all explicit: in a single SUBMIT, you can have multiple jobs targeting different queues, each of them having their own set of deps/signal ops. The combination of all the signal ops into a container is left to the UMD. It could be automated kernel side, but that would be a flag on the SIGNAL op leading to the creation of a
fence_arraycontaining fences from multiple submitted jobs, rather than the driver combining stuff in the fence it returns in ::run_job().
See above. We have a dedicated queue type for these type of submissions and single job that submits to the all rings. We had multiple queue / jobs in the i915 to implemented this but it turns out it is much cleaner with a single queue / singler job / multiple rings model.
Quoted 9 lines
Quoted 8 lines
Another case is late arming of hardware fences in
run_job(which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available atrun_job, though.
Quoted 6 lines
Yep, I also think we can arm the driver fence early in the case of JobQueue. The reason it couldn't be done before is because the scheduler was in the middle, deciding which entity to pull the next job from, which was changing the seqno a job driver-fence would be assigned (you can't guess that at queue time in that case).
Xe doesn't need to late arming, but it look like multiple drivers to implement the late arming which may be required (?).
Quoted 20 lines
[...]
Quoted 18 lines
Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue fromdrm_dep_job_init()untilQuoted 8 lines
+ *
drm_dep_job_put()drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * indrm_dep_queue_run_job()untildrm_dep_job_done()kicks theput_job
+ * worker, which callsdrm_dep_job_put()to release that reference.Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
Quoted 7 lines
Quoted 2 lines
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context)
Did you mean in !IRQ (or !atomic) context here? Feels weird to not defer the cleanup when you're in an IRQ/atomic context, but defer it when you're in a thread context.
The put of a job in this design can be from an IRQ context (opt-in) feature. xa_destroy blows up if it is called from an IRQ context, although maybe that could be workaround.
Quoted 5 lines
Quoted 3 lines
so we avoid
switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.Well, the cleanup path is probably not where latency matters the most.
Agree. But I do think avoiding a CPU context switch (work item) for a very lightweight job cleanup (usually just drop refs) will save of CPU cycles, thus also things like power, etc...
Quoted 5 lines
It's adding scheduling overhead, sure, but given all the stuff we defer already, I'm not too sure we're at saving a few cycles to get the cleanup done immediately. What's important to have is a way to signal fences in an atomic context, because this has an impact on latency.
Yes. The signaling happens first then drm_dep_job_put if IRQ opt-in.
Quoted 21 lines
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
+ /*
+ * Drop all input dependency fences now, in process context, before the
+ * final job put. Once the job is on the pending list its last reference
+ * may be dropped from adma_fencecallback (IRQ context), where calling
+ *xa_destroy()would be unsafe.
+ */I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
Quoted 8 lines
If it significantly hurts the perf, I'd like to understand why, because to me it looks like pure-cleanup (no signaling involved), and thus no other process waiting for us to do the cleanup. The only thing that might have an impact is how fast you release the resources, and given it's only a partial cleanup (
xa_destroy()still has to be deferred), I'd like to understand which part of the immediate cleanup is causing a contention (basically which kind of resources the system is starving of)
It was more of once we moved to a ref counted model, it is pretty trivial allow drm_dep_job_put when the fence is signaling. It doesn't really add any complexity either, thus why I added it is.
Matt
Quoted 3 lines
Regards,
Boris
Boris Brezillon
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 40 lines · On Wed, 18 Mar 2026 15:40:35 -0700 Matthew Brost <[email protected]>…
Quoted 34 lines
Quoted 3 lines
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Rust won't magically defer the cleanup, nor will it dictate how you want to do the queue teardown, those are things you need to implement. But it should give visibility about object lifetimes, and guarantee that an object that's still visible to some owners is usable (the notion of usable is highly dependent on the object implementation).
Just a purely theoretical example of a multi-step queue teardown that might be possible to encode in rust:
- MyJobQueue<Usable>: The job queue is currently exposed and usable. There's a ::destroy() method consuming 'self' and returning a MyJobQueue<Destroyed> object
- MyJobQueue<Destroyed>: The user asked for the workqueue to be destroyed. No new job can be pushed. Existing jobs that didn't make it to the FW queue are cancelled, jobs that are in-flight are cancelled if they can, or are just waited upon if they can't. When the whole destruction step is done, ::destroyed() is called, it consumes 'self' and returns a MyJobQueue<Inactive> object.
- MyJobQueue<Inactive>: The queue is no longer active (HW doesn't have any resources on this queue). It's ready to be cleaned up. ::cleanup() (or just ::drop()) defers the cleanup of some inner object that has been passed around between the various MyJobQueue<State> wrappers.Each of the state transition can happen asynchronously. A state transition consumes the object in one state, and returns a new object in its new state. None of the transition involves dropping a refcnt, ownership is just transferred. The final MyJobQueue<Inactive> object is the object we'll defer cleanup on.
Quoted 5 lines
It's a very high-level view of one way this can be implemented (I'm sure there are others, probably better than my suggestion) in order to make sure the object doesn't go away without the compiler enforcing proper state transitions.
Quoted 4 lines
I'm sure Rust can implement this. My point about Rust is it doesn't magically solve hard software arch probles, but I will admit the ownership model, way it can enforce locking at compile time is pretty cool.
It's not quite about rust directly solving those problems for you, it's about rust forcing you to think about those problems in the first place. So no, rust won't magically solve your multi-step teardown with crazy CPU <-> Device synchronization etc, but it allows you to clearly identify those steps, and think about how you want to represent them without abusing other concepts, like object refcounting/ownership. Everything I described, you can code it in C BTW, it's just that C is so lax that you can also abuse other stuff to get to your ends, which might or might not be safe, but more importantly, will very likely obfuscate the code (even with good docs).
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each structdrm_dep_jobhas an associated structdrm_dep_fencethat
+ * provides a singledma_fence(@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent viadrm_dep_job_done_cb()and
+ * is signalled once @parent signals (or immediately ifrun_job()returns
+ * NULL or an error).Quoted 3 lines
Quoted 3 lines
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here.
Quoted 6 lines
Quoted 4 lines
Hm, so that's a driver returning a dma_fence_array/chain through ::run_job()? Why would we not want to have them directly exposed and split up into singular fence objects at resv insertion time (I don't think syncobjs care, but I might be wrong). I mean, one of the point
You can stick dma-fence-arrays in syncobjs, but not chains.
Yeah, kinda makes sense, since timeline syncobjs use chains, and if the chain reject inner chains, it won't work.
Quoted 2 lines
Neither dma-fence-arrays/chain can go into dma-resv.
They can't go directly in it, but those can be split into individual fences and be inserted, which would achieve the same goal.
Quoted 3 lines
Hence why disconnecting a job's finished fence from hardware fence IMO is good idea to keep so gives drivers flexiblity on the hardware fences.
The thing is, I'm not sure drivers were ever meant to expose containers through ::run_job().
Quoted 2 lines
e.g., If this design didn't have a job's finished fence, I'd have to open code one Xe side.
There might be other reasons we'd like to keep the drm_sched_fence-like proxy that I'm missing. But if it's the only one, and the fence-combining pattern you're describing is common to multiple drivers, we can provide a container implementation that's not a fence_array, so you can use it to insert driver fences into other containers. This way we wouldn't force the proxy model to all drivers, but we would keep the code generic/re-usable.
Quoted 7 lines
Quoted 5 lines
behind the container extraction is so fences coming from the same context/timeline can be detected and merged. If you insert the container through a proxy, you're defeating the whole fence merging optimization.
Right. Finished fences have single timeline too...
Aren't you faking a single timeline though if you combine fences from different engines running at their own pace into a container?
Quoted 10 lines
Quoted 9 lines
The second thing is that I'm not sure drivers were ever supposed to return fence containers in the first place, because the whole idea behind a fence context is that fences are emitted/signalled in seqno-order, and if the fence is encoding the state of multiple timelines that progress at their own pace, it becomes tricky to control that. I guess if it's always the same set of timelines that are combined, that would work.
Quoted 6 lines
Xe does this is definitely works. We submit to multiple rings, when all rings signal a seqno, a chain or array signals -> finished fence signals. The queues used in this manor can only submit multiple ring jobs so the finished fence timeline stays intact. If you could a multiple rings followed by a single ring submission on the same queue, yes this could break.
Okay, I had the same understanding, thanks for confirming.
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
Quoted 8 lines
We do have that in Panthor, but that's all explicit: in a single SUBMIT, you can have multiple jobs targeting different queues, each of them having their own set of deps/signal ops. The combination of all the signal ops into a container is left to the UMD. It could be automated kernel side, but that would be a flag on the SIGNAL op leading to the creation of a
fence_arraycontaining fences from multiple submitted jobs, rather than the driver combining stuff in the fence it returns in ::run_job().
Quoted 4 lines
See above. We have a dedicated queue type for these type of submissions and single job that submits to the all rings. We had multiple queue / jobs in the i915 to implemented this but it turns out it is much cleaner with a single queue / singler job / multiple rings model.
Hm, okay. It didn't turn into a mess in Panthor, but Xe is likely an order of magnitude more complicated that Mali, so I'll refrain from judging this design decision.
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Another case is late arming of hardware fences in
run_job(which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available atrun_job, though.Quoted 6 lines
Yep, I also think we can arm the driver fence early in the case of JobQueue. The reason it couldn't be done before is because the scheduler was in the middle, deciding which entity to pull the next job from, which was changing the seqno a job driver-fence would be assigned (you can't guess that at queue time in that case).
Quoted 2 lines
Xe doesn't need to late arming, but it look like multiple drivers to implement the late arming which may be required (?).
As I said, it's mostly a problem when you have a single-HW-queue:multiple-contexts model, which is exactly what drm_sched was designed for. I suspect early arming is not an issue for any of the HW supporting FW-based scheduling (PVR, Mali, NVidia, ...). If you want to use drm_dep for all drivers currently using drm_sched (I'm still not convinced this is a good idea to do that just yet, because then you're going to pull a lot of the complexity we're trying to get rid of), then you need late arming of driver fences.
Quoted 24 lines
Quoted 24 lines
[...]
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue fromdrm_dep_job_init()until
+ *drm_dep_job_put()drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * indrm_dep_queue_run_job()untildrm_dep_job_done()kicks theput_job
+ * worker, which callsdrm_dep_job_put()to release that reference.Quoted 3 lines
Quoted 3 lines
Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context)
Quoted 8 lines
Quoted 4 lines
Did you mean in !IRQ (or !atomic) context here? Feels weird to not defer the cleanup when you're in an IRQ/atomic context, but defer it when you're in a thread context.
The put of a job in this design can be from an IRQ context (opt-in) feature.
xa_destroyblows up if it is called from an IRQ context, although maybe that could be workaround.
Making it so _put() in IRQ context is safe is fine, what I'm saying is that instead of doing a partial immediate cleanup, and the rest in a worker, we can just defer everything: that is, have some _deref_release() function called by kref_put() that would queue a work item from which the actual release is done.
Quoted 10 lines
Quoted 6 lines
Quoted 4 lines
so we avoid
switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.Well, the cleanup path is probably not where latency matters the most.
Agree. But I do think avoiding a CPU context switch (work item) for a very lightweight job cleanup (usually just drop refs) will save of CPU cycles, thus also things like power, etc...
That's the sort of statements I'd like to be backed by actual numbers/scenarios proving that it actually makes a difference. The mixed model where things are partially freed immediately/partially deferred, and sometimes even with conditionals for whether the deferral happens or not, it just makes building a mental model of this thing a nightmare, which in turn usually leads to subtle bugs.
Quoted 39 lines
Quoted 6 lines
It's adding scheduling overhead, sure, but given all the stuff we defer already, I'm not too sure we're at saving a few cycles to get the cleanup done immediately. What's important to have is a way to signal fences in an atomic context, because this has an impact on latency.
Yes. The signaling happens first then
drm_dep_job_putif IRQ opt-in.Quoted 21 lines
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
+ /*
+ * Drop all input dependency fences now, in process context, before the
+ * final job put. Once the job is on the pending list its last reference
+ * may be dropped from adma_fencecallback (IRQ context), where calling
+ *xa_destroy()would be unsafe.
+ */I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
Quoted 8 lines
If it significantly hurts the perf, I'd like to understand why, because to me it looks like pure-cleanup (no signaling involved), and thus no other process waiting for us to do the cleanup. The only thing that might have an impact is how fast you release the resources, and given it's only a partial cleanup (
xa_destroy()still has to be deferred), I'd like to understand which part of the immediate cleanup is causing a contention (basically which kind of resources the system is starving of)
Quoted 3 lines
It was more of once we moved to a ref counted model, it is pretty trivial allow
drm_dep_job_putwhen the fence is signaling. It doesn't really add any complexity either, thus why I added it is.
It's not the refcount model I'm complaining about, it's the "part of it is always freed immediately, part of it is deferred, but not always ..." that happens in drm_dep_job_release() I'm questioning. I'd really prefer something like:
static void drm_dep_job_release()
{
// do it all unconditionally
}static void drm_dep_job_defer_release()
{
queue_work(&job->cleanup_work);
}static void drm_dep_job_put()
{
kref_put(job, drm_dep_job_defer_release);
}Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 48 lines · Boris Brezillon
Quoted 40 lines · On Wed, 18 Mar 2026 15:40:35 -0700 Matthew Brost <[email protected]>…
Quoted 34 lines
Quoted 3 lines
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Rust won't magically defer the cleanup, nor will it dictate how you want to do the queue teardown, those are things you need to implement. But it should give visibility about object lifetimes, and guarantee that an object that's still visible to some owners is usable (the notion of usable is highly dependent on the object implementation).
Just a purely theoretical example of a multi-step queue teardown that might be possible to encode in rust:
- MyJobQueue<Usable>: The job queue is currently exposed and usable. There's a ::destroy() method consuming 'self' and returning a MyJobQueue<Destroyed> object
- MyJobQueue<Destroyed>: The user asked for the workqueue to be destroyed. No new job can be pushed. Existing jobs that didn't make it to the FW queue are cancelled, jobs that are in-flight are cancelled if they can, or are just waited upon if they can't. When the whole destruction step is done, ::destroyed() is called, it consumes 'self' and returns a MyJobQueue<Inactive> object.
- MyJobQueue<Inactive>: The queue is no longer active (HW doesn't have any resources on this queue). It's ready to be cleaned up. ::cleanup() (or just ::drop()) defers the cleanup of some inner object that has been passed around between the various MyJobQueue<State> wrappers.Each of the state transition can happen asynchronously. A state transition consumes the object in one state, and returns a new object in its new state. None of the transition involves dropping a refcnt, ownership is just transferred. The final MyJobQueue<Inactive> object is the object we'll defer cleanup on.
Quoted 5 lines
It's a very high-level view of one way this can be implemented (I'm sure there are others, probably better than my suggestion) in order to make sure the object doesn't go away without the compiler enforcing proper state transitions.
Quoted 4 lines
I'm sure Rust can implement this. My point about Rust is it doesn't magically solve hard software arch probles, but I will admit the ownership model, way it can enforce locking at compile time is pretty cool.
Quoted 11 lines
It's not quite about rust directly solving those problems for you, it's about rust forcing you to think about those problems in the first place. So no, rust won't magically solve your multi-step teardown with crazy CPU <-> Device synchronization etc, but it allows you to clearly identify those steps, and think about how you want to represent them without abusing other concepts, like object refcounting/ownership. Everything I described, you can code it in C BTW, it's just that C is so lax that you can also abuse other stuff to get to your ends, which might or might not be safe, but more importantly, will very likely obfuscate the code (even with good docs).
This is very well put, and I completely agree. Sorry—I get annoyed by the Rust comments. It solves some classes of problems, but it doesn’t magically solve complex software architecture issues that need to be thoughtfully designed.
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each structdrm_dep_jobhas an associated structdrm_dep_fencethat
+ * provides a singledma_fence(@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent viadrm_dep_job_done_cb()and
+ * is signalled once @parent signals (or immediately ifrun_job()returns
+ * NULL or an error).Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here.
Hm, so that's a driver returning a dma_fence_array/chain through ::run_job()? Why would we not want to have them directly exposed and split up into singular fence objects at resv insertion time (I don't think syncobjs care, but I might be wrong). I mean, one of the point
Quoted 5 lines
Quoted 1 line
You can stick dma-fence-arrays in syncobjs, but not chains.
Yeah, kinda makes sense, since timeline syncobjs use chains, and if the chain reject inner chains, it won't work.
+1, Exactly.
Quoted 6 lines
Quoted 2 lines
Neither dma-fence-arrays/chain can go into dma-resv.
They can't go directly in it, but those can be split into individual fences and be inserted, which would achieve the same goal.
Yes, but now it becomes a driver problem (maybe only mine) rather than an opaque job fence that can be inserted. In my opinion, it’s best to keep the job vs. hardware fence abstraction.
Quoted 7 lines
Quoted 3 lines
Hence why disconnecting a job's finished fence from hardware fence IMO is good idea to keep so gives drivers flexiblity on the hardware fences.
The thing is, I'm not sure drivers were ever meant to expose containers through ::run_job().
Well there haven't been any rules...
Quoted 32 lines
Quoted 2 lines
e.g., If this design didn't have a job's finished fence, I'd have to open code one Xe side.
There might be other reasons we'd like to keep the drm_sched_fence-like proxy that I'm missing. But if it's the only one, and the fence-combining pattern you're describing is common to multiple drivers, we can provide a container implementation that's not a
fence_array, so you can use it to insert driver fences into other containers. This way we wouldn't force the proxy model to all drivers, but we would keep the code generic/re-usable.Quoted 7 lines
Quoted 5 lines
behind the container extraction is so fences coming from the same context/timeline can be detected and merged. If you insert the container through a proxy, you're defeating the whole fence merging optimization.
Right. Finished fences have single timeline too...
Aren't you faking a single timeline though if you combine fences from different engines running at their own pace into a container?
Quoted 10 lines
Quoted 9 lines
The second thing is that I'm not sure drivers were ever supposed to return fence containers in the first place, because the whole idea behind a fence context is that fences are emitted/signalled in seqno-order, and if the fence is encoding the state of multiple timelines that progress at their own pace, it becomes tricky to control that. I guess if it's always the same set of timelines that are combined, that would work.
Quoted 9 lines
Quoted 6 lines
Xe does this is definitely works. We submit to multiple rings, when all rings signal a seqno, a chain or array signals -> finished fence signals. The queues used in this manor can only submit multiple ring jobs so the finished fence timeline stays intact. If you could a multiple rings followed by a single ring submission on the same queue, yes this could break.
Okay, I had the same understanding, thanks for confirming.
I think the last three comments are resolved here—it’s a queue timeline. As long as the queue has consistent rules (i.e., submits to a consistent set of rings), this whole approach makes sense?
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
We do have that in Panthor, but that's all explicit: in a single SUBMIT, you can have multiple jobs targeting different queues, each of them having their own set of deps/signal ops. The combination of all the signal ops into a container is left to the UMD. It could be automated kernel side, but that would be a flag on the SIGNAL op leading to the creation of a
fence_arraycontaining fences from multiple submitted jobs, rather than the driver combining stuff in the fence it returns in ::run_job().
Quoted 9 lines
Quoted 4 lines
See above. We have a dedicated queue type for these type of submissions and single job that submits to the all rings. We had multiple queue / jobs in the i915 to implemented this but it turns out it is much cleaner with a single queue / singler job / multiple rings model.
Hm, okay. It didn't turn into a mess in Panthor, but Xe is likely an order of magnitude more complicated that Mali, so I'll refrain from judging this design decision.
Yes, Xe is a beast, but we tend to build complexity into components and layers to manage it. That is what I’m attempting to do here.
Quoted 20 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Another case is late arming of hardware fences in
run_job(which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available atrun_job, though.Quoted 6 lines
Yep, I also think we can arm the driver fence early in the case of JobQueue. The reason it couldn't be done before is because the scheduler was in the middle, deciding which entity to pull the next job from, which was changing the seqno a job driver-fence would be assigned (you can't guess that at queue time in that case).
Quoted 2 lines
Xe doesn't need to late arming, but it look like multiple drivers to implement the late arming which may be required (?).
Quoted 9 lines
As I said, it's mostly a problem when you have a single-HW-queue:multiple-contexts model, which is exactly what
drm_schedwas designed for. I suspect early arming is not an issue for any of the HW supporting FW-based scheduling (PVR, Mali, NVidia, ...). If you want to usedrm_depfor all drivers currently usingdrm_sched(I'm still not convinced this is a good idea to do that just yet, because then you're going to pull a lot of the complexity we're trying to get rid of), then you need late arming of driver fences.
Yes, even the hardware scheduling component [1] I hacked together relied on no late arming. But even then, you can arm a dma-fence early and assign a hardware seqno later in run_job()—those are two different things.
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
[...]
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue fromdrm_dep_job_init()until
+ *drm_dep_job_put()drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * indrm_dep_queue_run_job()untildrm_dep_job_done()kicks theput_job
+ * worker, which callsdrm_dep_job_put()to release that reference.Quoted 3 lines
Quoted 3 lines
Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context)
Quoted 8 lines
Quoted 4 lines
Did you mean in !IRQ (or !atomic) context here? Feels weird to not defer the cleanup when you're in an IRQ/atomic context, but defer it when you're in a thread context.
The put of a job in this design can be from an IRQ context (opt-in) feature.
xa_destroyblows up if it is called from an IRQ context, although maybe that could be workaround.
Quoted 6 lines
Making it so
_put()in IRQ context is safe is fine, what I'm saying is that instead of doing a partial immediate cleanup, and the rest in a worker, we can just defer everything: that is, have some_deref_release()function called bykref_put()that would queue a work item from which the actual release is done.
See below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
so we avoid
switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Well, the cleanup path is probably not where latency matters the most.
Agree. But I do think avoiding a CPU context switch (work item) for a very lightweight job cleanup (usually just drop refs) will save of CPU cycles, thus also things like power, etc...
That's the sort of statements I'd like to be backed by actual numbers/scenarios proving that it actually makes a difference. The
I disagree. This is not a locking micro-optimization, for example. It is a software architecture choice that says “do not trigger a CPU context to free a job,” which costs thousands of cycles. This will have an effect on CPU utilization and, thus, power.
Quoted 5 lines
mixed model where things are partially freed immediately/partially deferred, and sometimes even with conditionals for whether the deferral happens or not, it just makes building a mental model of this thing a nightmare, which in turn usually leads to subtle bugs.
See above—managing complexity in components. This works in both modes. I refactored Xe so it also works in IRQ context. If it would make you feel better, I can ask my company commits CI resources so non-IRQ mode consistently works too—it’s just a single API flag on the queue. But then maybe other companies should also commit to public CI.
Quoted 39 lines
Quoted 39 lines
Quoted 6 lines
It's adding scheduling overhead, sure, but given all the stuff we defer already, I'm not too sure we're at saving a few cycles to get the cleanup done immediately. What's important to have is a way to signal fences in an atomic context, because this has an impact on latency.
Yes. The signaling happens first then
drm_dep_job_putif IRQ opt-in.Quoted 29 lines
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
+ /*
+ * Drop all input dependency fences now, in process context, before the
+ * final job put. Once the job is on the pending list its last reference
+ * may be dropped from adma_fencecallback (IRQ context), where calling
+ *xa_destroy()would be unsafe.
+ */I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
If it significantly hurts the perf, I'd like to understand why, because to me it looks like pure-cleanup (no signaling involved), and thus no other process waiting for us to do the cleanup. The only thing that might have an impact is how fast you release the resources, and given it's only a partial cleanup (
xa_destroy()still has to be deferred), I'd like to understand which part of the immediate cleanup is causing a contention (basically which kind of resources the system is starving of)
Quoted 9 lines
Quoted 3 lines
It was more of once we moved to a ref counted model, it is pretty trivial allow
drm_dep_job_putwhen the fence is signaling. It doesn't really add any complexity either, thus why I added it is.It's not the refcount model I'm complaining about, it's the "part of it is always freed immediately, part of it is deferred, but not always ..." that happens in
drm_dep_job_release()I'm questioning. I'd really prefer something like:
You are completely missing the point here.
Here is what I’ve reduced my job put to:
188 xe_sched_job_free_fences(job); 189 dma_fence_put(job->fence); 190 job_free(job); 191 atomic_dec(&q->job_cnt); 192 xe_pm_runtime_put(xe);
These are lightweight (IRQ-safe) operations that never need to be done in a work item—so why kick one?
Matt
Quoted 14 lines
static void
drm_dep_job_release(){ // do it all unconditionally }static void
drm_dep_job_defer_release(){ queue_work(&job->cleanup_work); }static void
drm_dep_job_put(){ kref_put(job, drm_dep_job_defer_release); }
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 348 lines · Matthew Brost
Quoted 48 lines · Boris Brezillon
Quoted 40 lines · On Wed, 18 Mar 2026 15:40:35 -0700 Matthew Brost <[email protected]>…
Quoted 34 lines
Quoted 3 lines
So I don’t think Rust natively solves these types of problems, although I’ll concede that it does make refcounting a bit more sane.
Rust won't magically defer the cleanup, nor will it dictate how you want to do the queue teardown, those are things you need to implement. But it should give visibility about object lifetimes, and guarantee that an object that's still visible to some owners is usable (the notion of usable is highly dependent on the object implementation).
Just a purely theoretical example of a multi-step queue teardown that might be possible to encode in rust:
- MyJobQueue<Usable>: The job queue is currently exposed and usable. There's a ::destroy() method consuming 'self' and returning a MyJobQueue<Destroyed> object
- MyJobQueue<Destroyed>: The user asked for the workqueue to be destroyed. No new job can be pushed. Existing jobs that didn't make it to the FW queue are cancelled, jobs that are in-flight are cancelled if they can, or are just waited upon if they can't. When the whole destruction step is done, ::destroyed() is called, it consumes 'self' and returns a MyJobQueue<Inactive> object.
- MyJobQueue<Inactive>: The queue is no longer active (HW doesn't have any resources on this queue). It's ready to be cleaned up. ::cleanup() (or just ::drop()) defers the cleanup of some inner object that has been passed around between the various MyJobQueue<State> wrappers.Each of the state transition can happen asynchronously. A state transition consumes the object in one state, and returns a new object in its new state. None of the transition involves dropping a refcnt, ownership is just transferred. The final MyJobQueue<Inactive> object is the object we'll defer cleanup on.
Quoted 5 lines
It's a very high-level view of one way this can be implemented (I'm sure there are others, probably better than my suggestion) in order to make sure the object doesn't go away without the compiler enforcing proper state transitions.
Quoted 4 lines
I'm sure Rust can implement this. My point about Rust is it doesn't magically solve hard software arch probles, but I will admit the ownership model, way it can enforce locking at compile time is pretty cool.
Quoted 11 lines
It's not quite about rust directly solving those problems for you, it's about rust forcing you to think about those problems in the first place. So no, rust won't magically solve your multi-step teardown with crazy CPU <-> Device synchronization etc, but it allows you to clearly identify those steps, and think about how you want to represent them without abusing other concepts, like object refcounting/ownership. Everything I described, you can code it in C BTW, it's just that C is so lax that you can also abuse other stuff to get to your ends, which might or might not be safe, but more importantly, will very likely obfuscate the code (even with good docs).
This is very well put, and I completely agree. Sorry—I get annoyed by the Rust comments. It solves some classes of problems, but it doesn’t magically solve complex software architecture issues that need to be thoughtfully designed.
Quoted 27 lines
Quoted 27 lines
Quoted 26 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each structdrm_dep_jobhas an associated structdrm_dep_fencethat
+ * provides a singledma_fence(@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent viadrm_dep_job_done_cb()and
+ * is signalled once @parent signals (or immediately ifrun_job()returns
+ * NULL or an error).Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I thought this fence proxy mechanism was going away due to recent work being carried out by Christian?
Consider the case where a driver’s hardware fence is implemented as a dma-fence-array or dma-fence-chain. You cannot install these types of fences into a dma-resv or into syncobjs, so a proxy fence is useful here.
Hm, so that's a driver returning a dma_fence_array/chain through ::run_job()? Why would we not want to have them directly exposed and split up into singular fence objects at resv insertion time (I don't think syncobjs care, but I might be wrong). I mean, one of the point
Quoted 5 lines
Quoted 1 line
You can stick dma-fence-arrays in syncobjs, but not chains.
Yeah, kinda makes sense, since timeline syncobjs use chains, and if the chain reject inner chains, it won't work.
+1, Exactly.
Quoted 6 lines
Quoted 2 lines
Neither dma-fence-arrays/chain can go into dma-resv.
They can't go directly in it, but those can be split into individual fences and be inserted, which would achieve the same goal.
Yes, but now it becomes a driver problem (maybe only mine) rather than an opaque job fence that can be inserted. In my opinion, it’s best to keep the job vs. hardware fence abstraction.
Quoted 7 lines
Quoted 3 lines
Hence why disconnecting a job's finished fence from hardware fence IMO is good idea to keep so gives drivers flexiblity on the hardware fences.
The thing is, I'm not sure drivers were ever meant to expose containers through ::run_job().
Well there haven't been any rules...
Quoted 32 lines
Quoted 2 lines
e.g., If this design didn't have a job's finished fence, I'd have to open code one Xe side.
There might be other reasons we'd like to keep the drm_sched_fence-like proxy that I'm missing. But if it's the only one, and the fence-combining pattern you're describing is common to multiple drivers, we can provide a container implementation that's not a
fence_array, so you can use it to insert driver fences into other containers. This way we wouldn't force the proxy model to all drivers, but we would keep the code generic/re-usable.Quoted 7 lines
Quoted 5 lines
behind the container extraction is so fences coming from the same context/timeline can be detected and merged. If you insert the container through a proxy, you're defeating the whole fence merging optimization.
Right. Finished fences have single timeline too...
Aren't you faking a single timeline though if you combine fences from different engines running at their own pace into a container?
Quoted 10 lines
Quoted 9 lines
The second thing is that I'm not sure drivers were ever supposed to return fence containers in the first place, because the whole idea behind a fence context is that fences are emitted/signalled in seqno-order, and if the fence is encoding the state of multiple timelines that progress at their own pace, it becomes tricky to control that. I guess if it's always the same set of timelines that are combined, that would work.
Quoted 9 lines
Quoted 6 lines
Xe does this is definitely works. We submit to multiple rings, when all rings signal a seqno, a chain or array signals -> finished fence signals. The queues used in this manor can only submit multiple ring jobs so the finished fence timeline stays intact. If you could a multiple rings followed by a single ring submission on the same queue, yes this could break.
Okay, I had the same understanding, thanks for confirming.
I think the last three comments are resolved here—it’s a queue timeline. As long as the queue has consistent rules (i.e., submits to a consistent set of rings), this whole approach makes sense?
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
One example is when a single job submits work to multiple rings that are flipped in hardware at the same time.
We do have that in Panthor, but that's all explicit: in a single SUBMIT, you can have multiple jobs targeting different queues, each of them having their own set of deps/signal ops. The combination of all the signal ops into a container is left to the UMD. It could be automated kernel side, but that would be a flag on the SIGNAL op leading to the creation of a
fence_arraycontaining fences from multiple submitted jobs, rather than the driver combining stuff in the fence it returns in ::run_job().Quoted 9 lines
Quoted 4 lines
See above. We have a dedicated queue type for these type of submissions and single job that submits to the all rings. We had multiple queue / jobs in the i915 to implemented this but it turns out it is much cleaner with a single queue / singler job / multiple rings model.
Hm, okay. It didn't turn into a mess in Panthor, but Xe is likely an order of magnitude more complicated that Mali, so I'll refrain from judging this design decision.
Yes, Xe is a beast, but we tend to build complexity into components and layers to manage it. That is what I’m attempting to do here.
Quoted 20 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Another case is late arming of hardware fences in
run_job(which many drivers do). The proxy fence is immediately available at arm time and can be installed into dma-resv or syncobjs even though the actual hardware fence is not yet available. I think most drivers could be refactored to make the hardware fence immediately available atrun_job, though.Quoted 6 lines
Yep, I also think we can arm the driver fence early in the case of JobQueue. The reason it couldn't be done before is because the scheduler was in the middle, deciding which entity to pull the next job from, which was changing the seqno a job driver-fence would be assigned (you can't guess that at queue time in that case).
Quoted 2 lines
Xe doesn't need to late arming, but it look like multiple drivers to implement the late arming which may be required (?).
Quoted 9 lines
As I said, it's mostly a problem when you have a single-HW-queue:multiple-contexts model, which is exactly what
drm_schedwas designed for. I suspect early arming is not an issue for any of the HW supporting FW-based scheduling (PVR, Mali, NVidia, ...). If you want to usedrm_depfor all drivers currently usingdrm_sched(I'm still not convinced this is a good idea to do that just yet, because then you're going to pull a lot of the complexity we're trying to get rid of), then you need late arming of driver fences.Yes, even the hardware scheduling component [1] I hacked together relied on no late arming. But even then, you can arm a dma-fence early and assign a hardware seqno later in run_job()—those are two different things.
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines
[...]
Quoted 20 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue fromdrm_dep_job_init()until
+ *drm_dep_job_put()drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * indrm_dep_queue_run_job()untildrm_dep_job_done()kicks theput_job
+ * worker, which callsdrm_dep_job_put()to release that reference.Quoted 3 lines
Quoted 3 lines
Why not simply keep track that the job was completed, instead of relinquishing the reference? We can then release the reference once the job is cleaned up (by the queue, using a worker) in process context.
I think that’s what I’m doing, while also allowing an opt-in path to drop the job reference when it signals (in IRQ context)
Quoted 8 lines
Quoted 4 lines
Did you mean in !IRQ (or !atomic) context here? Feels weird to not defer the cleanup when you're in an IRQ/atomic context, but defer it when you're in a thread context.
The put of a job in this design can be from an IRQ context (opt-in) feature.
xa_destroyblows up if it is called from an IRQ context, although maybe that could be workaround.Quoted 6 lines
Making it so
_put()in IRQ context is safe is fine, what I'm saying is that instead of doing a partial immediate cleanup, and the rest in a worker, we can just defer everything: that is, have some_deref_release()function called bykref_put()that would queue a work item from which the actual release is done.See below.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
so we avoid
switching to a work item just to drop a ref. That seems like a significant win in terms of CPU cycles.Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Well, the cleanup path is probably not where latency matters the most.
Agree. But I do think avoiding a CPU context switch (work item) for a very lightweight job cleanup (usually just drop refs) will save of CPU cycles, thus also things like power, etc...
That's the sort of statements I'd like to be backed by actual numbers/scenarios proving that it actually makes a difference. The
I disagree. This is not a locking micro-optimization, for example. It is a software architecture choice that says “do not trigger a CPU context to free a job,” which costs thousands of cycles. This will have an effect on CPU utilization and, thus, power.
Quoted 5 lines
mixed model where things are partially freed immediately/partially deferred, and sometimes even with conditionals for whether the deferral happens or not, it just makes building a mental model of this thing a nightmare, which in turn usually leads to subtle bugs.
See above—managing complexity in components. This works in both modes. I refactored Xe so it also works in IRQ context. If it would make you feel better, I can ask my company commits CI resources so non-IRQ mode consistently works too—it’s just a single API flag on the queue. But then maybe other companies should also commit to public CI.
Quoted 43 lines
Quoted 42 lines
Quoted 6 lines
It's adding scheduling overhead, sure, but given all the stuff we defer already, I'm not too sure we're at saving a few cycles to get the cleanup done immediately. What's important to have is a way to signal fences in an atomic context, because this has an impact on latency.
Yes. The signaling happens first then
drm_dep_job_putif IRQ opt-in.Quoted 21 lines
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 6 lines
+ /*
+ * Drop all input dependency fences now, in process context, before the
+ * final job put. Once the job is on the pending list its last reference
+ * may be dropped from adma_fencecallback (IRQ context), where calling
+ *xa_destroy()would be unsafe.
+ */I assume that “pending” is the list of jobs that have been handed to the driver via
ops->run_job()?Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Can’t this problem be solved by not doing anything inside a
dma_fencecallback other than scheduling the queue worker?Yes, this code is required to support dropping job refs directly in the dma-fence callback (an opt-in feature). Again, this seems like a significant win in terms of CPU cycles, although I haven’t collected data yet.
Quoted 8 lines
If it significantly hurts the perf, I'd like to understand why, because to me it looks like pure-cleanup (no signaling involved), and thus no other process waiting for us to do the cleanup. The only thing that might have an impact is how fast you release the resources, and given it's only a partial cleanup (
xa_destroy()still has to be deferred), I'd like to understand which part of the immediate cleanup is causing a contention (basically which kind of resources the system is starving of)It was more of once we moved to a ref counted model, it is pretty trivial allow
drm_dep_job_putwhen the fence is signaling. It doesn't really add any complexity either, thus why I added it is.
Quoted 8 lines
Quoted 5 lines
It's not the refcount model I'm complaining about, it's the "part of it is always freed immediately, part of it is deferred, but not always ..." that happens in
drm_dep_job_release()I'm questioning. I'd really prefer something like:You are completely missing the point here.
Let me rephrase this — I realize this may come across as rude, which is not my intent. I believe there is simply a disconnect in understanding the constraints.
In my example below, the job release completes within bounded time constraints, which makes it suitable for direct release in IRQ context, bypassing the need for a work item that would otherwise incur a costly CPU context switch.
Matt
Quoted 27 lines
Here is what I’ve reduced my job put to:
188 xe_sched_job_free_fences(job); 189 dma_fence_put(job->fence); 190 job_free(job); 191 atomic_dec(&q->job_cnt); 192 xe_pm_runtime_put(xe);These are lightweight (IRQ-safe) operations that never need to be done in a work item—so why kick one?
Matt
Quoted 14 lines
static void
drm_dep_job_release(){ // do it all unconditionally }static void
drm_dep_job_defer_release(){ queue_work(&job->cleanup_work); }static void
drm_dep_job_put(){ kref_put(job, drm_dep_job_defer_release); }
Boris Brezillon
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 11 lines · On Mon, 23 Mar 2026 00:58:51 -0700 Matthew Brost <[email protected]>…
Quoted 8 lines
Quoted 5 lines
It's not the refcount model I'm complaining about, it's the "part of it is always freed immediately, part of it is deferred, but not always ..." that happens in
drm_dep_job_release()I'm questioning. I'd really prefer something like:You are completely missing the point here.
Let me rephrase this — I realize this may come across as rude, which is not my intent.
No offense taken ;-).
Quoted 7 lines
I believe there is simply a disconnect in understanding the constraints.
In my example below, the job release completes within bounded time constraints, which makes it suitable for direct release in IRQ context, bypassing the need for a work item that would otherwise incur a costly CPU context switch.
In the other thread, I've explained in more details why I think deferred cleanup of jobs is not as bad as you make it sound (context switch amortized by the fact it's already there for queue progress checking). But let's assume it is, I'd prefer a model where we say "ops->job_release() has to be IRQ-safe" and have implementations defer their cleanup if they have to, than this mixed approach with a flag. Of course, I'd still like to have numbers proving that this job cleanup deferral actually makes a difference in practice :P.
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 25 lines · Boris Brezillon
Quoted 11 lines · On Mon, 23 Mar 2026 00:58:51 -0700 Matthew Brost <[email protected]>…
Quoted 8 lines
Quoted 5 lines
It's not the refcount model I'm complaining about, it's the "part of it is always freed immediately, part of it is deferred, but not always ..." that happens in
drm_dep_job_release()I'm questioning. I'd really prefer something like:You are completely missing the point here.
Let me rephrase this — I realize this may come across as rude, which is not my intent.
No offense taken ;-).
Quoted 7 lines
I believe there is simply a disconnect in understanding the constraints.
In my example below, the job release completes within bounded time constraints, which makes it suitable for direct release in IRQ context, bypassing the need for a work item that would otherwise incur a costly CPU context switch.
Quoted 8 lines
In the other thread, I've explained in more details why I think deferred cleanup of jobs is not as bad as you make it sound (context switch amortized by the fact it's already there for queue progress checking). But let's assume it is, I'd prefer a model where we say "
ops->job_release()has to be IRQ-safe" and have implementations defer their cleanup if they have to, than this mixed approach with a flag. Of course, I'd still like to have numbers proving that this job cleanup deferral actually makes a difference in practice :P.
Yes, I replied there will either drop this or have solid numbers showing yes, the CPU utlization shows this is worth while.
Matt
Danilo Krummrich
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
In reply to Daniel AlmeidaQuoted 3 lines · Daniel Almeida
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
This is not exactly what I said. I was talking about the maintainance aspects and that a Rust Jobqueue implementation (for the reasons explained in my initial reply) is easily justifiable in this aspect, whereas another C implementation, that does *not* replace the existing DRM scheduler entirely, is much harder to justify from a maintainance perspective.
I'm also not sure whether a C interface from the Rust side is easy to establish. We don't want to limit ourselves in terms of language capabilities for this and passing through all the additional infromation Rust carries in the type system might not be straight forward.
It would be an experiment, and it was one of the ideas behind the Rust Jobqueue to see how it turns if we try. Always with the fallback of having C infrastructure as an alternative when it doesn't work out well.
Having this said, I don't see an issue with the drm_dep thing going forward if there is a path to replacing DRM sched entirely.
The Rust component should remain independent from this for the reasons mentioned in [1].
Daniel Almeida
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 9 lines · Danilo Krummrich
Quoted 3 lines · Daniel Almeida
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Quoted 5 lines
This is not exactly what I said. I was talking about the maintainance aspects and that a Rust Jobqueue implementation (for the reasons explained in my initial reply) is easily justifiable in this aspect, whereas another C implementation, that does *not* replace the existing DRM scheduler entirely, is much harder to justify from a maintainance perspective.
Ok, I misunderstood your point a bit.
Quoted 9 lines
I'm also not sure whether a C interface from the Rust side is easy to establish. We don't want to limit ourselves in terms of language capabilities for this and passing through all the additional infromation Rust carries in the type system might not be straight forward.
It would be an experiment, and it was one of the ideas behind the Rust Jobqueue to see how it turns if we try. Always with the fallback of having C infrastructure as an alternative when it doesn't work out well.
From previous experience in doing Rust to C FFI in NVK, I don’t see, at first, why this can’t work. But I agree with you, there may very well be unanticipated things here and this part is indeed an experiment. No argument from me here.
Quoted 3 lines
Having this said, I don't see an issue with the
drm_depthing going forward if there is a path to replacing DRM sched entirely.
The issues I pointed out remain. Even if the plan is to have drm_dep + JobQueue (and no drm_sched). I feel that my point of considering doing it in Rust remains.
Quoted 5 lines
The Rust component should remain independent from this for the reasons mentioned in [1].
Ok
— Daniel
Danilo Krummrich
reviewerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 33 lines · Daniel Almeida
Quoted 9 lines · Danilo Krummrich
Quoted 3 lines · Daniel Almeida
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Quoted 5 lines
This is not exactly what I said. I was talking about the maintainance aspects and that a Rust Jobqueue implementation (for the reasons explained in my initial reply) is easily justifiable in this aspect, whereas another C implementation, that does *not* replace the existing DRM scheduler entirely, is much harder to justify from a maintainance perspective.
Ok, I misunderstood your point a bit.
Quoted 9 lines
I'm also not sure whether a C interface from the Rust side is easy to establish. We don't want to limit ourselves in terms of language capabilities for this and passing through all the additional infromation Rust carries in the type system might not be straight forward.
It would be an experiment, and it was one of the ideas behind the Rust Jobqueue to see how it turns if we try. Always with the fallback of having C infrastructure as an alternative when it doesn't work out well.
From previous experience in doing Rust to C FFI in NVK, I don’t see, at first, why this can’t work. But I agree with you, there may very well be unanticipated things here and this part is indeed an experiment. No argument from me here.
Quoted 1 line
Quoted 5 lines
Quoted 2 lines
Having this said, I don't see an issue with the
drm_depthing going forward if there is a path to replacing DRM sched entirely.The issues I pointed out remain. Even if the plan is to have
drm_dep+ JobQueue (and nodrm_sched). I feel that my point of considering doing it in Rust remains.
I mean, as mentioned below, we should have a Rust Jobqueue as independent component. Or are you saying you'd consdider having only a Rust component with a C API eventually? If so, that'd be way too early to consider for various reasons.
Quoted 8 lines
Quoted 4 lines
The Rust component should remain independent from this for the reasons mentioned in [1].
Ok
— Daniel
Matthew Brost
maintainerRe: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer
Quoted 28 lines · Danilo Krummrich
Quoted 27 lines · Daniel Almeida
Quoted 9 lines · Danilo Krummrich
Quoted 3 lines · Daniel Almeida
I agree with what Danilo said below, i.e.: IMHO, with the direction that DRM is going, it is much more ergonomic to add a Rust component with a nice C interface than doing it the other way around.
Quoted 5 lines
This is not exactly what I said. I was talking about the maintainance aspects and that a Rust Jobqueue implementation (for the reasons explained in my initial reply) is easily justifiable in this aspect, whereas another C implementation, that does *not* replace the existing DRM scheduler entirely, is much harder to justify from a maintainance perspective.
Ok, I misunderstood your point a bit.
Quoted 9 lines
I'm also not sure whether a C interface from the Rust side is easy to establish. We don't want to limit ourselves in terms of language capabilities for this and passing through all the additional infromation Rust carries in the type system might not be straight forward.
It would be an experiment, and it was one of the ideas behind the Rust Jobqueue to see how it turns if we try. Always with the fallback of having C infrastructure as an alternative when it doesn't work out well.
Quoted 8 lines
Quoted 8 lines
From previous experience in doing Rust to C FFI in NVK, I don’t see, at first, why this can’t work. But I agree with you, there may very well be unanticipated things here and this part is indeed an experiment. No argument from me here.
Quoted 3 lines
Having this said, I don't see an issue with the
drm_depthing going forward if there is a path to replacing DRM sched entirely.
The only weird case I haven't wrapped my head around quite yet is the ganged submissions that rely on the scheduled fence (PVR, AMDGPU do this). Pretty much every other driver seems like it could be coverted with what I have in place in this series + local work to provide a hardware scheduler...
Quoted 9 lines
Quoted 3 lines
The issues I pointed out remain. Even if the plan is to have
drm_dep+ JobQueue (and nodrm_sched). I feel that my point of considering doing it in Rust remains.I mean, as mentioned below, we should have a Rust Jobqueue as independent component. Or are you saying you'd consdider having only a Rust component with a C API eventually? If so, that'd be way too early to consider for various reasons.
We need to some C story one way or another as we have C drivers and DRM sched is not cutting it nor is maitainable.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
The Rust component should remain independent from this for the reasons mentioned in [1].
Fair enough. I read through [1], let me respond there.
Matt
Quoted 5 lines
Quoted 4 lines
Ok
— Daniel