Skip to contentSkip to conversation

Introduce DRM dep queue

RFC PATCH In review Matthew Brost 10 participants 65 messages Mar 16 – Mar 27 Page 2 of 3

Subsystems (9)
lore
Share & tools
lore
#36 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

In reply to Matthew Brost
Quoted 9 lines · Matthew Brost

Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that drm_gpu_scheduler is no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendered drm_gpu_scheduler unmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.

Quoted 9 lines

Add drm_dep, a lightweight GPU submission queue intended as a replacement for drm_gpu_scheduler for firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlike drm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination, drm_dep merges both roles into a single struct drm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.

Yeah I can't count how often I considered re-writing the GPU scheduler from scratch.

But if that is done I completely agree that it should probably be done in Rust instead of C.

I've worked enough with save languages to aknoledge the advantages they have.

Regards,
Christian.

Quoted 3,579 lines

Unlike drm_gpu_scheduler, which relies on external locking and lifetime management by the driver, drm_dep uses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until the put_job worker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.

The core objects are:

struct drm_dep_queue - a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.

struct drm_dep_job - a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of input dma_fence dependencies and produce a drm_dep_fence as their finished fence.

struct drm_dep_fence - a dma_fence subclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).

Job lifecycle:
1. drm_dep_job_init() - allocate and initialise; job acquires a queue reference.
2. drm_dep_job_add_dependency() and friends - register input fences; duplicates from the same context are deduplicated.
3. drm_dep_job_arm() - assign sequence number, obtain finished fence.
4. drm_dep_job_push() - submit to queue.

Submission paths under queue lock:
- Bypass path: if DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and the run_job worker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before calling ops->run_job().

Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.

Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds q->job.timeout jiffies, calling ops->timedout_job(). drm_dep_queue_trigger_timeout() forces immediate expiry for device teardown.

IRQ-safe completion: queues flagged DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE allow drm_dep_job_done() to be called from hardirq context (e.g. a dma_fence callback). Dependency cleanup is deferred to process context after ops->run_job() returns to avoid calling xa_destroy() from IRQ.

Zombie-state guard: workers use kref_get_unless_zero() on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.

Teardown is always deferred to a module-private workqueue (dep_free_wq) so that destroy_workqueue() is never called from within one of the queue's own workers. Each queue holds a drm_dev_get() reference on its owning struct drm_device, released as the final step of teardown via drm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.

Cc Boris Brezillon <[email protected]>
Cc Tvrtko Ursulin <[email protected]>
Cc Rodrigo Vivi <[email protected]>
Cc Thomas Hellström <[email protected]>
Cc Christian König <[email protected]>
Cc Danilo Krummrich <[email protected]>
Cc David Airlie <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Philipp Stanner <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Sumit Semwal <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Assisted-by GitHub Copilot:claude-sonnet-4.6
---
 drivers/gpu/drm/Kconfig             |    4 +
 drivers/gpu/drm/Makefile            |    1 +
 drivers/gpu/drm/dep/Makefile        |    5 +
 drivers/gpu/drm/dep/drm_dep_fence.c |  406 +++++++
 drivers/gpu/drm/dep/drm_dep_fence.h |   25 +
 drivers/gpu/drm/dep/drm_dep_job.c   |  675 +++++++++++
 drivers/gpu/drm/dep/drm_dep_job.h   |   13 +
 drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++
 drivers/gpu/drm/dep/drm_dep_queue.h |   31 +
 include/drm/drm_dep.h               |  597 ++++++++++
 10 files changed, 3404 insertions(+)
 create mode 100644 drivers/gpu/drm/dep/Makefile
 create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h
 create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h
 create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h
 create mode 100644 include/drm/drm_dep.h
drivers/gpu/drm/Kconfig +4 -0
index 5386248e75b6..834f6e210551 100644
@@ -276,6 +276,10 @@ config DRM_SCHED
 	tristate
 	depends on DRM
 
+config DRM_DEP
+	tristate
+	depends on DRM
+
 # Separate option as not all DRM drivers use it
 config DRM_PANEL_BACKLIGHT_QUIRKS
 	tristate
drivers/gpu/drm/Makefile +1 -0
index e97faabcd783..1ad87cc0e545 100644
@@ -173,6 +173,7 @@ obj-y			+= clients/
 obj-y			+= display/
 obj-$(CONFIG_DRM_TTM)	+= ttm/
 obj-$(CONFIG_DRM_SCHED)	+= scheduler/
+obj-$(CONFIG_DRM_DEP)	+= dep/
 obj-$(CONFIG_DRM_RADEON)+= radeon/
 obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
 obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/
drivers/gpu/drm/dep/Makefile new file +5 -0
new file mode 100644
index 000000000000..335f1af46a7b
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o
+
+obj-$(CONFIG_DRM_DEP) += drm_dep.o
drivers/gpu/drm/dep/drm_dep_fence.c new file +406 -0
new file mode 100644
index 000000000000..ae05b9077772
@@ -0,0 +1,406 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each struct drm_dep_job has an associated struct drm_dep_fence that
+ * provides a single dma_fence (@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent via drm_dep_job_done_cb() and
+ * is signalled once @parent signals (or immediately if run_job() returns
+ * NULL or an error).
+ *
+ * Drivers should expose @finished as the out-fence for GPU work since it is
+ * valid from the moment drm_dep_job_arm() returns, whereas the hardware fence
+ * could be a compound fence, which is disallowed when installed into
+ * drm_syncobjs or dma-resv.
+ *
+ * The fence uses the kernel's inline spinlock (NULL passed to dma_fence_init())
+ * so no separate lock allocation is required.
+ *
+ * Deadline propagation is supported: if a consumer sets a deadline via
+ * dma_fence_set_deadline(), it is forwarded to @parent when @parent is set.
+ * If @parent has not been set yet the deadline is stored in @deadline and
+ * forwarded at that point.
+ *
+ * Memory management: drm_dep_fence objects are allocated with kzalloc() and
+ * freed via kfree_rcu() once the fence is released, ensuring safety with
+ * RCU-protected fence accesses.
+ */
+
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include "drm_dep_fence.h"
+
+/**
+ * DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has been set
+ *
+ * Set by the deadline callback on the finished fence to indicate a deadline
+ * has been set which may need to be propagated to the parent hardware fence.
+ */
+#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT	(DMA_FENCE_FLAG_USER_BITS + 1)
+
+/**
+ * struct drm_dep_fence - fence tracking the completion of a dep job
+ *
+ * Contains a single dma_fence (@finished) that is signalled when the
+ * hardware completes the job. The fence uses the kernel's inline_lock
+ * (no external spinlock required).
+ *
+ * This struct is private to the drm_dep module; external code interacts
+ * through the accessor functions declared in drm_dep_fence.h.
+ */
+struct drm_dep_fence {
+	/**
+	 * @finished: signalled when the job completes on hardware.
+	 *
+	 * Drivers should use this fence as the out-fence for a job since it
+	 * is available immediately upon drm_dep_job_arm().
+	 */
+	struct dma_fence finished;
+
+	/**
+	 * @deadline: deadline set on @finished which potentially needs to be
+	 * propagated to @parent.
+	 */
+	ktime_t	deadline;
+
+	/**
+	 * @parent: The hardware fence returned by &drm_dep_queue_ops.run_job.
+	 *
+	 * @finished is signaled once @parent is signaled. The initial store is
+	 * performed via smp_store_release to synchronize with deadline handling.
+	 *
+	 * All readers must access this under the fence lock and take a reference to
+	 * it, as @parent is set to NULL under the fence lock when the drm_dep_fence
+	 * signals, and this drop also releases its internal reference.
+	 */
+	struct dma_fence *parent;
+
+	/**
+	 * @q: the queue this fence belongs to.
+	 */
+	struct drm_dep_queue *q;
+};
+
+static const struct dma_fence_ops drm_dep_fence_ops;
+
+/**
+ * to_drm_dep_fence() - cast a dma_fence to its enclosing drm_dep_fence
+ * @f: dma_fence to cast
+ *
+ * Context: No context requirements (inline helper).
+ * Return: pointer to the enclosing &drm_dep_fence.
+ */
+static struct drm_dep_fence *to_drm_dep_fence(struct dma_fence *f)
+{
+	return container_of(f, struct drm_dep_fence, finished);
+}
+
+/**
+ * drm_dep_fence_set_parent() - store the hardware fence and propagate
+ *   any deadline
+ * @dfence: dep fence
+ * @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or NULL/error
+ *
+ * Stores @parent on @dfence under smp_store_release() so that a concurrent
+ * drm_dep_fence_set_deadline() call sees the parent before checking the
+ * deadline bit. If a deadline has already been set on @dfence->finished it is
+ * forwarded to @parent immediately. Does nothing if @parent is NULL or an
+ * error pointer.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_set_parent(struct drm_dep_fence *dfence,
+			      struct dma_fence *parent)
+{
+	if (IS_ERR_OR_NULL(parent))
+		return;
+
+	/*
+	 * smp_store_release() to ensure a thread racing us in
+	 * drm_dep_fence_set_deadline() sees the parent set before
+	 * it calls test_bit(HAS_DEADLINE_BIT).
+	 */
+	smp_store_release(&dfence->parent, dma_fence_get(parent));
+	if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT,
+		     &dfence->finished.flags))
+		dma_fence_set_deadline(parent, dfence->deadline);
+}
+
+/**
+ * drm_dep_fence_finished() - signal the finished fence with a result
+ * @dfence: dep fence to signal
+ * @result: error code to set, or 0 for success
+ *
+ * Sets the fence error to @result if non-zero, then signals
+ * @dfence->finished. Also removes parent visibility under the fence lock
+ * and drops the parent reference. Dropping the parent here allows the
+ * DRM dep fence to be completely decoupled from the DRM dep module.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_fence_finished(struct drm_dep_fence *dfence, int result)
+{
+	struct dma_fence *parent;
+	unsigned long flags;
+
+	dma_fence_lock_irqsave(&dfence->finished, flags);
+	if (result)
+		dma_fence_set_error(&dfence->finished, result);
+	dma_fence_signal_locked(&dfence->finished);
+	parent = dfence->parent;
+	dfence->parent = NULL;
+	dma_fence_unlock_irqrestore(&dfence->finished, flags);
+
+	dma_fence_put(parent);
+}
+
+static const char *drm_dep_fence_get_driver_name(struct dma_fence *fence)
+{
+	return "drm_dep";
+}
+
+static const char *drm_dep_fence_get_timeline_name(struct dma_fence *f)
+{
+	struct drm_dep_fence *dfence = to_drm_dep_fence(f);
+
+	return dfence->q->name;
+}
+
+/**
+ * drm_dep_fence_get_parent() - get a reference to the parent hardware fence
+ * @dfence: dep fence to query
+ *
+ * Returns a new reference to @dfence->parent, or NULL if the parent has
+ * already been cleared (i.e. @dfence->finished has signalled and the parent
+ * reference was dropped under the fence lock).
+ *
+ * Uses smp_load_acquire() to pair with the smp_store_release() in
+ * drm_dep_fence_set_parent(), ensuring that if we race a concurrent
+ * drm_dep_fence_set_parent() call we observe the parent pointer only after
+ * the store is fully visible — before set_parent() tests
+ * %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT.
+ *
+ * Caller must hold the fence lock on @dfence->finished.
+ *
+ * Context: Any context, fence lock on @dfence->finished must be held.
+ * Return: a new reference to the parent fence, or NULL.
+ */
+static struct dma_fence *drm_dep_fence_get_parent(struct drm_dep_fence *dfence)
+{
+	dma_fence_assert_held(&dfence->finished);
+
+	return dma_fence_get(smp_load_acquire(&dfence->parent));
+}
+
+/**
+ * drm_dep_fence_set_deadline() - dma_fence_ops deadline callback
+ * @f: fence on which the deadline is being set
+ * @deadline: the deadline hint to apply
+ *
+ * Stores the earliest deadline under the fence lock, then propagates
+ * it to the parent hardware fence via smp_load_acquire() to race
+ * safely with drm_dep_fence_set_parent().
+ *
+ * Context: Any context.
+ */
+static void drm_dep_fence_set_deadline(struct dma_fence *f, ktime_t deadline)
+{
+	struct drm_dep_fence *dfence = to_drm_dep_fence(f);
+	struct dma_fence *parent;
+	unsigned long flags;
+
+	dma_fence_lock_irqsave(f, flags);
+
+	/* If we already have an earlier deadline, keep it: */
+	if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags) &&
+	    ktime_before(dfence->deadline, deadline)) {
+		dma_fence_unlock_irqrestore(f, flags);
+		return;
+	}
+
+	dfence->deadline = deadline;
+	set_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags);
+
+	parent = drm_dep_fence_get_parent(dfence);
+	dma_fence_unlock_irqrestore(f, flags);
+
+	if (parent)
+		dma_fence_set_deadline(parent, deadline);
+
+	dma_fence_put(parent);
+}
+
+static const struct dma_fence_ops drm_dep_fence_ops = {
+	.get_driver_name = drm_dep_fence_get_driver_name,
+	.get_timeline_name = drm_dep_fence_get_timeline_name,
+	.set_deadline = drm_dep_fence_set_deadline,
+};
+
+/**
+ * drm_dep_fence_alloc() - allocate a dep fence
+ *
+ * Allocates a &drm_dep_fence with kzalloc() without initialising the
+ * dma_fence. Call drm_dep_fence_init() to fully initialise it.
+ *
+ * Context: Process context.
+ * Return: new &drm_dep_fence on success, NULL on allocation failure.
+ */
+struct drm_dep_fence *drm_dep_fence_alloc(void)
+{
+	return kzalloc_obj(struct drm_dep_fence);
+}
+
+/**
+ * drm_dep_fence_init() - initialise the dma_fence inside a dep fence
+ * @dfence: dep fence to initialise
+ * @q: queue the owning job belongs to
+ *
+ * Initialises @dfence->finished using the context and sequence number from @q.
+ * Passes NULL as the lock so the fence uses its inline spinlock.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_init(struct drm_dep_fence *dfence, struct drm_dep_queue *q)
+{
+	u32 seq = ++q->fence.seqno;
+
+	/*
+	 * XXX: Inline fence hazard: currently all expected users of DRM dep
+	 * hardware fences have a unique lockdep class. If that ever changes,
+	 * we will need to assign a unique lockdep class here so lockdep knows
+	 * this fence is allowed to nest with driver hardware fences.
+	 */
+
+	dfence->q = q;
+	dma_fence_init(&dfence->finished, &drm_dep_fence_ops,
+		       NULL, q->fence.context, seq);
+}
+
+/**
+ * drm_dep_fence_cleanup() - release a dep fence at job teardown
+ * @dfence: dep fence to clean up
+ *
+ * Called from drm_dep_job_fini(). If the dep fence was armed (refcount > 0)
+ * it is released via dma_fence_put() and will be freed by the RCU release
+ * callback once all waiters have dropped their references. If it was never
+ * armed it is freed directly with kfree().
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_cleanup(struct drm_dep_fence *dfence)
+{
+	if (drm_dep_fence_is_armed(dfence))
+		dma_fence_put(&dfence->finished);
+	else
+		kfree(dfence);
+}
+
+/**
+ * drm_dep_fence_is_armed() - check whether the fence has been armed
+ * @dfence: dep fence to check
+ *
+ * Returns true if drm_dep_job_arm() has been called, i.e. @dfence->finished
+ * has been initialised and its reference count is non-zero.  Used by
+ * assertions to enforce correct job lifecycle ordering (arm before push,
+ * add_dependency before arm).
+ *
+ * Context: Any context.
+ * Return: true if the fence is armed, false otherwise.
+ */
+bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence)
+{
+	return !!kref_read(&dfence->finished.refcount);
+}
+
+/**
+ * drm_dep_fence_is_finished() - test whether the finished fence has signalled
+ * @dfence: dep fence to check
+ *
+ * Uses dma_fence_test_signaled_flag() to read %DMA_FENCE_FLAG_SIGNALED_BIT
+ * directly without invoking the fence's ->signaled() callback or triggering
+ * any signalling side-effects.
+ *
+ * Context: Any context.
+ * Return: true if @dfence->finished has been signalled, false otherwise.
+ */
+bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence)
+{
+	return dma_fence_test_signaled_flag(&dfence->finished);
+}
+
+/**
+ * drm_dep_fence_is_complete() - test whether the job has completed
+ * @dfence: dep fence to check
+ *
+ * Takes the fence lock on @dfence->finished and calls
+ * drm_dep_fence_get_parent() to safely obtain a reference to the parent
+ * hardware fence — or NULL if the parent has already been cleared after
+ * signalling.  Calls dma_fence_is_signaled() on @parent outside the lock,
+ * which may invoke the fence's ->signaled() callback and trigger signalling
+ * side-effects if the fence has completed but the signalled flag has not yet
+ * been set.  The finished fence is tested via dma_fence_test_signaled_flag(),
+ * without side-effects.
+ *
+ * May only be called on a stopped queue (see drm_dep_queue_is_stopped()).
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if the job is complete, false otherwise.
+ */
+bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence)
+{
+	struct dma_fence *parent;
+	unsigned long flags;
+	bool complete;
+
+	dma_fence_lock_irqsave(&dfence->finished, flags);
+	parent = drm_dep_fence_get_parent(dfence);
+	dma_fence_unlock_irqrestore(&dfence->finished, flags);
+
+	complete = (parent && dma_fence_is_signaled(parent)) ||
+		dma_fence_test_signaled_flag(&dfence->finished);
+
+	dma_fence_put(parent);
+
+	return complete;
+}
+
+/**
+ * drm_dep_fence_to_dma() - return the finished dma_fence for a dep fence
+ * @dfence: dep fence to query
+ *
+ * No reference is taken; the caller must hold its own reference to the owning
+ * &drm_dep_job for the duration of the access.
+ *
+ * Context: Any context.
+ * Return: the finished &dma_fence.
+ */
+struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence)
+{
+	return &dfence->finished;
+}
+
+/**
+ * drm_dep_fence_done() - signal the finished fence on job completion
+ * @dfence: dep fence to signal
+ * @result: job error code, or 0 on success
+ *
+ * Gets a temporary reference to @dfence->finished to guard against a racing
+ * last-put, signals the fence with @result, then drops the temporary
+ * reference. Called from drm_dep_job_done() in the queue core when a
+ * hardware completion callback fires or when run_job() returns immediately.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_done(struct drm_dep_fence *dfence, int result)
+{
+	dma_fence_get(&dfence->finished);
+	drm_dep_fence_finished(dfence, result);
+	dma_fence_put(&dfence->finished);
+}
drivers/gpu/drm/dep/drm_dep_fence.h new file +25 -0
new file mode 100644
index 000000000000..65a1582f858b
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_FENCE_H_
+#define _DRM_DEP_FENCE_H_
+
+#include <linux/dma-fence.h>
+
+struct drm_dep_fence;
+struct drm_dep_queue;
+
+struct drm_dep_fence *drm_dep_fence_alloc(void);
+void drm_dep_fence_init(struct drm_dep_fence *dfence, struct drm_dep_queue *q);
+void drm_dep_fence_cleanup(struct drm_dep_fence *dfence);
+void drm_dep_fence_set_parent(struct drm_dep_fence *dfence,
+			      struct dma_fence *parent);
+void drm_dep_fence_done(struct drm_dep_fence *dfence, int result);
+bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence);
+bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence);
+bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence);
+struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence);
+
+#endif /* _DRM_DEP_FENCE_H_ */
drivers/gpu/drm/dep/drm_dep_job.c new file +675 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
+	 * drm_dep_job_drop_dependencies() in process context after run_job().
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+	if (job)
+		kref_get(&job->refcount);
+	return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+	struct drm_dep_job *job =
+		container_of(kref, struct drm_dep_job, refcount);
+	struct drm_dep_queue *q = job->q;
+
+	drm_dep_job_fini(job);
+
+	if (job->ops && job->ops->release)
+		job->ops->release(job);
+	else
+		kfree(job);
+
+	drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+	if (job)
+		kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+	drm_dep_queue_push_job_begin(job->q);
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	drm_dep_fence_init(job->dfence, job->q);
+	job->signalling_cookie = dma_fence_begin_signalling();
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+	struct drm_dep_queue *q = job->q;
+	struct dma_fence *entry;
+	unsigned long index;
+	u32 id = 0;
+	int ret;
+
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	might_alloc(GFP_KERNEL);
+
+	if (!fence)
+		return 0;
+
+	if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+		goto add_fence;
+
+	/*
+	 * Ignore signalled fences or fences from our own queue — finished
+	 * fences use q->fence.context.
+	 */
+	if (dma_fence_test_signaled_flag(fence) ||
+	    fence->context == q->fence.context) {
+		dma_fence_put(fence);
+		return 0;
+	}
+
+	/* Deduplicate if we already depend on a fence from the same context.
+	 * This lets the size of the array of deps scale with the number of
+	 * engines involved, rather than the number of BOs.
+	 */
+	xa_for_each(&job->dependencies, index, entry) {
+		if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+		    entry->context != fence->context)
+			continue;
+
+		if (dma_fence_is_later(fence, entry)) {
+			dma_fence_put(entry);
+			xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+		} else {
+			dma_fence_put(fence);
+		}
+		return 0;
+	}
+
+add_fence:
+	ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+		       GFP_KERNEL);
+	if (ret != 0) {
+		if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+			dma_fence_put(fence);
+		return ret;
+	}
+
+	return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+				    struct dma_fence *fence)
+{
+	WARN_ON(xa_load(&job->dependencies, index) !=
+		DRM_DEP_JOB_FENCE_PREALLOC);
+
+	if (dma_fence_test_signaled_flag(fence)) {
+		xa_erase(&job->dependencies, index);
+		dma_fence_put(fence);
+		return;
+	}
+
+	if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
+				       GFP_NOWAIT)))) {
+		dma_fence_put(fence);
+		return;
+	}
+}
+EXPORT_SYMBOL(drm_dep_job_replace_dependency);
+
+/**
+ * drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a
+ *   job dependency
+ * @job: dep job to add the dependencies to
+ * @file: drm file private pointer
+ * @handle: syncobj handle to lookup
+ * @point: timeline point
+ *
+ * This adds the fence matching the given syncobj to @job.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_syncobj_dependency(struct drm_dep_job *job,
+				       struct drm_file *file, u32 handle,
+				       u32 point)
+{
+	struct dma_fence *fence;
+	int ret;
+
+	ret = drm_syncobj_find_fence(file, handle, point, 0, &fence);
+	if (ret)
+		return ret;
+
+	return drm_dep_job_add_dependency(job, fence);
+}
+EXPORT_SYMBOL(drm_dep_job_add_syncobj_dependency);
+
+/**
+ * drm_dep_job_add_resv_dependencies() - add all fences from the resv to the job
+ * @job: dep job to add the dependencies to
+ * @resv: the dma_resv object to get the fences from
+ * @usage: the dma_resv_usage to use to filter the fences
+ *
+ * This adds all fences matching the given usage from @resv to @job.
+ * Must be called with the @resv lock held.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_resv_dependencies(struct drm_dep_job *job,
+				      struct dma_resv *resv,
+				      enum dma_resv_usage usage)
+{
+	struct dma_resv_iter cursor;
+	struct dma_fence *fence;
+	int ret;
+
+	dma_resv_assert_held(resv);
+
+	dma_resv_for_each_fence(&cursor, resv, usage, fence) {
+		/*
+		 * As drm_dep_job_add_dependency always consumes the fence
+		 * reference (even when it fails), and dma_resv_for_each_fence
+		 * is not obtaining one, we need to grab one before calling.
+		 */
+		ret = drm_dep_job_add_dependency(job, dma_fence_get(fence));
+		if (ret)
+			return ret;
+	}
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_resv_dependencies);
+
+/**
+ * drm_dep_job_add_implicit_dependencies() - adds implicit dependencies
+ *   as job dependencies
+ * @job: dep job to add the dependencies to
+ * @obj: the gem object to add new dependencies from.
+ * @write: whether the job might write the object (so we need to depend on
+ * shared fences in the reservation object).
+ *
+ * This should be called after drm_gem_lock_reservations() on your array of
+ * GEM objects used in the job but before updating the reservations with your
+ * own fences.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_implicit_dependencies(struct drm_dep_job *job,
+					  struct drm_gem_object *obj,
+					  bool write)
+{
+	return drm_dep_job_add_resv_dependencies(job, obj->resv,
+						 dma_resv_usage_rw(write));
+}
+EXPORT_SYMBOL(drm_dep_job_add_implicit_dependencies);
+
+/**
+ * drm_dep_job_is_signaled() - check whether a dep job has completed
+ * @job: dep job to check
+ *
+ * Determines whether @job has signalled. The queue should be stopped before
+ * calling this to obtain a stable snapshot of state. Both the parent hardware
+ * fence and the finished software fence are checked.
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if the job is signalled, false otherwise.
+ */
+bool drm_dep_job_is_signaled(struct drm_dep_job *job)
+{
+	WARN_ON(!drm_dep_queue_is_stopped(job->q));
+	return drm_dep_fence_is_complete(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_is_signaled);
+
+/**
+ * drm_dep_job_is_finished() - test whether a dep job's finished fence has signalled
+ * @job: dep job to check
+ *
+ * Tests whether the job's software finished fence has been signalled, using
+ * dma_fence_test_signaled_flag() to avoid any signalling side-effects. Unlike
+ * drm_dep_job_is_signaled(), this does not require the queue to be stopped and
+ * does not check the parent hardware fence — it is a lightweight test of the
+ * finished fence only.
+ *
+ * Context: Any context.
+ * Return: true if the job's finished fence has been signalled, false otherwise.
+ */
+bool drm_dep_job_is_finished(struct drm_dep_job *job)
+{
+	return drm_dep_fence_is_finished(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_is_finished);
+
+/**
+ * drm_dep_job_invalidate_job() - increment the invalidation count for a job
+ * @job: dep job to invalidate
+ * @threshold: threshold above which the job is considered invalidated
+ *
+ * Increments @job->invalidate_count and returns true if it exceeds @threshold,
+ * indicating the job should be considered hung and discarded. The queue must
+ * be stopped before calling this function.
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if @job->invalidate_count exceeds @threshold, false otherwise.
+ */
+bool drm_dep_job_invalidate_job(struct drm_dep_job *job, int threshold)
+{
+	WARN_ON(!drm_dep_queue_is_stopped(job->q));
+	return ++job->invalidate_count > threshold;
+}
+EXPORT_SYMBOL(drm_dep_job_invalidate_job);
+
+/**
+ * drm_dep_job_finished_fence() - return the finished fence for a job
+ * @job: dep job to query
+ *
+ * No reference is taken on the returned fence; the caller must hold its own
+ * reference to @job for the duration of any access.
+ *
+ * Context: Any context.
+ * Return: the finished &dma_fence for @job.
+ */
+struct dma_fence *drm_dep_job_finished_fence(struct drm_dep_job *job)
+{
+	return drm_dep_fence_to_dma(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_finished_fence);
drivers/gpu/drm/dep/drm_dep_job.h new file +13 -0
new file mode 100644
index 000000000000..35c61d258fa1
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_JOB_H_
+#define _DRM_DEP_JOB_H_
+
+struct drm_dep_queue;
+
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job);
+
+#endif /* _DRM_DEP_JOB_H_ */
drivers/gpu/drm/dep/drm_dep_queue.c new file +1647 -0
new file mode 100644
index 000000000000..dac02d0d22c4
@@ -0,0 +1,1647 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency queue
+ *
+ * The drm_dep subsystem provides a lightweight GPU submission queue that
+ * combines the roles of drm_gpu_scheduler and drm_sched_entity into a
+ * single object (struct drm_dep_queue). Each queue owns its own ordered
+ * submit workqueue, timeout workqueue, and TDR delayed-work.
+ *
+ * **Job lifecycle**
+ *
+ * 1. Allocate and initialise a job with drm_dep_job_init().
+ * 2. Add dependency fences with drm_dep_job_add_dependency() and friends.
+ * 3. Arm the job with drm_dep_job_arm() to obtain its out-fences.
+ * 4. Submit with drm_dep_job_push().
+ *
+ * **Submission paths**
+ *
+ * drm_dep_job_push() decides between two paths under @q->sched.lock:
+ *
+ * - **Bypass path** (drm_dep_queue_can_job_bypass()): if
+ *   %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not stopped,
+ *   the SPSC queue is empty, the job has no dependency fences, and credits
+ *   are available, the job is submitted inline on the calling thread without
+ *   touching the submit workqueue.
+ *
+ * - **Queued path** (drm_dep_queue_push_job()): the job is pushed onto an
+ *   SPSC queue and the run_job worker is kicked. The run_job worker pops the
+ *   job, resolves any remaining dependency fences (installing wakeup
+ *   callbacks for unresolved ones), and calls drm_dep_queue_run_job().
+ *
+ * **Running a job**
+ *
+ * drm_dep_queue_run_job() accounts credits, appends the job to the pending
+ * list (starting the TDR timer only when the list was previously empty),
+ * calls @ops->run_job(), stores the returned hardware fence as the parent
+ * of the job's dep fence, then installs a callback on it. When the hardware
+ * fence fires (or the job completes synchronously), drm_dep_job_done()
+ * signals the finished fence, returns credits, and kicks the put_job worker
+ * to free the job.
+ *
+ * **Timeout detection and recovery (TDR)**
+ *
+ * A delayed work item fires when a job on the pending list takes longer than
+ * @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on the
+ * returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or
+ * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB).
+ * drm_dep_queue_trigger_timeout() forces the timer to fire immediately (without
+ * changing the stored timeout), for example during device teardown.
+ *
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue from drm_dep_job_init() until
+ * drm_dep_job_put() drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * in drm_dep_queue_run_job() until drm_dep_job_done() kicks the put_job
+ * worker, which calls drm_dep_job_put() to release that reference.
+ *
+ * **Hazard: use-after-free from within a worker**
+ *
+ * Because a job holds a queue reference, drm_dep_job_put() dropping the last
+ * job reference will also drop a queue reference via the job's release path.
+ * If that happens to be the last queue reference, drm_dep_queue_fini() can be
+ * called, which queues @q->free_work on dep_free_wq and returns immediately.
+ * free_work calls disable_work_sync() / disable_delayed_work_sync() on the
+ * queue's own workers before destroying its workqueues, so in practice a
+ * running worker always completes before the queue memory is freed.
+ *
+ * However, there is a secondary hazard: a worker can be queued while the
+ * queue is in a "zombie" state — refcount has already reached zero and async
+ * teardown is in flight, but the work item has not yet been disabled by
+ * free_work.  To guard against this every worker uses
+ * drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero
+ * the worker bails immediately without touching the queue state.
+ *
+ * Because all actual teardown (disable_*_sync, destroy_workqueue) runs on
+ * dep_free_wq — which is independent of the queue's own submit/timeout
+ * workqueues — there is no deadlock risk.  Each queue holds a drm_dev_get()
+ * reference on its owning &drm_device, which is released as the last step of
+ * teardown.  This ensures the driver module cannot be unloaded while any queue
+ * is still alive.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/module.h>
+#include <linux/overflow.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_print.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/*
+ * Dedicated workqueue for deferred drm_dep_queue teardown.  Using a
+ * module-private WQ instead of system_percpu_wq keeps teardown isolated
+ * from unrelated kernel subsystems.
+ */
+static struct workqueue_struct *dep_free_wq;
+
+/**
+ * drm_dep_queue_flags_set() - set a flag on the queue under sched.lock
+ * @q: dep queue
+ * @flag: flag to set (one of &enum drm_dep_queue_flags)
+ *
+ * Sets @flag in @q->sched.flags. Must be called with @q->sched.lock
+ * held; the lockdep assertion enforces this.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_flags_set(struct drm_dep_queue *q,
+				    enum drm_dep_queue_flags flag)
+{
+	lockdep_assert_held(&q->sched.lock);
+	q->sched.flags |= flag;
+}
+
+/**
+ * drm_dep_queue_flags_clear() - clear a flag on the queue under sched.lock
+ * @q: dep queue
+ * @flag: flag to clear (one of &enum drm_dep_queue_flags)
+ *
+ * Clears @flag in @q->sched.flags. Must be called with @q->sched.lock
+ * held; the lockdep assertion enforces this.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_flags_clear(struct drm_dep_queue *q,
+				      enum drm_dep_queue_flags flag)
+{
+	lockdep_assert_held(&q->sched.lock);
+	q->sched.flags &= ~flag;
+}
+
+/**
+ * drm_dep_queue_has_credits() - check whether the queue has enough credits
+ * @q: dep queue
+ * @job: job requesting credits
+ *
+ * Checks whether the queue has enough available credits to dispatch
+ * @job. If @job->credits exceeds the queue's credit limit, it is
+ * clamped with a WARN.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if available credits >= @job->credits, false otherwise.
+ */
+static bool drm_dep_queue_has_credits(struct drm_dep_queue *q,
+				      struct drm_dep_job *job)
+{
+	u32 available;
+
+	lockdep_assert_held(&q->sched.lock);
+
+	if (job->credits > q->credit.limit) {
+		drm_warn(q->drm,
+			 "Jobs may not exceed the credit limit, truncate.\n");
+		job->credits = q->credit.limit;
+	}
+
+	WARN_ON(check_sub_overflow(q->credit.limit,
+				   atomic_read(&q->credit.count),
+				   &available));
+
+	return available >= job->credits;
+}
+
+/**
+ * drm_dep_queue_run_job_queue() - kick the run-job worker
+ * @q: dep queue
+ *
+ * Queues @q->sched.run_job on @q->sched.submit_wq unless the queue is stopped
+ * or the job queue is empty.  The empty-queue check avoids queueing a work item
+ * that would immediately return with nothing to do.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_run_job_queue(struct drm_dep_queue *q)
+{
+	if (!drm_dep_queue_is_stopped(q) && spsc_queue_count(&q->job.queue))
+		queue_work(q->sched.submit_wq, &q->sched.run_job);
+}
+
+/**
+ * drm_dep_queue_put_job_queue() - kick the put-job worker
+ * @q: dep queue
+ *
+ * Queues @q->sched.put_job on @q->sched.submit_wq unless the queue
+ * is stopped.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_put_job_queue(struct drm_dep_queue *q)
+{
+	if (!drm_dep_queue_is_stopped(q))
+		queue_work(q->sched.submit_wq, &q->sched.put_job);
+}
+
+/**
+ * drm_queue_start_timeout() - arm or re-arm the TDR delayed work
+ * @q: dep queue
+ *
+ * Arms the TDR delayed work with @q->job.timeout. No-op if
+ * @q->ops->timedout_job is NULL, the timeout is MAX_SCHEDULE_TIMEOUT,
+ * or the pending list is empty.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_queue_start_timeout(struct drm_dep_queue *q)
+{
+	lockdep_assert_held(&q->job.lock);
+
+	if (!q->ops->timedout_job ||
+	    q->job.timeout == MAX_SCHEDULE_TIMEOUT ||
+	    list_empty(&q->job.pending))
+		return;
+
+	mod_delayed_work(q->sched.timeout_wq, &q->sched.tdr, q->job.timeout);
+}
+
+/**
+ * drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock
+ * @q: dep queue
+ *
+ * Acquires @q->job.lock with irq and calls
+ * drm_queue_start_timeout().
+ *
+ * Context: Process context (workqueue).
+ */
+static void drm_queue_start_timeout_unlocked(struct drm_dep_queue *q)
+{
+	guard(spinlock_irq)(&q->job.lock);
+	drm_queue_start_timeout(q);
+}
+
+/**
+ * drm_dep_queue_remove_dependency() - clear the active dependency and wake
+ *   the run-job worker
+ * @q: dep queue
+ * @f: the dependency fence being removed
+ *
+ * Stores @f into @q->dep.removed_fence via smp_store_release() so that the
+ * run-job worker can drop the reference to it in drm_dep_queue_is_ready(),
+ * paired with smp_load_acquire().  Clears @q->dep.fence and kicks the
+ * run-job worker.
+ *
+ * The fence reference is not dropped here; it is deferred to the run-job
+ * worker via @q->dep.removed_fence to keep this path suitable dma_fence
+ * callback removal in drm_dep_queue_kill().
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_remove_dependency(struct drm_dep_queue *q,
+					    struct dma_fence *f)
+{
+	/* removed_fence must be visible to the reader before &q->dep.fence */
+	smp_store_release(&q->dep.removed_fence, f);
+
+	WRITE_ONCE(q->dep.fence, NULL);
+	drm_dep_queue_run_job_queue(q);
+}
+
+/**
+ * drm_dep_queue_wakeup() - dma_fence callback to wake the run-job worker
+ * @f: the signalled dependency fence
+ * @cb: callback embedded in the dep queue
+ *
+ * Called from dma_fence_signal() when the active dependency fence signals.
+ * Delegates to drm_dep_queue_remove_dependency() to clear @q->dep.fence and
+ * kick the run-job worker.  The fence reference is not dropped here; it is
+ * deferred to the run-job worker via @q->dep.removed_fence.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+	struct drm_dep_queue *q =
+		container_of(cb, struct drm_dep_queue, dep.cb);
+
+	drm_dep_queue_remove_dependency(q, f);
+}
+
+/**
+ * drm_dep_queue_is_ready() - check whether the queue has a dispatchable job
+ * @q: dep queue
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if SPSC queue non-empty and no dep fence pending,
+ *   false otherwise.
+ */
+static bool drm_dep_queue_is_ready(struct drm_dep_queue *q)
+{
+	lockdep_assert_held(&q->sched.lock);
+
+	if (!spsc_queue_count(&q->job.queue))
+		return false;
+
+	if (READ_ONCE(q->dep.fence))
+		return false;
+
+	/* Paired with smp_store_release in drm_dep_queue_remove_dependency() */
+	dma_fence_put(smp_load_acquire(&q->dep.removed_fence));
+
+	q->dep.removed_fence = NULL;
+
+	return true;
+}
+
+/**
+ * drm_dep_queue_is_killed() - check whether a dep queue has been killed
+ * @q: dep queue to check
+ *
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_killed(struct drm_dep_queue *q)
+{
+	return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_KILLED);
+}
+EXPORT_SYMBOL(drm_dep_queue_is_killed);
+
+/**
+ * drm_dep_queue_is_initialized() - check whether a dep queue has been initialized
+ * @q: dep queue to check
+ *
+ * A queue is considered initialized once its ops pointer has been set by a
+ * successful call to drm_dep_queue_init().  Drivers that embed a
+ * &drm_dep_queue inside a larger structure may call this before attempting any
+ * other queue operation to confirm that initialization has taken place.
+ * drm_dep_queue_put() must be called if this function returns true to drop the
+ * initialization reference from drm_dep_queue_init().
+ *
+ * Return: true if @q has been initialized, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_initialized(struct drm_dep_queue *q)
+{
+	return !!q->ops;
+}
+EXPORT_SYMBOL(drm_dep_queue_is_initialized);
+
+/**
+ * drm_dep_queue_set_stopped() - pre-mark a queue as stopped before first use
+ * @q: dep queue to mark
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going through the
+ * normal drm_dep_queue_stop() path.  This is only valid during the driver-side
+ * queue initialisation sequence — i.e. after drm_dep_queue_init() returns but
+ * before the queue is made visible to other threads (e.g. before it is added
+ * to any lookup structures).  Using this after the queue is live is a driver
+ * bug; use drm_dep_queue_stop() instead.
+ *
+ * Context: Process context, queue not yet visible to other threads.
+ */
+void drm_dep_queue_set_stopped(struct drm_dep_queue *q)
+{
+	q->sched.flags |= DRM_DEP_QUEUE_FLAGS_STOPPED;
+}
+EXPORT_SYMBOL(drm_dep_queue_set_stopped);
+
+/**
+ * drm_dep_queue_refcount() - read the current reference count of a queue
+ * @q: dep queue to query
+ *
+ * Returns the instantaneous kref value.  The count may change immediately
+ * after this call; callers must not make safety decisions based solely on
+ * the returned value.  Intended for diagnostic snapshots and debugfs output.
+ *
+ * Context: Any context.
+ * Return: current reference count.
+ */
+unsigned int drm_dep_queue_refcount(const struct drm_dep_queue *q)
+{
+	return kref_read(&q->refcount);
+}
+EXPORT_SYMBOL(drm_dep_queue_refcount);
+
+/**
+ * drm_dep_queue_timeout() - read the per-job TDR timeout for a queue
+ * @q: dep queue to query
+ *
+ * Returns the per-job timeout in jiffies as set at init time.
+ * %MAX_SCHEDULE_TIMEOUT means no timeout is configured.
+ *
+ * Context: Any context.
+ * Return: timeout in jiffies.
+ */
+long drm_dep_queue_timeout(const struct drm_dep_queue *q)
+{
+	return q->job.timeout;
+}
+EXPORT_SYMBOL(drm_dep_queue_timeout);
+
+/**
+ * drm_dep_queue_is_job_put_irq_safe() - test whether job-put from IRQ is allowed
+ * @q: dep queue
+ *
+ * Context: Any context.
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set,
+ *   false otherwise.
+ */
+static bool drm_dep_queue_is_job_put_irq_safe(const struct drm_dep_queue *q)
+{
+	return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE);
+}
+
+/**
+ * drm_dep_queue_job_dependency() - get next unresolved dep fence
+ * @q: dep queue
+ * @job: job whose dependencies to advance
+ *
+ * Returns NULL immediately if the queue has been killed via
+ * drm_dep_queue_kill(), bypassing all dependency waits so that jobs
+ * drain through run_job as quickly as possible.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: next unresolved &dma_fence with a new reference, or NULL
+ *   when all dependencies have been consumed (or the queue is killed).
+ */
+static struct dma_fence *
+drm_dep_queue_job_dependency(struct drm_dep_queue *q,
+			     struct drm_dep_job *job)
+{
+	struct dma_fence *f;
+
+	lockdep_assert_held(&q->sched.lock);
+
+	if (drm_dep_queue_is_killed(q))
+		return NULL;
+
+	f = xa_load(&job->dependencies, job->last_dependency);
+	if (f) {
+		job->last_dependency++;
+		if (WARN_ON(DRM_DEP_JOB_FENCE_PREALLOC == f))
+			return dma_fence_get_stub();
+		return dma_fence_get(f);
+	}
+
+	return NULL;
+}
+
+/**
+ * drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence
+ * @q: dep queue
+ * @job: job whose dependency fence is stored in @q->dep.fence
+ *
+ * Installs a wakeup callback on @q->dep.fence. Returns true if the
+ * callback was installed (the queue must wait), false if the fence is
+ * already signalled or is a self-fence from the same queue context.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if callback installed, false if fence already done.
+ */
+static bool drm_dep_queue_add_dep_cb(struct drm_dep_queue *q,
+				     struct drm_dep_job *job)
+{
+	struct dma_fence *fence = q->dep.fence;
+
+	lockdep_assert_held(&q->sched.lock);
+
+	if (WARN_ON(fence->context == q->fence.context)) {
+		dma_fence_put(q->dep.fence);
+		q->dep.fence = NULL;
+		return false;
+	}
+
+	if (!dma_fence_add_callback(q->dep.fence, &q->dep.cb,
+				    drm_dep_queue_wakeup))
+		return true;
+
+	dma_fence_put(q->dep.fence);
+	q->dep.fence = NULL;
+
+	return false;
+}
+
+/**
+ * drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC queue
+ * @q: dep queue
+ *
+ * Peeks at the head of the SPSC queue and drains all resolved
+ * dependencies. If a dependency is still pending, installs a wakeup
+ * callback and returns NULL. On success pops the job and returns it.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: next dispatchable job, or NULL if a dep is still pending.
+ */
+static struct drm_dep_job *drm_dep_queue_pop_job(struct drm_dep_queue *q)
+{
+	struct spsc_node *node;
+	struct drm_dep_job *job;
+
+	lockdep_assert_held(&q->sched.lock);
+
+	node = spsc_queue_peek(&q->job.queue);
+	if (!node)
+		return NULL;
+
+	job = container_of(node, struct drm_dep_job, queue_node);
+
+	while ((q->dep.fence = drm_dep_queue_job_dependency(q, job))) {
+		if (drm_dep_queue_add_dep_cb(q, job))
+			return NULL;
+	}
+
+	spsc_queue_pop(&q->job.queue);
+
+	return job;
+}
+
+/*
+ * drm_dep_queue_get_unless_zero() - try to acquire a queue reference
+ *
+ * Workers use this instead of drm_dep_queue_get() to guard against the zombie
+ * state: the queue's refcount has already reached zero (async teardown is in
+ * flight) but a work item was queued before free_work had a chance to cancel
+ * it.  If kref_get_unless_zero() fails the caller must bail immediately.
+ *
+ * Context: Any context.
+ * Returns true if the reference was acquired, false if the queue is zombie.
+ */
+bool drm_dep_queue_get_unless_zero(struct drm_dep_queue *q)
+{
+	return kref_get_unless_zero(&q->refcount);
+}
+EXPORT_SYMBOL(drm_dep_queue_get_unless_zero);
+
+/**
+ * drm_dep_queue_run_job_work() - run-job worker
+ * @work: work item embedded in the dep queue
+ *
+ * Acquires @q->sched.lock, checks stopped state, queue readiness and
+ * available credits, pops the next job via drm_dep_queue_pop_job(),
+ * dispatches it via drm_dep_queue_run_job(), then re-kicks itself.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_run_job_work(struct work_struct *work)
+{
+	struct drm_dep_queue *q =
+		container_of(work, struct drm_dep_queue, sched.run_job);
+	struct spsc_node *node;
+	struct drm_dep_job *job;
+	bool cookie = dma_fence_begin_signalling();
+
+	/* Bail if queue is zombie (refcount already zero, teardown in flight). */
+	if (!drm_dep_queue_get_unless_zero(q)) {
+		dma_fence_end_signalling(cookie);
+		return;
+	}
+
+	mutex_lock(&q->sched.lock);
+
+	if (drm_dep_queue_is_stopped(q))
+		goto put_queue;
+
+	if (!drm_dep_queue_is_ready(q))
+		goto put_queue;
+
+	/* Peek to check credits before committing to pop and dep resolution */
+	node = spsc_queue_peek(&q->job.queue);
+	if (!node)
+		goto put_queue;
+
+	job = container_of(node, struct drm_dep_job, queue_node);
+	if (!drm_dep_queue_has_credits(q, job))
+		goto put_queue;
+
+	job = drm_dep_queue_pop_job(q);
+	if (!job)
+		goto put_queue;
+
+	drm_dep_queue_run_job(q, job);
+	drm_dep_queue_run_job_queue(q);
+
+put_queue:
+	mutex_unlock(&q->sched.lock);
+	drm_dep_queue_put(q);
+	dma_fence_end_signalling(cookie);
+}
+
+/*
+ * drm_dep_queue_remove_job() - unlink a job from the pending list and reset TDR
+ * @q:   dep queue owning @job
+ * @job: job to remove
+ *
+ * Splices @job out of @q->job.pending, cancels any pending TDR delayed work,
+ * and arms the timeout for the new list head (if any).
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_remove_job(struct drm_dep_queue *q,
+				     struct drm_dep_job *job)
+{
+	lockdep_assert_held(&q->job.lock);
+
+	list_del_init(&job->pending_link);
+	cancel_delayed_work(&q->sched.tdr);
+	drm_queue_start_timeout(q);
+}
+
+/**
+ * drm_dep_queue_get_finished_job() - dequeue a finished job
+ * @q: dep queue
+ *
+ * Under @q->job.lock checks the head of the pending list for a
+ * finished dep fence. If found, removes the job from the list,
+ * cancels the TDR, and re-arms it for the new head.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ * Return: the finished &drm_dep_job, or NULL if none is ready.
+ */
+static struct drm_dep_job *
+drm_dep_queue_get_finished_job(struct drm_dep_queue *q)
+{
+	struct drm_dep_job *job;
+
+	guard(spinlock_irq)(&q->job.lock);
+
+	job = list_first_entry_or_null(&q->job.pending, struct drm_dep_job,
+				       pending_link);
+	if (job && drm_dep_fence_is_finished(job->dfence))
+		drm_dep_queue_remove_job(q, job);
+	else
+		job = NULL;
+
+	return job;
+}
+
+/**
+ * drm_dep_queue_put_job_work() - put-job worker
+ * @work: work item embedded in the dep queue
+ *
+ * Drains all finished jobs by calling drm_dep_job_put() in a loop,
+ * then kicks the run-job worker.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * because workqueue is shared with other items in the fence signaling path.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_put_job_work(struct work_struct *work)
+{
+	struct drm_dep_queue *q =
+		container_of(work, struct drm_dep_queue, sched.put_job);
+	struct drm_dep_job *job;
+	bool cookie = dma_fence_begin_signalling();
+
+	/* Bail if queue is zombie (refcount already zero, teardown in flight). */
+	if (!drm_dep_queue_get_unless_zero(q)) {
+		dma_fence_end_signalling(cookie);
+		return;
+	}
+
+	while ((job = drm_dep_queue_get_finished_job(q)))
+		drm_dep_job_put(job);
+
+	drm_dep_queue_run_job_queue(q);
+
+	drm_dep_queue_put(q);
+	dma_fence_end_signalling(cookie);
+}
+
+/**
+ * drm_dep_queue_tdr_work() - TDR worker
+ * @work: work item embedded in the delayed TDR work
+ *
+ * Removes the head job from the pending list under @q->job.lock,
+ * asserts @q->ops->timedout_job is non-NULL, calls it outside the lock,
+ * requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the
+ * queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and always
+ * restarts the TDR timer after handling the job (unless @q is stopping).
+ * Any other return value triggers a WARN.
+ *
+ * The TDR is never armed when @q->ops->timedout_job is NULL, so firing
+ * this worker without a timedout_job callback is a driver bug.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * because timedout_job() is expected to signal the guilty job's fence as part
+ * of reset.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_tdr_work(struct work_struct *work)
+{
+	struct drm_dep_queue *q =
+		container_of(work, struct drm_dep_queue, sched.tdr.work);
+	struct drm_dep_job *job;
+	bool cookie = dma_fence_begin_signalling();
+
+	/* Bail if queue is zombie (refcount already zero, teardown in flight). */
+	if (!drm_dep_queue_get_unless_zero(q)) {
+		dma_fence_end_signalling(cookie);
+		return;
+	}
+
+	scoped_guard(spinlock_irq, &q->job.lock) {
+		job = list_first_entry_or_null(&q->job.pending,
+					       struct drm_dep_job,
+					       pending_link);
+		if (job)
+			/*
+			 * Remove from pending so it cannot be freed
+			 * concurrently by drm_dep_queue_get_finished_job() or
+			 * .drm_dep_job_done().
+			 */
+			list_del_init(&job->pending_link);
+	}
+
+	if (job) {
+		enum drm_dep_timedout_stat status;
+
+		if (WARN_ON(!q->ops->timedout_job)) {
+			drm_dep_job_put(job);
+			goto out;
+		}
+
+		status = q->ops->timedout_job(job);
+
+		switch (status) {
+		case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB:
+			scoped_guard(spinlock_irq, &q->job.lock)
+				list_add(&job->pending_link, &q->job.pending);
+			drm_dep_queue_put_job_queue(q);
+			break;
+		case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED:
+			drm_dep_job_put(job);
+			break;
+		default:
+			WARN_ON("invalid drm_dep_timedout_stat");
+			break;
+		}
+	}
+
+out:
+	drm_queue_start_timeout_unlocked(q);
+	drm_dep_queue_put(q);
+	dma_fence_end_signalling(cookie);
+}
+
+/**
+ * drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue
+ * @name: name for the workqueue
+ * @flags: DRM_DEP_QUEUE_FLAGS_* flags
+ *
+ * Allocates an ordered workqueue for job submission with %WQ_MEM_RECLAIM and
+ * %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use from
+ * memory reclaim context and properly annotated for lockdep taint tracking.
+ * Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When
+ * CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for annotation.
+ *
+ * Context: Process context.
+ * Return: the new &workqueue_struct, or NULL on failure.
+ */
+static struct workqueue_struct *
+drm_dep_alloc_submit_wq(const char *name, enum drm_dep_queue_flags flags)
+{
+	unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM;
+
+	if (flags & DRM_DEP_QUEUE_FLAGS_HIGHPRI)
+		wq_flags |= WQ_HIGHPRI;
+
+#if IS_ENABLED(CONFIG_LOCKDEP)
+	static struct lockdep_map map = {
+		.name = "drm_dep_submit_lockdep_map"
+	};
+	return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map);
+#else
+	return alloc_ordered_workqueue(name, wq_flags);
+#endif
+}
+
+/**
+ * drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue
+ * @name: name for the workqueue
+ *
+ * Allocates an ordered workqueue for timeout detection and recovery with
+ * %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring consistent taint
+ * annotation with the submit workqueue. When CONFIG_LOCKDEP is enabled, uses
+ * a dedicated lockdep map for annotation.
+ *
+ * Context: Process context.
+ * Return: the new &workqueue_struct, or NULL on failure.
+ */
+static struct workqueue_struct *drm_dep_alloc_timeout_wq(const char *name)
+{
+	unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM;
+
+#if IS_ENABLED(CONFIG_LOCKDEP)
+	static struct lockdep_map map = {
+		.name = "drm_dep_timeout_lockdep_map"
+	};
+	return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map);
+#else
+	return alloc_ordered_workqueue(name, wq_flags);
+#endif
+}
+
+/**
+ * drm_dep_queue_init() - initialize a dep queue
+ * @q: dep queue to initialize
+ * @args: initialization arguments
+ *
+ * Initializes all fields of @q from @args. If @args->submit_wq is NULL an
+ * ordered workqueue is allocated and owned by the queue
+ * (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL an
+ * ordered workqueue is allocated and owned by the queue
+ * (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds one kref
+ * reference and drm_dep_queue_put() must be called to drop this reference
+ * (i.e., drivers cannot directly free the queue).
+ *
+ * When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against the
+ * fs_reclaim pseudo-lock so that lockdep can detect any lock ordering
+ * inversion between @sched.lock and memory reclaim.
+ *
+ * Return: 0 on success, %-EINVAL when @args->credit_limit is zero, @args->ops
+ * is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when
+ * @args->submit_wq or @args->timeout_wq is non-NULL but was not allocated with
+ * %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails.
+ *
+ * Context: Process context. May allocate memory and create workqueues.
+ */
+int drm_dep_queue_init(struct drm_dep_queue *q,
+		       const struct drm_dep_queue_init_args *args)
+{
+	if (!args->credit_limit || !args->drm || !args->ops ||
+	    !args->ops->run_job)
+		return -EINVAL;
+
+	if (args->submit_wq && !workqueue_is_reclaim_annotated(args->submit_wq))
+		return -EINVAL;
+
+	if (args->timeout_wq &&
+	    !workqueue_is_reclaim_annotated(args->timeout_wq))
+		return -EINVAL;
+
+	memset(q, 0, sizeof(*q));
+
+	q->name = args->name;
+	q->drm = args->drm;
+	q->credit.limit = args->credit_limit;
+	q->job.timeout = args->timeout ? args->timeout : MAX_SCHEDULE_TIMEOUT;
+
+	init_rcu_head(&q->rcu);
+	INIT_LIST_HEAD(&q->job.pending);
+	spin_lock_init(&q->job.lock);
+	spsc_queue_init(&q->job.queue);
+
+	mutex_init(&q->sched.lock);
+	if (IS_ENABLED(CONFIG_LOCKDEP)) {
+		fs_reclaim_acquire(GFP_KERNEL);
+		might_lock(&q->sched.lock);
+		fs_reclaim_release(GFP_KERNEL);
+	}
+
+	if (args->submit_wq) {
+		q->sched.submit_wq = args->submit_wq;
+	} else {
+		q->sched.submit_wq = drm_dep_alloc_submit_wq(args->name ?: "drm_dep",
+							     args->flags);
+		if (!q->sched.submit_wq)
+			return -ENOMEM;
+
+		q->sched.flags |= DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ;
+	}
+
+	if (args->timeout_wq) {
+		q->sched.timeout_wq = args->timeout_wq;
+	} else {
+		q->sched.timeout_wq = drm_dep_alloc_timeout_wq(args->name ?: "drm_dep");
+		if (!q->sched.timeout_wq)
+			goto err_submit_wq;
+
+		q->sched.flags |= DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ;
+	}
+
+	q->sched.flags |= args->flags &
+		~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ |
+		  DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ);
+
+	INIT_DELAYED_WORK(&q->sched.tdr, drm_dep_queue_tdr_work);
+	INIT_WORK(&q->sched.run_job, drm_dep_queue_run_job_work);
+	INIT_WORK(&q->sched.put_job, drm_dep_queue_put_job_work);
+
+	q->fence.context = dma_fence_context_alloc(1);
+
+	kref_init(&q->refcount);
+	q->ops = args->ops;
+	drm_dev_get(q->drm);
+
+	return 0;
+
+err_submit_wq:
+	if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ)
+		destroy_workqueue(q->sched.submit_wq);
+	mutex_destroy(&q->sched.lock);
+
+	return -ENOMEM;
+}
+EXPORT_SYMBOL(drm_dep_queue_init);
+
+#if IS_ENABLED(CONFIG_PROVE_LOCKING)
+/**
+ * drm_dep_queue_push_job_begin() - mark the start of an arm/push critical section
+ * @q: dep queue the job belongs to
+ *
+ * Called at the start of drm_dep_job_arm() and warns if the push context is
+ * already owned by another task, which would indicate concurrent arm/push on
+ * the same queue.
+ *
+ * No-op when CONFIG_PROVE_LOCKING is disabled.
+ *
+ * Context: Process context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job_begin(struct drm_dep_queue *q)
+{
+	WARN_ON(q->job.push.owner);
+	q->job.push.owner = current;
+}
+
+/**
+ * drm_dep_queue_push_job_end() - mark the end of an arm/push critical section
+ * @q: dep queue the job belongs to
+ *
+ * Called at the end of drm_dep_job_push() and warns if the push context is not
+ * owned by the current task, which would indicate a mismatched begin/end pair
+ * or a push from the wrong thread.
+ *
+ * No-op when CONFIG_PROVE_LOCKING is disabled.
+ *
+ * Context: Process context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job_end(struct drm_dep_queue *q)
+{
+	WARN_ON(q->job.push.owner != current);
+	q->job.push.owner = NULL;
+}
+#endif
+
+/**
+ * drm_dep_queue_assert_teardown_invariants() - assert teardown invariants
+ * @q: dep queue being torn down
+ *
+ * Warns if the pending-job list, the SPSC submission queue, or the credit
+ * counter is non-zero when called, or if the queue still has a non-zero
+ * reference count.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_assert_teardown_invariants(struct drm_dep_queue *q)
+{
+	WARN_ON(!list_empty(&q->job.pending));
+	WARN_ON(spsc_queue_count(&q->job.queue));
+	WARN_ON(atomic_read(&q->credit.count));
+	WARN_ON(drm_dep_queue_refcount(q));
+}
+
+/**
+ * drm_dep_queue_release() - final internal cleanup of a dep queue
+ * @q: dep queue to clean up
+ *
+ * Asserts teardown invariants and destroys internal resources allocated by
+ * drm_dep_queue_init() that cannot be torn down earlier in the teardown
+ * sequence.  Currently this destroys @q->sched.lock.
+ *
+ * Drivers that implement &drm_dep_queue_ops.release **must** call this
+ * function after removing @q from any internal bookkeeping (e.g. lookup
+ * tables or lists) but before freeing the memory that contains @q.  When
+ * &drm_dep_queue_ops.release is NULL, drm_dep follows the default teardown
+ * path and calls this function automatically.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_release(struct drm_dep_queue *q)
+{
+	drm_dep_queue_assert_teardown_invariants(q);
+	mutex_destroy(&q->sched.lock);
+}
+EXPORT_SYMBOL(drm_dep_queue_release);
+
+/**
+ * drm_dep_queue_free() - final cleanup of a dep queue
+ * @q: dep queue to free
+ *
+ * Invokes &drm_dep_queue_ops.release if set, in which case the driver is
+ * responsible for calling drm_dep_queue_release() and freeing @q itself.
+ * If &drm_dep_queue_ops.release is NULL, calls drm_dep_queue_release()
+ * and then frees @q with kfree_rcu().
+ *
+ * In either case, releases the drm_dev_get() reference taken at init time
+ * via drm_dev_put(), allowing the owning &drm_device to be unloaded once
+ * all queues have been freed.
+ *
+ * Context: Process context (workqueue), reclaim safe.
+ */
+static void drm_dep_queue_free(struct drm_dep_queue *q)
+{
+	struct drm_device *drm = q->drm;
+
+	if (q->ops->release) {
+		q->ops->release(q);
+	} else {
+		drm_dep_queue_release(q);
+		kfree_rcu(q, rcu);
+	}
+	drm_dev_put(drm);
+}
+
+/**
+ * drm_dep_queue_free_work() - deferred queue teardown worker
+ * @work: free_work item embedded in the dep queue
+ *
+ * Runs on dep_free_wq. Disables all work items synchronously
+ * (preventing re-queue and waiting for in-flight instances),
+ * destroys any owned workqueues, then calls drm_dep_queue_free().
+ * Running on dep_free_wq ensures destroy_workqueue() is never
+ * called from within one of the queue's own workers (deadlock)
+ * and disable_*_sync() cannot deadlock either.
+ *
+ * Context: Process context (workqueue), reclaim safe.
+ */
+static void drm_dep_queue_free_work(struct work_struct *work)
+{
+	struct drm_dep_queue *q =
+		container_of(work, struct drm_dep_queue, free_work);
+
+	drm_dep_queue_assert_teardown_invariants(q);
+
+	disable_delayed_work_sync(&q->sched.tdr);
+	disable_work_sync(&q->sched.run_job);
+	disable_work_sync(&q->sched.put_job);
+
+	if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ)
+		destroy_workqueue(q->sched.timeout_wq);
+
+	if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ)
+		destroy_workqueue(q->sched.submit_wq);
+
+	drm_dep_queue_free(q);
+}
+
+/**
+ * drm_dep_queue_fini() - tear down a dep queue
+ * @q: dep queue to tear down
+ *
+ * Asserts teardown invariants  and nitiates teardown of @q by queuing the
+ * deferred free work onto tht module-private dep_free_wq workqueue.  The work
+ * item disables any pending TDR and run/put-job work synchronously, destroys
+ * any workqueues that were allocated by drm_dep_queue_init(), and then releases
+ * the queue memory.
+ *
+ * Running teardown from dep_free_wq ensures that destroy_workqueue() is never
+ * called from within one of the queue's own workers (e.g. via
+ * drm_dep_queue_put()), which would deadlock.
+ *
+ * Drivers can wait for all outstanding deferred work to complete by waiting
+ * for the last drm_dev_put() reference on their &drm_device, which is
+ * released as the final step of each queue's teardown.
+ *
+ * Drivers that implement &drm_dep_queue_ops.fini **must** call this
+ * function after removing @q from any device bookkeeping but before freeing the
+ * memory that contains @q.  When &drm_dep_queue_ops.fini is NULL, drm_dep
+ * follows the default teardown path and calls this function automatically.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_fini(struct drm_dep_queue *q)
+{
+	drm_dep_queue_assert_teardown_invariants(q);
+
+	INIT_WORK(&q->free_work, drm_dep_queue_free_work);
+	queue_work(dep_free_wq, &q->free_work);
+}
+EXPORT_SYMBOL(drm_dep_queue_fini);
+
+/**
+ * drm_dep_queue_get() - acquire a reference to a dep queue
+ * @q: dep queue to acquire a reference on, or NULL
+ *
+ * Return: @q with an additional reference held, or NULL if @q is NULL.
+ *
+ * Context: Any context.
+ */
+struct drm_dep_queue *drm_dep_queue_get(struct drm_dep_queue *q)
+{
+	if (q)
+		kref_get(&q->refcount);
+	return q;
+}
+EXPORT_SYMBOL(drm_dep_queue_get);
+
+/**
+ * __drm_dep_queue_release() - kref release callback for a dep queue
+ * @kref: kref embedded in the dep queue
+ *
+ * Calls &drm_dep_queue_ops.fini if set, otherwise calls
+ * drm_dep_queue_fini() to initiate deferred teardown.
+ *
+ * Context: Any context.
+ */
+static void __drm_dep_queue_release(struct kref *kref)
+{
+	struct drm_dep_queue *q =
+		container_of(kref, struct drm_dep_queue, refcount);
+
+	if (q->ops->fini)
+		q->ops->fini(q);
+	else
+		drm_dep_queue_fini(q);
+}
+
+/**
+ * drm_dep_queue_put() - release a reference to a dep queue
+ * @q: dep queue to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_queue_ops.fini if set,
+ * otherwise calls drm_dep_queue_fini(). Final memory release is handled by
+ * &drm_dep_queue_ops.release (which must call drm_dep_queue_release()) if set,
+ * or drm_dep_queue_release() followed by kfree_rcu() otherwise.
+ * Does nothing if @q is NULL.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_put(struct drm_dep_queue *q)
+{
+	if (q)
+		kref_put(&q->refcount, __drm_dep_queue_release);
+}
+EXPORT_SYMBOL(drm_dep_queue_put);
+
+/**
+ * drm_dep_queue_stop() - stop a dep queue from processing new jobs
+ * @q: dep queue to stop
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex)
+ * and @q->job.lock (spinlock_irq), making the flag safe to test from finished
+ * fenced signaling context. Then cancels any in-flight run_job and put_job work
+ * items. Once stopped, the bypass path and the submit workqueue will not
+ * dispatch further jobs nor will any jobs be removed from the pending list.
+ * Call drm_dep_queue_start() to resume processing.
+ *
+ * Context: Process context. Waits for in-flight workers to complete.
+ */
+void drm_dep_queue_stop(struct drm_dep_queue *q)
+{
+	scoped_guard(mutex, &q->sched.lock) {
+		scoped_guard(spinlock_irq, &q->job.lock)
+			drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_STOPPED);
+	}
+	cancel_work_sync(&q->sched.run_job);
+	cancel_work_sync(&q->sched.put_job);
+}
+EXPORT_SYMBOL(drm_dep_queue_stop);
+
+/**
+ * drm_dep_queue_start() - resume a stopped dep queue
+ * @q: dep queue to start
+ *
+ * Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex)
+ * and @q->job.lock (spinlock_irq), making the flag safe to test from IRQ
+ * context. Then re-queues the run_job and put_job work items so that any jobs
+ * pending since the queue was stopped are processed. Must only be called after
+ * drm_dep_queue_stop().
+ *
+ * Context: Process context.
+ */
+void drm_dep_queue_start(struct drm_dep_queue *q)
+{
+	scoped_guard(mutex, &q->sched.lock) {
+		scoped_guard(spinlock_irq, &q->job.lock)
+			drm_dep_queue_flags_clear(q, DRM_DEP_QUEUE_FLAGS_STOPPED);
+	}
+	drm_dep_queue_run_job_queue(q);
+	drm_dep_queue_put_job_queue(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_start);
+
+/**
+ * drm_dep_queue_trigger_timeout() - trigger the TDR immediately for
+ *   all pending jobs
+ * @q: dep queue to trigger timeout on
+ *
+ * Sets @q->job.timeout to 1 and arms the TDR delayed work with a one-jiffy
+ * delay, causing it to fire almost immediately without hot-spinning at zero
+ * delay. This is used to force-expire any pendind jobs on the queue, for
+ * example when the device is being torn down or has encountered an
+ * unrecoverable error.
+ *
+ * It is suggested that when this function is used, the first timedout_job call
+ * causes the driver to kick the queue off the hardware and signal all pending
+ * job fences. Subsequent calls continue to signal all pending job fences.
+ *
+ * Has no effect if the pending list is empty.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_trigger_timeout(struct drm_dep_queue *q)
+{
+	guard(spinlock_irqsave)(&q->job.lock);
+	q->job.timeout = 1;
+	drm_queue_start_timeout(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_trigger_timeout);
+
+/**
+ * drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait
+ *   for it to finish
+ * @q: dep queue whose TDR to cancel
+ *
+ * Cancels the TDR delayed work item if it has not yet started, and waits for
+ * it to complete if it is already running.  After this call returns, the TDR
+ * worker is guaranteed not to be executing and will not fire again until
+ * explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by a new
+ * job being submitted).
+ *
+ * Useful during error recovery or queue teardown when the caller needs to
+ * know that no timeout handling races with its own reset logic.
+ *
+ * Context: Process context. May sleep waiting for the TDR worker to finish.
+ */
+void drm_dep_queue_cancel_tdr_sync(struct drm_dep_queue *q)
+{
+	cancel_delayed_work_sync(&q->sched.tdr);
+}
+EXPORT_SYMBOL(drm_dep_queue_cancel_tdr_sync);
+
+/**
+ * drm_dep_queue_resume_timeout() - restart the TDR timer with the
+ *   configured timeout
+ * @q: dep queue to resume the timeout for
+ *
+ * Restarts the TDR delayed work using @q->job.timeout. Called after device
+ * recovery to give pending jobs a fresh full timeout window. Has no effect
+ * if the pending list is empty.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_resume_timeout(struct drm_dep_queue *q)
+{
+	drm_queue_start_timeout_unlocked(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_resume_timeout);
+
+/**
+ * drm_dep_queue_is_stopped() - check whether a dep queue is stopped
+ * @q: dep queue to check
+ *
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_stopped(struct drm_dep_queue *q)
+{
+	return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_STOPPED);
+}
+EXPORT_SYMBOL(drm_dep_queue_is_stopped);
+
+/**
+ * drm_dep_queue_kill() - kill a dep queue and flush all pending jobs
+ * @q: dep queue to kill
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock.  If a
+ * dependency fence is currently being waited on, its callback is removed and
+ * the run-job worker is kicked immediately so that the blocked job drains
+ * without waiting.
+ *
+ * Once killed, drm_dep_queue_job_dependency() returns NULL for all jobs,
+ * bypassing dependency waits so that every queued job drains through
+ * &drm_dep_queue_ops.run_job without blocking.
+ *
+ * The &drm_dep_queue_ops.run_job callback is guaranteed to be called for every
+ * job that was pushed before or after drm_dep_queue_kill(), even during queue
+ * teardown.  Drivers should use this guarantee to perform any necessary
+ * bookkeeping cleanup without executing the actual backend operation when the
+ * queue is killed.
+ *
+ * Unlike drm_dep_queue_stop(), killing is one-way: there is no corresponding
+ * start function.
+ *
+ * **Driver safety requirement**
+ *
+ * drm_dep_queue_kill() must only be called once the driver can guarantee that
+ * no job in the queue will touch memory associated with any of its fences
+ * (i.e., the queue has been removed from the device and will never be put back
+ * on).
+ *
+ * Context: Process context.
+ */
+void drm_dep_queue_kill(struct drm_dep_queue *q)
+{
+	scoped_guard(mutex, &q->sched.lock) {
+		struct dma_fence *fence;
+
+		drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_KILLED);
+
+		/*
+		 * Holding &q->sched.lock guarantees that the run-job work item
+		 * cannot drop its reference to q->dep.fence concurrently, so
+		 * reading q->dep.fence here is safe.
+		 */
+		fence = READ_ONCE(q->dep.fence);
+		if (fence && dma_fence_remove_callback(fence, &q->dep.cb))
+			drm_dep_queue_remove_dependency(q, fence);
+	}
+}
+EXPORT_SYMBOL(drm_dep_queue_kill);
+
+/**
+ * drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep queue
+ * @q: dep queue whose workqueue to retrieve
+ *
+ * Drivers may use this to queue their own work items alongside the queue's
+ * internal run-job and put-job workers — for example to process incoming
+ * messages in the same serialisation domain.
+ *
+ * Prefer drm_dep_queue_work_enqueue() when the only need is to enqueue a
+ * work item, as it additionally checks the stopped state.  Use this accessor
+ * when the workqueue itself is required (e.g. for alloc_ordered_workqueue
+ * replacement or drain_workqueue calls).
+ *
+ * Context: Any context.
+ * Return: the &workqueue_struct used by @q for job submission.
+ */
+struct workqueue_struct *drm_dep_queue_submit_wq(struct drm_dep_queue *q)
+{
+	return q->sched.submit_wq;
+}
+EXPORT_SYMBOL(drm_dep_queue_submit_wq);
+
+/**
+ * drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a dep queue
+ * @q: dep queue whose workqueue to retrieve
+ *
+ * Returns the workqueue used by @q to run TDR (timeout detection and recovery)
+ * work.  Drivers may use this to queue their own timeout-domain work items, or
+ * to call drain_workqueue() when tearing down and needing to ensure all pending
+ * timeout callbacks have completed before proceeding.
+ *
+ * Context: Any context.
+ * Return: the &workqueue_struct used by @q for TDR work.
+ */
+struct workqueue_struct *drm_dep_queue_timeout_wq(struct drm_dep_queue *q)
+{
+	return q->sched.timeout_wq;
+}
+EXPORT_SYMBOL(drm_dep_queue_timeout_wq);
+
+/**
+ * drm_dep_queue_work_enqueue() - queue work on the dep queue's submit workqueue
+ * @q: dep queue to enqueue work on
+ * @work: work item to enqueue
+ *
+ * Queues @work on @q->sched.submit_wq if the queue is not stopped.  This
+ * allows drivers to schedule custom work items that run serialised with the
+ * queue's own run-job and put-job workers.
+ *
+ * Return: true if the work was queued, false if the queue is stopped or the
+ * work item was already pending.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_work_enqueue(struct drm_dep_queue *q,
+				struct work_struct *work)
+{
+	if (drm_dep_queue_is_stopped(q))
+		return false;
+
+	return queue_work(q->sched.submit_wq, work);
+}
+EXPORT_SYMBOL(drm_dep_queue_work_enqueue);
+
+/**
+ * drm_dep_queue_can_job_bypass() - test whether a job can skip the SPSC queue
+ * @q: dep queue
+ * @job: job to test
+ *
+ * A job may bypass the submit workqueue and run inline on the calling thread
+ * if all of the following hold:
+ *
+ *  - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue
+ *  - the queue is not stopped
+ *  - the SPSC submission queue is empty (no other jobs waiting)
+ *  - the queue has enough credits for @job
+ *  - @job has no unresolved dependency fences
+ *
+ * Must be called under @q->sched.lock.
+ *
+ * Context: Process context. Must hold @q->sched.lock (a mutex).
+ * Return: true if the job may be run inline, false otherwise.
+ */
+bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q,
+				  struct drm_dep_job *job)
+{
+	lockdep_assert_held(&q->sched.lock);
+
+	return q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED &&
+		!drm_dep_queue_is_stopped(q) &&
+		!spsc_queue_count(&q->job.queue) &&
+		drm_dep_queue_has_credits(q, job) &&
+		xa_empty(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_done() - mark a job as complete
+ * @job: the job that finished
+ * @result: error code to propagate, or 0 for success
+ *
+ * Subtracts @job->credits from the queue credit counter, then signals the
+ * job's dep fence with @result.
+ *
+ * When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), a
+ * temporary extra reference is taken on @job before signalling the fence.
+ * This prevents a concurrent put-job worker — which may be woken by timeouts or
+ * queue starting — from freeing the job while this function still holds a
+ * pointer to it.  The extra reference is released at the end of the function.
+ *
+ * After signalling, the IRQ-safe path removes the job from the pending list
+ * under @q->job.lock, provided the queue is not stopped.  Removal is skipped
+ * when the queue is stopped so that drm_dep_queue_for_each_pending_job() can
+ * iterate the list without racing with the completion path.  On successful
+ * removal, kicks the run-job worker so the next queued job can be dispatched
+ * immediately, then drops the job reference.  If the job was already removed
+ * by TDR, or removal was skipped because the queue is stopped, kicks the
+ * put-job worker instead to allow the deferred put to complete.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_done(struct drm_dep_job *job, int result)
+{
+	struct drm_dep_queue *q = job->q;
+	bool irq_safe = drm_dep_queue_is_job_put_irq_safe(q), removed = false;
+
+	/*
+	 * Local ref to ensure the put worker—which may be woken by external
+	 * forces (TDR, driver-side queue starting)—doesn't free the job behind
+	 * this function's back after drm_dep_fence_done() while it is still on
+	 * the pending list.
+	 */
+	if (irq_safe)
+		drm_dep_job_get(job);
+
+	atomic_sub(job->credits, &q->credit.count);
+	drm_dep_fence_done(job->dfence, result);
+
+	/* Only safe to touch job after fence signal if we have a local ref. */
+
+	if (irq_safe) {
+		scoped_guard(spinlock_irqsave, &q->job.lock) {
+			removed = !list_empty(&job->pending_link) &&
+				!drm_dep_queue_is_stopped(q);
+
+			/* Guard against TDR operating on job */
+			if (removed)
+				drm_dep_queue_remove_job(q, job);
+		}
+	}
+
+	if (removed) {
+		drm_dep_queue_run_job_queue(q);
+		drm_dep_job_put(job);
+	} else {
+		drm_dep_queue_put_job_queue(q);
+	}
+
+	if (irq_safe)
+		drm_dep_job_put(job);
+}
+
+/**
+ * drm_dep_job_done_cb() - dma_fence callback to complete a job
+ * @f: the hardware fence that signalled
+ * @cb: fence callback embedded in the dep job
+ *
+ * Extracts the job from @cb and calls drm_dep_job_done() with
+ * @f->error as the result.
+ *
+ * Context: Any context, but with IRQ disabled. May not sleep.
+ */
+static void drm_dep_job_done_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+	struct drm_dep_job *job = container_of(cb, struct drm_dep_job, cb);
+
+	drm_dep_job_done(job, f->error);
+}
+
+/**
+ * drm_dep_queue_run_job() - submit a job to hardware and set up
+ *   completion tracking
+ * @q: dep queue
+ * @job: job to run
+ *
+ * Accounts @job->credits against the queue, appends the job to the pending
+ * list, then calls @q->ops->run_job(). The TDR timer is started only when
+ * @job is the first entry on the pending list; subsequent jobs added while
+ * a TDR is already in flight do not reset the timer (which would otherwise
+ * extend the deadline for the already-running head job). Stores the returned
+ * hardware fence as the parent of the job's dep fence, then installs
+ * drm_dep_job_done_cb() on it. If the hardware fence is already signalled
+ * (%-ENOENT from dma_fence_add_callback()) or run_job() returns NULL/error,
+ * the job is completed immediately. Must be called under @q->sched.lock.
+ *
+ * Context: Process context. Must hold @q->sched.lock (a mutex). DMA fence
+ * signaling path.
+ */
+void drm_dep_queue_run_job(struct drm_dep_queue *q, struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	int r;
+
+	lockdep_assert_held(&q->sched.lock);
+
+	drm_dep_job_get(job);
+	atomic_add(job->credits, &q->credit.count);
+
+	scoped_guard(spinlock_irq, &q->job.lock) {
+		bool first = list_empty(&q->job.pending);
+
+		list_add_tail(&job->pending_link, &q->job.pending);
+		if (first)
+			drm_queue_start_timeout(q);
+	}
+
+	fence = q->ops->run_job(job);
+	drm_dep_fence_set_parent(job->dfence, fence);
+
+	if (!IS_ERR_OR_NULL(fence)) {
+		r = dma_fence_add_callback(fence, &job->cb,
+					   drm_dep_job_done_cb);
+		if (r == -ENOENT)
+			drm_dep_job_done(job, fence->error);
+		else if (r)
+			drm_err(q->drm, "fence add callback failed (%d)\n", r);
+		dma_fence_put(fence);
+	} else {
+		drm_dep_job_done(job, IS_ERR(fence) ? PTR_ERR(fence) : 0);
+	}
+
+	/*
+	 * Drop all input dependency fences now, in process context, before the
+	 * final job put. Once the job is on the pending list its last reference
+	 * may be dropped from a dma_fence callback (IRQ context), where calling
+	 * xa_destroy() would be unsafe.
+	 */
+	drm_dep_job_drop_dependencies(job);
+	drm_dep_job_put(job);
+}
+
+/**
+ * drm_dep_queue_push_job() - enqueue a job on the SPSC submission queue
+ * @q: dep queue
+ * @job: job to push
+ *
+ * Pushes @job onto the SPSC queue. If the queue was previously empty
+ * (i.e. this is the first pending job), kicks the run_job worker so it
+ * processes the job promptly without waiting for the next wakeup.
+ * May be called with or without @q->sched.lock held.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job(struct drm_dep_queue *q, struct drm_dep_job *job)
+{
+	/*
+	 * spsc_queue_push() returns true if the queue was previously empty,
+	 * i.e. this is the first pending job. Kick the run_job worker so it
+	 * picks it up without waiting for the next wakeup.
+	 */
+	if (spsc_queue_push(&q->job.queue, &job->queue_node))
+		drm_dep_queue_run_job_queue(q);
+}
+
+/**
+ * drm_dep_init() - module initialiser
+ *
+ * Allocates the module-private dep_free_wq unbound workqueue used for
+ * deferred queue teardown.
+ *
+ * Return: 0 on success, %-ENOMEM if workqueue allocation fails.
+ */
+static int __init drm_dep_init(void)
+{
+	dep_free_wq = alloc_workqueue("drm_dep_free", WQ_UNBOUND, 0);
+	if (!dep_free_wq)
+		return -ENOMEM;
+
+	return 0;
+}
+
+/**
+ * drm_dep_exit() - module exit
+ *
+ * Destroys the module-private dep_free_wq workqueue.
+ */
+static void __exit drm_dep_exit(void)
+{
+	destroy_workqueue(dep_free_wq);
+	dep_free_wq = NULL;
+}
+
+module_init(drm_dep_init);
+module_exit(drm_dep_exit);
+
+MODULE_DESCRIPTION("DRM dependency queue");
+MODULE_LICENSE("Dual MIT/GPL");
drivers/gpu/drm/dep/drm_dep_queue.h new file +31 -0
new file mode 100644
index 000000000000..e5c217a3fab5
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_QUEUE_H_
+#define _DRM_DEP_QUEUE_H_
+
+#include <linux/types.h>
+
+struct drm_dep_job;
+struct drm_dep_queue;
+
+bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q,
+				  struct drm_dep_job *job);
+void drm_dep_queue_run_job(struct drm_dep_queue *q, struct drm_dep_job *job);
+void drm_dep_queue_push_job(struct drm_dep_queue *q, struct drm_dep_job *job);
+
+#if IS_ENABLED(CONFIG_PROVE_LOCKING)
+void drm_dep_queue_push_job_begin(struct drm_dep_queue *q);
+void drm_dep_queue_push_job_end(struct drm_dep_queue *q);
+#else
+static inline void drm_dep_queue_push_job_begin(struct drm_dep_queue *q)
+{
+}
+static inline void drm_dep_queue_push_job_end(struct drm_dep_queue *q)
+{
+}
+#endif
+
+#endif /* _DRM_DEP_QUEUE_H_ */
include/drm/drm_dep.h new file +597 -0
new file mode 100644
index 000000000000..615926584506
@@ -0,0 +1,597 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_H_
+#define _DRM_DEP_H_
+
+#include <drm/spsc_queue.h>
+#include <linux/dma-fence.h>
+#include <linux/xarray.h>
+#include <linux/workqueue.h>
+
+enum dma_resv_usage;
+struct dma_resv;
+struct drm_dep_fence;
+struct drm_dep_job;
+struct drm_dep_queue;
+struct drm_file;
+struct drm_gem_object;
+
+/**
+ * enum drm_dep_timedout_stat - return value of &drm_dep_queue_ops.timedout_job
+ * @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's finished
+ *   fence during reset; drm_dep may safely drop its reference to the job.
+ * @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; reinsert the
+ *   job at the head of the pending list so it can complete normally.
+ */
+enum drm_dep_timedout_stat {
+	DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED,
+	DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB,
+};
+
+/**
+ * struct drm_dep_queue_ops - driver callbacks for a dep queue
+ */
+struct drm_dep_queue_ops {
+	/**
+	 * @run_job: submit the job to hardware. Returns the hardware completion
+	 * fence (with a reference held for the scheduler), or NULL/ERR_PTR on
+	 * synchronous completion or error.
+	 */
+	struct dma_fence *(*run_job)(struct drm_dep_job *job);
+
+	/**
+	 * @timedout_job: called when the TDR fires for the head job. Must stop
+	 * the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if the
+	 * job's fence was signalled during reset, or
+	 * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or
+	 * signalling was otherwise delayed, and the job should be re-inserted
+	 * at the head of the pending list. Any other value triggers a WARN.
+	 */
+	enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job);
+
+	/**
+	 * @release: called when the last kref on the queue is dropped and
+	 * drm_dep_queue_fini() has completed.  The driver is responsible for
+	 * removing @q from any internal bookkeeping, calling
+	 * drm_dep_queue_release(), and then freeing the memory containing @q
+	 * (e.g. via kfree_rcu() using @q->rcu).  If NULL, drm_dep calls
+	 * drm_dep_queue_release() and frees @q automatically via kfree_rcu().
+	 * Use this when the queue is embedded in a larger structure.
+	 */
+	void (*release)(struct drm_dep_queue *q);
+
+	/**
+	 * @fini: if set, called instead of drm_dep_queue_fini() when the last
+	 * kref is dropped. The driver is responsible for calling
+	 * drm_dep_queue_fini() itself after it is done with the queue. Use this
+	 * when additional teardown logic must run before fini (e.g., cleanup
+	 * firmware resources associated with the queue).
+	 */
+	void (*fini)(struct drm_dep_queue *q);
+};
+
+/**
+ * enum drm_dep_queue_flags - flags for &drm_dep_queue and
+ *   &drm_dep_queue_init_args
+ *
+ * Flags are divided into three categories:
+ *
+ * - **Private static**: set internally at init time and never changed.
+ *   Drivers must not read or write these.
+ *   %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ,
+ *   %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ.
+ *
+ * - **Public dynamic**: toggled at runtime by drivers via accessors.
+ *   Any modification must be performed under &drm_dep_queue.sched.lock.
+ *   Accessor functions provide unstable reads.
+ *   %DRM_DEP_QUEUE_FLAGS_STOPPED,
+ *   %DRM_DEP_QUEUE_FLAGS_KILLED.
+ *
+ * - **Public static**: supplied by the driver in
+ *   &drm_dep_queue_init_args.flags at queue creation time and not modified
+ *   thereafter.
+ *   %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED,
+ *   %DRM_DEP_QUEUE_FLAGS_HIGHPRI,
+ *   %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE.
+ *
+ * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit workqueue was
+ *   allocated by drm_dep_queue_init() and will be destroyed by
+ *   drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout workqueue
+ *   was allocated by drm_dep_queue_init() and will be destroyed by
+ *   drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is stopped and
+ *   will not dispatch new jobs or remove jobs from the pending list, dropping
+ *   the drm_dep-owned reference. Set by drm_dep_queue_stop(), cleared by
+ *   drm_dep_queue_start().
+ * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been killed
+ *   via drm_dep_queue_kill(). Any active dependency wait is cancelled
+ *   immediately.  Jobs continue to flow through run_job for bookkeeping
+ *   cleanup, but dependency waiting is skipped so that queued work drains
+ *   as quickly as possible.
+ * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue supports
+ *   the bypass path where eligible jobs skip the SPSC queue and run inline.
+ * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit workqueue owned
+ *   by the queue is created with %WQ_HIGHPRI, causing run-job and put-job
+ *   workers to execute at elevated priority. Only privileged clients (e.g.
+ *   drivers managing time-critical or real-time GPU contexts) should request
+ *   this flag; granting it to unprivileged userspace would allow priority
+ *   inversion attacks.
+ *   @drm_dep_queue_init_args.submit_wq is provided.
+ * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set,
+ *   drm_dep_job_done() may be called from hardirq context (e.g. from a
+ *   hardware-signalled dma_fence callback). drm_dep_job_done() will directly
+ *   dequeue the job and call drm_dep_job_put() without deferring to a
+ *   workqueue. The driver's &drm_dep_job_ops.release callback must therefore
+ *   be safe to invoke from IRQ context.
+ */
+enum drm_dep_queue_flags {
+	DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ	= BIT(0),
+	DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ	= BIT(1),
+	DRM_DEP_QUEUE_FLAGS_STOPPED		= BIT(2),
+	DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED	= BIT(3),
+	DRM_DEP_QUEUE_FLAGS_HIGHPRI		= BIT(4),
+	DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE	= BIT(5),
+	DRM_DEP_QUEUE_FLAGS_KILLED		= BIT(6),
+};
+
+/**
+ * struct drm_dep_queue - a dependency-tracked GPU submission queue
+ *
+ * Combines the role of &drm_gpu_scheduler and &drm_sched_entity into a single
+ * object.  Each queue owns a submit workqueue (or borrows one), a timeout
+ * workqueue, an SPSC submission queue, and a pending-job list used for TDR.
+ *
+ * Initialise with drm_dep_queue_init(), tear down with drm_dep_queue_fini().
+ * Reference counted via drm_dep_queue_get() / drm_dep_queue_put().
+ *
+ * All fields are **opaque to drivers**.  Do not read or write any field
+ * directly; use the provided helper functions instead.  The sole exception
+ * is @rcu, which drivers may pass to kfree_rcu() when the queue is embedded
+ * inside a larger driver-managed structure and the &drm_dep_queue_ops.release
+ * vfunc performs an RCU-deferred free.
+ */
+struct drm_dep_queue {
+	/** @ops: driver callbacks, set at init time. */
+	const struct drm_dep_queue_ops *ops;
+	/** @name: human-readable name used for workqueue and fence naming. */
+	const char *name;
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

maintainer
#37 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

In reply to Matthew Brost
Quoted 32 lines · On Sun, 15 Mar 2026 21:32:45 -0700 Matthew Brost <[email protected]>…
drivers/gpu/drm/dep/drm_dep_fence.c new file +32 -0
+/**
+ * struct drm_dep_fence - fence tracking the completion of a dep job
+ *
+ * Contains a single dma_fence (@finished) that is signalled when the
+ * hardware completes the job. The fence uses the kernel's inline_lock
+ * (no external spinlock required).
+ *
+ * This struct is private to the drm_dep module; external code interacts
+ * through the accessor functions declared in drm_dep_fence.h.
+ */
+struct drm_dep_fence {
+	/**
+	 * @finished: signalled when the job completes on hardware.
+	 *
+	 * Drivers should use this fence as the out-fence for a job since it
+	 * is available immediately upon drm_dep_job_arm().
+	 */
+	struct dma_fence finished;
+
+	/**
+	 * @deadline: deadline set on @finished which potentially needs to be
+	 * propagated to @parent.
+	 */
+	ktime_t	deadline;
+
+	/**
+	 * @parent: The hardware fence returned by &drm_dep_queue_ops.run_job.
+	 *
+	 * @finished is signaled once @parent is signaled. The initial store is
+	 * performed via smp_store_release to synchronize with deadline handling.
+	 *
+	 * All readers must access this under the fence lock and take a reference to

drivers/gpu/drm/dep/drm_dep_fence.c line 89

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_fence.c new file +10 -0
+	 * it, as @parent is set to NULL under the fence lock when the drm_dep_fence
+	 * signals, and this drop also releases its internal reference.
+	 */
+	struct dma_fence *parent;
+
+	/**
+	 * @q: the queue this fence belongs to.
+	 */
+	struct drm_dep_queue *q;
+};

As Daniel pointed out already, with Christian's recent changes to dma_fence (the ones that reset dma_fence::ops after ::signal()), the fence proxy that existed in drm_sched_fence is no longer required:
drivers and their implementations can safely vanish even if some fences they have emitted are still referenced by other subsystems. All we need is:

- fence must be signaled for dma_fence::ops to be set back to NULL
- no .cleanup and no .wait implementation

There might be an interest in having HW submission fences reflecting when the job is passed to the FW/HW queue, but that can done as a separate fence implementation using a different fence timeline/context.

Quoted 169 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +163 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10 -0
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
+	 * drm_dep_job_drop_dependencies() in process context after run_job().

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 82 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +82 -0
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+	if (job)
+		kref_get(&job->refcount);
+	return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+	struct drm_dep_job *job =
+		container_of(kref, struct drm_dep_job, refcount);
+	struct drm_dep_queue *q = job->q;
+
+	drm_dep_job_fini(job);
+
+	if (job->ops && job->ops->release)
+		job->ops->release(job);
+	else
+		kfree(job);
+
+	drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+	if (job)
+		kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *

drivers/gpu/drm/dep/drm_dep_job.c line 269

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10 -0
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+	drm_dep_queue_push_job_begin(job->q);
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	drm_dep_fence_init(job->dfence, job->q);
+	job->signalling_cookie = dma_fence_begin_signalling();

I'd really like DMA-signalling-path annotation to be something that doesn't leak to the job object. The way I see it, in the submit path, it should be some sort of block initializing an opaque token, and drm_dep_job_arm() should expect a valid token to be passed, thus guaranteeing that anything between arm and push, and more generally anything in that section is safe.

	struct drm_job_submit_context submit_ctx;

	// Do all the prep stuff, pre-alloc, resv setup, ...

	// Non-faillible section of the submit starts here.
	// This is properly annotated with
	// dma_fence_{begin,end}_signalling() to ensure we're
	// not taking locks or doing allocations forbidden in
	// the signalling path
	drm_job_submit_non_faillible_section(&submit_ctx) {
		for_each_job() {
			drm_dep_job_arm(&submit_ctx, &job);

			// pass the armed fence around, if needed

			drm_dep_job_push(&submit_ctx, &job);
		}
	}

With the current solution, there's no control that drm_dep_job_{arm,push}() calls are balanced, with the risk of leaving a DMA-signalling annotation behind.

Quoted 120 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +120 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet

drivers/gpu/drm/dep/drm_dep_job.c line 399

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10 -0
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

Quoted 104 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +104 -0
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+	struct drm_dep_queue *q = job->q;
+	struct dma_fence *entry;
+	unsigned long index;
+	u32 id = 0;
+	int ret;
+
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	might_alloc(GFP_KERNEL);
+
+	if (!fence)
+		return 0;
+
+	if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+		goto add_fence;
+
+	/*
+	 * Ignore signalled fences or fences from our own queue — finished
+	 * fences use q->fence.context.
+	 */
+	if (dma_fence_test_signaled_flag(fence) ||
+	    fence->context == q->fence.context) {
+		dma_fence_put(fence);
+		return 0;
+	}
+
+	/* Deduplicate if we already depend on a fence from the same context.
+	 * This lets the size of the array of deps scale with the number of
+	 * engines involved, rather than the number of BOs.
+	 */
+	xa_for_each(&job->dependencies, index, entry) {
+		if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+		    entry->context != fence->context)
+			continue;
+
+		if (dma_fence_is_later(fence, entry)) {
+			dma_fence_put(entry);
+			xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+		} else {
+			dma_fence_put(fence);
+		}
+		return 0;
+	}
+
+add_fence:
+	ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+		       GFP_KERNEL);
+	if (ret != 0) {
+		if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+			dma_fence_put(fence);
+		return ret;
+	}
+
+	return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+				    struct dma_fence *fence)
+{
+	WARN_ON(xa_load(&job->dependencies, index) !=
+		DRM_DEP_JOB_FENCE_PREALLOC);
+
+	if (dma_fence_test_signaled_flag(fence)) {

drivers/gpu/drm/dep/drm_dep_job.c line 513

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10 -0
+		xa_erase(&job->dependencies, index);
+		dma_fence_put(fence);
+		return;
+	}
+
+	if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
+				       GFP_NOWAIT)))) {
+		dma_fence_put(fence);
+		return;
+	}

You don't seem to go for the replace-if-earlier-fence-on-same-context-exists optimization that we have in drm_dep_job_add_dependency(). Any reason not to?

drivers/gpu/drm/dep/drm_dep_job.c line 516

Quoted 3 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +3 -0
+}
+EXPORT_SYMBOL(drm_dep_job_replace_dependency);
+

I'm going to stop here for today.

Regards,

Boris

author maintainer
#38 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 44 lines · Boris Brezillon
Quoted 41 lines · On Sun, 15 Mar 2026 21:32:45 -0700 Matthew Brost <[email protected]>…
drivers/gpu/drm/dep/drm_dep_fence.c new file +41 -0
+/**
+ * struct drm_dep_fence - fence tracking the completion of a dep job
+ *
+ * Contains a single dma_fence (@finished) that is signalled when the
+ * hardware completes the job. The fence uses the kernel's inline_lock
+ * (no external spinlock required).
+ *
+ * This struct is private to the drm_dep module; external code interacts
+ * through the accessor functions declared in drm_dep_fence.h.
+ */
+struct drm_dep_fence {
+	/**
+	 * @finished: signalled when the job completes on hardware.
+	 *
+	 * Drivers should use this fence as the out-fence for a job since it
+	 * is available immediately upon drm_dep_job_arm().
+	 */
+	struct dma_fence finished;
+
+	/**
+	 * @deadline: deadline set on @finished which potentially needs to be
+	 * propagated to @parent.
+	 */
+	ktime_t	deadline;
+
+	/**
+	 * @parent: The hardware fence returned by &drm_dep_queue_ops.run_job.
+	 *
+	 * @finished is signaled once @parent is signaled. The initial store is
+	 * performed via smp_store_release to synchronize with deadline handling.
+	 *
+	 * All readers must access this under the fence lock and take a reference to
+	 * it, as @parent is set to NULL under the fence lock when the drm_dep_fence
+	 * signals, and this drop also releases its internal reference.
+	 */
+	struct dma_fence *parent;
+
+	/**
+	 * @q: the queue this fence belongs to.
+	 */
+	struct drm_dep_queue *q;
Quoted 9 lines
Quoted 1 line

+};

As Daniel pointed out already, with Christian's recent changes to dma_fence (the ones that reset dma_fence::ops after ::signal()), the fence proxy that existed in drm_sched_fence is no longer required:
drivers and their implementations can safely vanish even if some fences they have emitted are still referenced by other subsystems. All we need is:

I believe the late arming or dma fence array / chain would need to address. I've replied in detail in another fork of this thread already so will not cover here.

Quoted 7 lines

- fence must be signaled for dma_fence::ops to be set back to NULL
- no .cleanup and no .wait implementation

There might be an interest in having HW submission fences reflecting when the job is passed to the FW/HW queue, but that can done as a separate fence implementation using a different fence timeline/context.

Yes, I removed scheduled side of drm sched fence as I figured that could be implemented driver side (or as an optional API in drm dep). Only AMDGPU / PVR use these too for ganged submissions which I need to wrap my head around. My initial thought is both of implementations likely could be simplified.

Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ). If we droppped that, then yes this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Quoted 91 lines
Quoted 91 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +91 -0
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+	if (job)
+		kref_get(&job->refcount);
+	return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+	struct drm_dep_job *job =
+		container_of(kref, struct drm_dep_job, refcount);
+	struct drm_dep_queue *q = job->q;
+
+	drm_dep_job_fini(job);
+
+	if (job->ops && job->ops->release)
+		job->ops->release(job);
+	else
+		kfree(job);
+
+	drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+	if (job)
+		kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+	drm_dep_queue_push_job_begin(job->q);
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	drm_dep_fence_init(job->dfence, job->q);
Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 269

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	job->signalling_cookie = dma_fence_begin_signalling();

I'd really like DMA-signalling-path annotation to be something that doesn't leak to the job object. The way I see it, in the submit path, it should be some sort of block initializing an opaque token, and drm_dep_job_arm() should expect a valid token to be passed, thus guaranteeing that anything between arm and push, and more generally anything in that section is safe.

Yes. drm_dep_queue_push_job_begin internally creates a token (current) that is paired drm_dep_queue_push_job_end. If you ever have imbalance between arm() and push() you will get complaints.

Quoted 19 lines
	struct drm_job_submit_context submit_ctx;

	// Do all the prep stuff, pre-alloc, resv setup, ...

	// Non-faillible section of the submit starts here.
	// This is properly annotated with
	// dma_fence_{begin,end}_signalling() to ensure we're
	// not taking locks or doing allocations forbidden in
	// the signalling path
	drm_job_submit_non_faillible_section(&submit_ctx) {
		for_each_job() {
			drm_dep_job_arm(&submit_ctx, &job);

			// pass the armed fence around, if needed

			drm_dep_job_push(&submit_ctx, &job);
		}
	}
Quoted 3 lines

With the current solution, there's no control that drm_dep_job_{arm,push}() calls are balanced, with the risk of leaving a DMA-signalling annotation behind.

See above, that is what drm_dep_queue_push_job_begin/end do.

Quoted 127 lines
Quoted 127 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 399

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

Quoted 5 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Quoted 110 lines
Quoted 110 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +110 -0
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+	struct drm_dep_queue *q = job->q;
+	struct dma_fence *entry;
+	unsigned long index;
+	u32 id = 0;
+	int ret;
+
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	might_alloc(GFP_KERNEL);
+
+	if (!fence)
+		return 0;
+
+	if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+		goto add_fence;
+
+	/*
+	 * Ignore signalled fences or fences from our own queue — finished
+	 * fences use q->fence.context.
+	 */
+	if (dma_fence_test_signaled_flag(fence) ||
+	    fence->context == q->fence.context) {
+		dma_fence_put(fence);
+		return 0;
+	}
+
+	/* Deduplicate if we already depend on a fence from the same context.
+	 * This lets the size of the array of deps scale with the number of
+	 * engines involved, rather than the number of BOs.
+	 */
+	xa_for_each(&job->dependencies, index, entry) {
+		if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+		    entry->context != fence->context)
+			continue;
+
+		if (dma_fence_is_later(fence, entry)) {
+			dma_fence_put(entry);
+			xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+		} else {
+			dma_fence_put(fence);
+		}
+		return 0;
+	}
+
+add_fence:
+	ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+		       GFP_KERNEL);
+	if (ret != 0) {
+		if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+			dma_fence_put(fence);
+		return ret;
+	}
+
+	return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+				    struct dma_fence *fence)
+{
+	WARN_ON(xa_load(&job->dependencies, index) !=
+		DRM_DEP_JOB_FENCE_PREALLOC);
+
+	if (dma_fence_test_signaled_flag(fence)) {
+		xa_erase(&job->dependencies, index);
+		dma_fence_put(fence);
+		return;
+	}
+
+	if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 513

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+				       GFP_NOWAIT)))) {
+		dma_fence_put(fence);
+		return;
+	}

You don't seem to go for the replace-if-earlier-fence-on-same-context-exists optimization that we have in drm_dep_job_add_dependency(). Any reason not to?

No, that could be added in. My reasoning for ommiting was if you are pre-alloc a slot you likely know that the same timeline hasn't already been added in but maybe that is bad assumption.

Matt

Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 516

Quoted 3 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +3 -0
+}
+EXPORT_SYMBOL(drm_dep_job_replace_dependency);
+

I'm going to stop here for today.

Regards,

Boris

maintainer
#39 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Hi Matthew,

Quoted 8 lines · On Wed, 18 Mar 2026 16:28:13 -0700 Matthew Brost <[email protected]>…
Quoted 7 lines

- fence must be signaled for dma_fence::ops to be set back to NULL
- no .cleanup and no .wait implementation

There might be an interest in having HW submission fences reflecting when the job is passed to the FW/HW queue, but that can done as a separate fence implementation using a different fence timeline/context.

Quoted 5 lines

Yes, I removed scheduled side of drm sched fence as I figured that could be implemented driver side (or as an optional API in drm dep). Only AMDGPU / PVR use these too for ganged submissions which I need to wrap my head around. My initial thought is both of implementations likely could be simplified.

IIRC, PVR was also relying on it to allow native FW waits: when we have a job that has deps that are backed by fences emitted by the same driver, they are detected and lowered to waits on the "scheduled" fence, the wait on the finished fence is done FW side.

Quoted 203 lines
Quoted 179 lines
Quoted 179 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().  

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);  

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well just defer the cleanup (AKA the stuff you currently have in _release()) so everything is always cleaned up in a thread context. Yes, there's scheduling overhead and extra latency, but it's also simpler, because there's just one path. So, if the latency and the overhead is not proven to be a problem (and it rarely is for cleanup operations), I'm still convinced this makes for an easier design to just defer the cleanup all the time.

Quoted 4 lines

If we droppped that, then yes
this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

But even in that context, it could still be deferred and work just fine, no?

Quoted 3 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Sure.

Quoted 125 lines
Quoted 92 lines
Quoted 92 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +91 -0
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+	if (job)
+		kref_get(&job->refcount);
+	return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+	struct drm_dep_job *job =
+		container_of(kref, struct drm_dep_job, refcount);
+	struct drm_dep_queue *q = job->q;
+
+	drm_dep_job_fini(job);
+
+	if (job->ops && job->ops->release)
+		job->ops->release(job);
+	else
+		kfree(job);
+
+	drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+	if (job)
+		kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+	drm_dep_queue_push_job_begin(job->q);
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	drm_dep_fence_init(job->dfence, job->q);
Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 269

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	job->signalling_cookie = dma_fence_begin_signalling();  

I'd really like DMA-signalling-path annotation to be something that doesn't leak to the job object. The way I see it, in the submit path, it should be some sort of block initializing an opaque token, and drm_dep_job_arm() should expect a valid token to be passed, thus guaranteeing that anything between arm and push, and more generally anything in that section is safe.

Yes. drm_dep_queue_push_job_begin internally creates a token (current) that is paired drm_dep_queue_push_job_end. If you ever have imbalance between arm() and push() you will get complaints.

Quoted 19 lines
	struct drm_job_submit_context submit_ctx;

	// Do all the prep stuff, pre-alloc, resv setup, ...

	// Non-faillible section of the submit starts here.
	// This is properly annotated with
	// dma_fence_{begin,end}_signalling() to ensure we're
	// not taking locks or doing allocations forbidden in
	// the signalling path
	drm_job_submit_non_faillible_section(&submit_ctx) {
		for_each_job() {
			drm_dep_job_arm(&submit_ctx, &job);

			// pass the armed fence around, if needed

			drm_dep_job_push(&submit_ctx, &job);
		}
	}
Quoted 5 lines
Quoted 3 lines

With the current solution, there's no control that drm_dep_job_{arm,push}() calls are balanced, with the risk of leaving a DMA-signalling annotation behind.

See above, that is what drm_dep_queue_push_job_begin/end do.

That's still error-prone, and the kind of errors you only detect at runtime. Let alone the fact you might not even notice if the unbalanced symptoms are caused by error paths that are rarely tested. I'm proposing something that's designed so you can't make those mistakes unless you really want to:

- drm_job_submit_non_faillible_section() is a block-like macro with a clear scope before/after which the token is invalid
- drm_job_submit_non_faillible_section() is the only place that can produce a valid token (not enforceable in C, but with an __drm_dep_queue_create_submit_token() and proper disclaimer, I guess we can discourage people to inadvertently use it)
- drm_dep_job_{arm,push}() calls requires a valid token to work, and with the two points mentioned above, that means you can't call drm_dep_job_{arm,push}() outside a drm_job_submit_non_faillible_section() block

It's not quite the compile-time checks rust would enforce, but it's a model that forces people to do it the right way, with extra runtime checks for the case where they still got it wrong (like, putting the _arm() and _push() in two different drm_job_submit_non_faillible_section() blocks).

Quoted 152 lines
Quoted 128 lines
Quoted 128 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 399

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));  

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Quoted 2 lines

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

I'm still not entirely convinced by that. To me _arm() is not quite the moment you make your fence public, and I'm not sure the extra complexity added for intra-batch dependencies (one job in a SUBMIT depending on a previous job in the same SUBMIT) is justified, because what really matters is not that we leave dangling/unsignalled dma_fence objects around, the problem is when you do so on an object that has been exposed publicly (syncobj, dma_resv, sync_file, ...).

Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Quoted 121 lines
Quoted 111 lines
Quoted 111 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +110 -0
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+	struct drm_dep_queue *q = job->q;
+	struct dma_fence *entry;
+	unsigned long index;
+	u32 id = 0;
+	int ret;
+
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	might_alloc(GFP_KERNEL);
+
+	if (!fence)
+		return 0;
+
+	if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+		goto add_fence;
+
+	/*
+	 * Ignore signalled fences or fences from our own queue — finished
+	 * fences use q->fence.context.
+	 */
+	if (dma_fence_test_signaled_flag(fence) ||
+	    fence->context == q->fence.context) {
+		dma_fence_put(fence);
+		return 0;
+	}
+
+	/* Deduplicate if we already depend on a fence from the same context.
+	 * This lets the size of the array of deps scale with the number of
+	 * engines involved, rather than the number of BOs.
+	 */
+	xa_for_each(&job->dependencies, index, entry) {
+		if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+		    entry->context != fence->context)
+			continue;
+
+		if (dma_fence_is_later(fence, entry)) {
+			dma_fence_put(entry);
+			xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+		} else {
+			dma_fence_put(fence);
+		}
+		return 0;
+	}
+
+add_fence:
+	ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+		       GFP_KERNEL);
+	if (ret != 0) {
+		if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+			dma_fence_put(fence);
+		return ret;
+	}
+
+	return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+				    struct dma_fence *fence)
+{
+	WARN_ON(xa_load(&job->dependencies, index) !=
+		DRM_DEP_JOB_FENCE_PREALLOC);
+
+	if (dma_fence_test_signaled_flag(fence)) {
+		xa_erase(&job->dependencies, index);
+		dma_fence_put(fence);
+		return;
+	}
+
+	if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 513

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+				       GFP_NOWAIT)))) {
+		dma_fence_put(fence);
+		return;
+	}  

You don't seem to go for the replace-if-earlier-fence-on-same-context-exists optimization that we have in drm_dep_job_add_dependency(). Any reason not to?

Quoted 3 lines

No, that could be added in. My reasoning for ommiting was if you are pre-alloc a slot you likely know that the same timeline hasn't already been added in but maybe that is bad assumption.

Hm, in Panthor that would mean extra checks driver side, because at the moment we don't check where deps come from. I'd be tempted to say, the more we can automate the better, dunno.

Regards,

Boris

[1]https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61/diffs#a5a71f917ff65cfe4c1a341fa7e55ae149d22863_300_693

author maintainer
#40 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 19 lines · Boris Brezillon

Hi Matthew,

Quoted 8 lines · On Wed, 18 Mar 2026 16:28:13 -0700 Matthew Brost <[email protected]>…
Quoted 7 lines

- fence must be signaled for dma_fence::ops to be set back to NULL
- no .cleanup and no .wait implementation

There might be an interest in having HW submission fences reflecting when the job is passed to the FW/HW queue, but that can done as a separate fence implementation using a different fence timeline/context.

Quoted 5 lines

Yes, I removed scheduled side of drm sched fence as I figured that could be implemented driver side (or as an optional API in drm dep). Only AMDGPU / PVR use these too for ganged submissions which I need to wrap my head around. My initial thought is both of implementations likely could be simplified.

Quoted 4 lines

IIRC, PVR was also relying on it to allow native FW waits: when we have a job that has deps that are backed by fences emitted by the same driver, they are detected and lowered to waits on the "scheduled" fence, the wait on the finished fence is done FW side.

Ah, ok. We can build in a scheduling concept if needed, but I’d likely insist it be an opt-in feature. These are the types of driver-side requirements I’d need help with.

Quoted 204 lines
Quoted 204 lines
Quoted 180 lines
Quoted 180 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().  

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);  

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code. I figured that if we’re going to invest the time and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

[1] https://patchwork.freedesktop.org/patch/711933/?series=163245&rev=1#comment_1312648

Quoted 4 lines

just defer the cleanup (AKA the stuff you currently have in _release()) so everything is always cleaned up in a thread context. Yes, there's scheduling overhead and extra latency, but it's also simpler, because there's just one path. So, if the latency and the overhead is not

This isn’t bolted on—it’s a built-in feature throughout. I can assure you that either mode works. I’ll likely add a debug Kconfig option to Xe that toggles DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE on each queue creation for CI runs, to ensure both paths work reliably and receive continuous testing.

Quoted 4 lines

proven to be a problem (and it rarely is for cleanup operations), I'm still convinced this makes for an easier design to just defer the cleanup all the time.

Quoted 8 lines
Quoted 4 lines

If we droppped that, then yes
this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

But even in that context, it could still be deferred and work just fine, no?

A work item context switch is thousands, if not tens of thousands, of cycles. If your job release is only ~20 instructions, this is a massive imbalance and an overall huge waste. Jobs are lightweight objects—they should really be thought of as an extension of fences. Fence release must be IRQ-safe per the documentation, so it follows that jobs can opt in to the same release rules.

In contrast, queues are heavyweight objects, typically with associated memory that also needs to be released. Here, a work item absolutely makes sense—hence the design in DRM dep.

Quoted 6 lines
Quoted 3 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

In Xe, we artificially enforce this rule through our own usage of DRM sched, but in DRM dep this is now an API-level contract. That allows drivers to embrace this semantic directly, simplifying their designs.

Quoted 125 lines
Quoted 125 lines
Quoted 92 lines
Quoted 92 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +91 -0
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+	if (job)
+		kref_get(&job->refcount);
+	return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+	struct drm_dep_job *job =
+		container_of(kref, struct drm_dep_job, refcount);
+	struct drm_dep_queue *q = job->q;
+
+	drm_dep_job_fini(job);
+
+	if (job->ops && job->ops->release)
+		job->ops->release(job);
+	else
+		kfree(job);
+
+	drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+	if (job)
+		kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+	drm_dep_queue_push_job_begin(job->q);
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	drm_dep_fence_init(job->dfence, job->q);
Quoted 9 lines

drivers/gpu/drm/dep/drm_dep_job.c line 269

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	job->signalling_cookie = dma_fence_begin_signalling();  

I'd really like DMA-signalling-path annotation to be something that doesn't leak to the job object. The way I see it, in the submit path, it should be some sort of block initializing an opaque token, and drm_dep_job_arm() should expect a valid token to be passed, thus guaranteeing that anything between arm and push, and more generally anything in that section is safe.

Yes. drm_dep_queue_push_job_begin internally creates a token (current) that is paired drm_dep_queue_push_job_end. If you ever have imbalance between arm() and push() you will get complaints.

Quoted 19 lines
	struct drm_job_submit_context submit_ctx;

	// Do all the prep stuff, pre-alloc, resv setup, ...

	// Non-faillible section of the submit starts here.
	// This is properly annotated with
	// dma_fence_{begin,end}_signalling() to ensure we're
	// not taking locks or doing allocations forbidden in
	// the signalling path
	drm_job_submit_non_faillible_section(&submit_ctx) {
		for_each_job() {
			drm_dep_job_arm(&submit_ctx, &job);

			// pass the armed fence around, if needed

			drm_dep_job_push(&submit_ctx, &job);
		}
	}
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

With the current solution, there's no control that drm_dep_job_{arm,push}() calls are balanced, with the risk of leaving a DMA-signalling annotation behind.

See above, that is what drm_dep_queue_push_job_begin/end do.

That's still error-prone, and the kind of errors you only detect at runtime. Let alone the fact you might not even notice if the unbalanced

I agree that this is only detectable at runtime, but it would complain immediately.

Quoted 1 line

symptoms are caused by error paths that are rarely tested. I'm

Yes, this an unfortunate truth.

Quoted 13 lines

proposing something that's designed so you can't make those mistakes unless you really want to:

- drm_job_submit_non_faillible_section() is a block-like macro with a clear scope before/after which the token is invalid
- drm_job_submit_non_faillible_section() is the only place that can produce a valid token (not enforceable in C, but with an __drm_dep_queue_create_submit_token() and proper disclaimer, I guess we can discourage people to inadvertently use it)
- drm_dep_job_{arm,push}() calls requires a valid token to work, and with the two points mentioned above, that means you can't call drm_dep_job_{arm,push}() outside a drm_job_submit_non_faillible_section() block

Ok, let me think about whether I can harden this semantic. I believe what I have in place already enforces it quite well, but I’m a big believer in using asserts to enforce behavior, and if we can do better, let’s do it.

Quoted 162 lines

It's not quite the compile-time checks rust would enforce, but it's a model that forces people to do it the right way, with extra runtime checks for the case where they still got it wrong (like, putting the _arm() and _push() in two different drm_job_submit_non_faillible_section() blocks).

Quoted 152 lines
Quoted 128 lines
Quoted 128 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 399

Quoted 4 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));  

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Quoted 2 lines

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

Quoted 8 lines

I'm still not entirely convinced by that. To me _arm() is not quite the moment you make your fence public, and I'm not sure the extra complexity added for intra-batch dependencies (one job in a SUBMIT depending on a previous job in the same SUBMIT) is justified, because what really matters is not that we leave dangling/unsignalled dma_fence objects around, the problem is when you do so on an object that has been exposed publicly (syncobj, dma_resv, sync_file, ...).

Let me give you an example of why a failure between arm() and push() is a huge problem:

arm()
dma_resv_install(fence_from_arm)
fail

How does one unwind this? Signal the fence from arm()? What if the fence from arm() is on a timeline currently being used by the device? The memory can move, and the device then can corrupt memory.

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

Quoted 10 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

287 void dma_resv_add_fence(struct dma_resv *obj, struct dma_fence *fence,
288 enum dma_resv_usage usage)
289 {

290         struct dma_resv_list *fobj;
291         struct dma_fence *old;
292         unsigned int i, count;

293
294 dma_fence_get(fence);
295
296 dma_resv_assert_held(obj);
297
298 /* Drivers should not add containers here, instead add each fence
299 * individually.
300 */
301 WARN_ON(dma_fence_is_container(fence));
302
303 fobj = dma_resv_fences_list(obj);
304 count = fobj->num_fences;
305
306 for (i = 0; i < count; ++i) {
307 enum dma_resv_usage old_usage;
308
309 dma_resv_list_entry(fobj, i, obj, &old, &old_usage);
310 if ((old->context == fence->context && old_usage >= usage &&
311 dma_fence_is_later_or_same(fence, old)) ||
312 dma_fence_is_signaled(old)) {

313                         dma_resv_list_set(fobj, i, fence, usage);
314                         dma_fence_put(old);
315                         return;

316 }
317 }

Imagine syncobjs have similar squashing, but I don't know that offhand.

Quoted 121 lines
Quoted 121 lines
Quoted 120 lines
Quoted 105 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +104 -0
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+	struct drm_dep_queue *q = job->q;
+	struct dma_fence *entry;
+	unsigned long index;
+	u32 id = 0;
+	int ret;
+
+	WARN_ON(drm_dep_fence_is_armed(job->dfence));
+	might_alloc(GFP_KERNEL);
+
+	if (!fence)
+		return 0;
+
+	if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+		goto add_fence;
+
+	/*
+	 * Ignore signalled fences or fences from our own queue — finished
+	 * fences use q->fence.context.
+	 */
+	if (dma_fence_test_signaled_flag(fence) ||
+	    fence->context == q->fence.context) {
+		dma_fence_put(fence);
+		return 0;
+	}
+
+	/* Deduplicate if we already depend on a fence from the same context.
+	 * This lets the size of the array of deps scale with the number of
+	 * engines involved, rather than the number of BOs.
+	 */
+	xa_for_each(&job->dependencies, index, entry) {
+		if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+		    entry->context != fence->context)
+			continue;
+
+		if (dma_fence_is_later(fence, entry)) {
+			dma_fence_put(entry);
+			xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+		} else {
+			dma_fence_put(fence);
+		}
+		return 0;
+	}
+
+add_fence:
+	ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+		       GFP_KERNEL);
+	if (ret != 0) {
+		if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+			dma_fence_put(fence);
+		return ret;
+	}
+
+	return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+				    struct dma_fence *fence)
+{
+	WARN_ON(xa_load(&job->dependencies, index) !=
+		DRM_DEP_JOB_FENCE_PREALLOC);
+
+	if (dma_fence_test_signaled_flag(fence)) {

drivers/gpu/drm/dep/drm_dep_job.c line 513

Quoted 10 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +10 -0
+		xa_erase(&job->dependencies, index);
+		dma_fence_put(fence);
+		return;
+	}
+
+	if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
+				       GFP_NOWAIT)))) {
+		dma_fence_put(fence);
+		return;
+	}  

You don't seem to go for the replace-if-earlier-fence-on-same-context-exists optimization that we have in drm_dep_job_add_dependency(). Any reason not to?

Quoted 8 lines
Quoted 3 lines

No, that could be added in. My reasoning for ommiting was if you are pre-alloc a slot you likely know that the same timeline hasn't already been added in but maybe that is bad assumption.

Hm, in Panthor that would mean extra checks driver side, because at the moment we don't check where deps come from. I'd be tempted to say, the more we can automate the better, dunno.

In my example of TLB invalidations this is a non-issue. We can always circle back to squashing here if needed or just do it now. Always open to ideas.

Matt

Quoted 5 lines

Regards,

Boris

[1]https://gitlab.freedesktop.org/panfrost/linux/-/merge_requests/61/diffs#a5a71f917ff65cfe4c1a341fa7e55ae149d22863_300_693

maintainer
#41 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Hi Matthew,

Quoted 215 lines · On Sun, 22 Mar 2026 21:50:07 -0700 Matthew Brost <[email protected]>…
Quoted 202 lines
Quoted 202 lines
Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().    

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);    

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Quoted 8 lines

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code.

It's not just about the number of lines of code added to the core to deal with that case, but also complexity of the API that results from these various modes.

Quoted 5 lines

I figured that if we’re going to invest the time
and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

That's key here. My gut feeling is that we have so much deferred already that adding one more work to the workqueue is not going to hurt in term of scheduling overhead (no context switch if it's scheduled on the same workqueue). Job cleanup is just the phase following the job_done() event, which also requires a deferred work to check progress on the queue anyway. And if you move the entirety of the job cleanup to job_release() instead of doing part of it in drm_dep_job_fini(), it makes for simpler design, where jobs are just cleaned up when their refcnt drops to zero.

IMHO, that's exactly the kind of premature optimization that led us to where we are with drm_sched: we think we need the optimization so we add the complexity upfront without actual numbers to back this theory (like, real GPU workloads to lead to actual differences in term on power consumption, speed, ...), and the complexity just piles up as you keep adding more and more of those flags.

Quoted 11 lines
Quoted 5 lines

just defer the cleanup (AKA the stuff you currently have in _release()) so everything is always cleaned up in a thread context. Yes, there's scheduling overhead and extra latency, but it's also simpler, because there's just one path. So, if the latency and the overhead is not

This isn’t bolted on—it’s a built-in feature throughout. I can assure you that either mode works. I’ll likely add a debug Kconfig option to Xe that toggles DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE on each queue creation for CI runs, to ensure both paths work reliably and receive continuous testing.

I'm not claiming this doesn't work, I just want to make sure we're not taking the same path drm_sched took with those premature optimizations. If you have numbers that prove the extra power-consumption or the fact the extra latency makes a difference in practice, that's a different story, but otherwise, I still think it's preferable to start with a smaller scope/simpler design, and add optimized cleanup path when we have a proof it makes a difference.

Quoted 10 lines
Quoted 10 lines

proven to be a problem (and it rarely is for cleanup operations), I'm still convinced this makes for an easier design to just defer the cleanup all the time.

Quoted 4 lines

If we droppped that, then yes
this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

Quoted 6 lines
Quoted 3 lines

But even in that context, it could still be deferred and work just fine, no?

A work item context switch is thousands, if not tens of thousands, of cycles.

We won't have a full context switch just caused by the cleanup in the normal execution case though. The context switch, you already have it to check progress on the job queue anyway, so adding an extra job cleanup is pretty cheap at this point. I'm not saying free either, there's still the extra work insertion, the dequeuing, etc.

Quoted 9 lines

If your job release is only ~20 instructions, this is a massive imbalance and an overall huge waste. Jobs are lightweight objects—they should really be thought of as an extension of fences. Fence release must be IRQ-safe per the documentation, so it follows that jobs can opt in to the same release rules.

In contrast, queues are heavyweight objects, typically with associated memory that also needs to be released. Here, a work item absolutely makes sense—hence the design in DRM dep.

And that's kinda my point: a job being reported as done will cause a work item to be scheduled to check progress on the queue, so you're already paying the price of a context switch anyway. At this point, all you'll gain by fast-tracking the job cleanup and allowing for IRQ-safe cleanups is just latency. If you have numbers/workloads saying otherwise, I'm fine reconsidering the extra complexity, but I'd to see those first.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Quoted 8 lines
Quoted 2 lines

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

You can do driver-side book-keeping after all jobs have been successfully initialized, which include arming their fences. The key turning point is when you start exposing those armed fences, not when you arm them. See below.

Quoted 5 lines

In Xe, we artificially enforce this rule through our own usage of DRM sched, but in DRM dep this is now an API-level contract. That allows drivers to embrace this semantic directly, simplifying their designs.

[...]

Quoted 164 lines
Quoted 155 lines
Quoted 152 lines
Quoted 128 lines
Quoted 128 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines
Quoted 4 lines

+ * for_each_mmu(mmu) {
+ * // Swap sentinel for bind job's finished fence
+ * drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ * dma_fence_get(bind_job->finished));

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Quoted 2 lines

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

Quoted 8 lines

I'm still not entirely convinced by that. To me _arm() is not quite the moment you make your fence public, and I'm not sure the extra complexity added for intra-batch dependencies (one job in a SUBMIT depending on a previous job in the same SUBMIT) is justified, because what really matters is not that we leave dangling/unsignalled dma_fence objects around, the problem is when you do so on an object that has been exposed publicly (syncobj, dma_resv, sync_file, ...).

Quoted 6 lines

Let me give you an example of why a failure between arm() and push() is a huge problem:

arm()
dma_resv_install(fence_from_arm)
fail

That's not what Panthor does. What we do is:

	for_each_job_in_batch() {
		ret = faillible_stuf()
		if (ret)
			goto rollback;

		arm(job)
	}

	// Nothing can fail after this point

	for_each_job_in_batch() {
		update_resvs(job->done_fence);
		push(job)
	}

	update_submit_syncobjs();

As you can see, an armed job doesn't mean the job fence is public, it only becomes public after we've updated the resv of the BOs that might be touched by this job.

Quoted 2 lines

How does one unwind this? Signal the fence from arm()?

That, or we just ignore the fact it's not been signalled. If the job that created the fence has never been submitted, and the fence has vanished before hitting any public container, it doesn't matter.

Quoted 3 lines

What if the fence
from arm() is on a timeline currently being used by the device? The memory can move, and the device then can corrupt memory.

What? No, the seqno is just consumed, but there's nothing attached to it, the previous job on this timeline (N-1) is still valid, and the next one will have a seqno of N+1, which will force an implicit dep on N-1 on the same timeline. That's all.

Quoted 3 lines

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

I don't think it's safer, it's just the semantics that have been defined by drm_sched/dma_fence and that we keep forcing ourselves into. I'd rather have a well defined dma_fence state that says "that's it, I'm exposed, you have to signal me now", than this half-enforced arm()+push() model.

Quoted 33 lines

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

Quoted 5 lines

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

That's a valid point, but you're not really enforcing things at compile/run-time it's just "don't do this/that" in the docs. If you encode the is_active() state at the dma_fence level, properly change the fence state anytime it's about to be added to a public container, and make it so an active fence that's released without being signalled triggers a WARN_ON(), you've achieved more. Once you've done that, you can also relax the rule that says that "an armed fence has to be signalled" to "a fence that's active has to be signalled". With this, the pre-alloc for intra-batch deps in your drm_dep_job::deps xarray is no longer required, because you would be able to store inactive fences there, as long as they become active before the job is pushed.

Quoted 12 lines
Quoted 11 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Quoted 4 lines

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

That's exactly my point: as soon as you want to insert the fence to a public container, you have to make it "active", so it will never be rolled back to the previous entry in the resv. Similarly, a wait/add_callback() on an inactive fence should be rejected.

Quoted 27 lines

287 void dma_resv_add_fence(struct dma_resv *obj, struct dma_fence *fence,
288 enum dma_resv_usage usage)
289 {

290         struct dma_resv_list *fobj;
291         struct dma_fence *old;
292         unsigned int i, count;

293
294 dma_fence_get(fence);
295
296 dma_resv_assert_held(obj);
297
298 /* Drivers should not add containers here, instead add each fence
299 * individually.
300 */
301 WARN_ON(dma_fence_is_container(fence));
302
303 fobj = dma_resv_fences_list(obj);
304 count = fobj->num_fences;
305
306 for (i = 0; i < count; ++i) {
307 enum dma_resv_usage old_usage;
308
309 dma_resv_list_entry(fobj, i, obj, &old, &old_usage);
310 if ((old->context == fence->context && old_usage >= usage &&
311 dma_fence_is_later_or_same(fence, old)) ||
312 dma_fence_is_signaled(old)) {

Quoted 7 lines
313                         dma_resv_list_set(fobj, i, fence, usage);
314                         dma_fence_put(old);
315                         return;

316 }
317 }

Imagine syncobjs have similar squashing, but I don't know that offhand.

Same goes for syncobjs.

Regards,

Boris

author maintainer
#42 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 226 lines · Boris Brezillon

Hi Matthew,

Quoted 221 lines · On Sun, 22 Mar 2026 21:50:07 -0700 Matthew Brost <[email protected]>…
Quoted 202 lines
Quoted 202 lines
Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().    

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);    

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

Quoted 7 lines
Quoted 2 lines

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code.

It's not just about the number of lines of code added to the core to deal with that case, but also complexity of the API that results from these various modes.

Fair enough.

Quoted 10 lines
Quoted 5 lines

I figured that if we’re going to invest the time
and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

That's key here. My gut feeling is that we have so much deferred already that adding one more work to the workqueue is not going to hurt in term of scheduling overhead (no context switch if it's scheduled on the same workqueue). Job cleanup is just the phase

Signaling of fences in many drivers occurs in hard IRQ context rather than in a work queue. I agree that if you are signaling fences from a work queue, the overhead of another work item is minimal.

Quoted 2 lines

following the job_done() event, which also requires a deferred work to check progress on the queue anyway. And if you move the entirety of the

Yes, I see Panthor signals fences from a work queue by looking at the seqnos, but again, in many drivers this flow is IRQ-driven for fence signaling latency reasons.

Quoted 11 lines

job cleanup to job_release() instead of doing part of it in drm_dep_job_fini(), it makes for simpler design, where jobs are just cleaned up when their refcnt drops to zero.

IMHO, that's exactly the kind of premature optimization that led us to where we are with drm_sched: we think we need the optimization so we add the complexity upfront without actual numbers to back this theory (like, real GPU workloads to lead to actual differences in term on power consumption, speed, ...), and the complexity just piles up as you keep adding more and more of those flags.

Fair enough. Let me measure the CPU utilization to get some data here. It’s not a huge deal to drop this—as I said, it’s a minimal change.

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

just defer the cleanup (AKA the stuff you currently have in _release()) so everything is always cleaned up in a thread context. Yes, there's scheduling overhead and extra latency, but it's also simpler, because there's just one path. So, if the latency and the overhead is not

This isn’t bolted on—it’s a built-in feature throughout. I can assure you that either mode works. I’ll likely add a debug Kconfig option to Xe that toggles DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE on each queue creation for CI runs, to ensure both paths work reliably and receive continuous testing.

Quoted 5 lines

I'm not claiming this doesn't work, I just want to make sure we're not taking the same path drm_sched took with those premature optimizations. If you have numbers that prove the extra power-consumption or the fact the extra latency makes a difference in practice, that's a different story, but otherwise, I still think it's preferable to start with a

+1, will drop this unless I have solid numbers to back this up.

Quoted 17 lines

smaller scope/simpler design, and add optimized cleanup path when we have a proof it makes a difference.

Quoted 14 lines
Quoted 13 lines

proven to be a problem (and it rarely is for cleanup operations), I'm still convinced this makes for an easier design to just defer the cleanup all the time.

Quoted 4 lines

If we droppped that, then yes
this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

But even in that context, it could still be deferred and work just fine, no?

Quoted 9 lines
Quoted 2 lines

A work item context switch is thousands, if not tens of thousands, of cycles.

We won't have a full context switch just caused by the cleanup in the normal execution case though. The context switch, you already have it to check progress on the job queue anyway, so adding an extra job cleanup is pretty cheap at this point. I'm not saying free either, there's still the extra work insertion, the dequeuing, etc.

See above. I believe really changes if a driver signals fences in an IRQ context or a work queue.

Quoted 10 lines
Quoted 9 lines

If your job release is only ~20 instructions, this is a massive imbalance and an overall huge waste. Jobs are lightweight objects—they should really be thought of as an extension of fences. Fence release must be IRQ-safe per the documentation, so it follows that jobs can opt in to the same release rules.

In contrast, queues are heavyweight objects, typically with associated memory that also needs to be released. Here, a work item absolutely makes sense—hence the design in DRM dep.

Quoted 8 lines

And that's kinda my point: a job being reported as done will cause a work item to be scheduled to check progress on the queue, so you're already paying the price of a context switch anyway. At this point, all you'll gain by fast-tracking the job cleanup and allowing for IRQ-safe cleanups is just latency. If you have numbers/workloads saying otherwise, I'm fine reconsidering the extra complexity, but I'd to see those first.

Yep, agree on getting data here.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Quoted 8 lines
Quoted 2 lines

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

Quoted 5 lines

You can do driver-side book-keeping after all jobs have been successfully initialized, which include arming their fences. The key turning point is when you start exposing those armed fences, not when you arm them. See below.

There is still the seqno critical section which starts at arm() and closed at push() or drop of fence.

Quoted 172 lines
Quoted 5 lines

In Xe, we artificially enforce this rule through our own usage of DRM sched, but in DRM dep this is now an API-level contract. That allows drivers to embrace this semantic directly, simplifying their designs.

[...]

Quoted 164 lines
Quoted 163 lines
Quoted 152 lines
Quoted 128 lines
Quoted 128 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines
Quoted 4 lines

+ * for_each_mmu(mmu) {
+ * // Swap sentinel for bind job's finished fence
+ * drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ * dma_fence_get(bind_job->finished));

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Quoted 2 lines

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

I'm still not entirely convinced by that. To me _arm() is not quite the moment you make your fence public, and I'm not sure the extra complexity added for intra-batch dependencies (one job in a SUBMIT depending on a previous job in the same SUBMIT) is justified, because what really matters is not that we leave dangling/unsignalled dma_fence objects around, the problem is when you do so on an object that has been exposed publicly (syncobj, dma_resv, sync_file, ...).

Quoted 9 lines
Quoted 6 lines

Let me give you an example of why a failure between arm() and push() is a huge problem:

arm()
dma_resv_install(fence_from_arm)
fail

That's not what Panthor does. What we do is:

That's good as it would be a bug, just using this as a possible hazard.

Quoted 22 lines
	for_each_job_in_batch() {
		ret = faillible_stuf()
		if (ret)
			goto rollback;

		arm(job)
	}

	// Nothing can fail after this point

	for_each_job_in_batch() {
		update_resvs(job->done_fence);
		push(job)
	}

	update_submit_syncobjs();

As you can see, an armed job doesn't mean the job fence is public, it only becomes public after we've updated the resv of the BOs that might be touched by this job.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

How does one unwind this? Signal the fence from arm()?

That, or we just ignore the fact it's not been signalled. If the job that created the fence has never been submitted, and the fence has vanished before hitting any public container, it doesn't matter.

Ah, this is actually a source of my confusion. I thought the dma-fence API would complain if you made a fence disappear before it was signaled, but it looks like it only complains when the fence is unsignaled and callbacks are attached, i.e. once it has been made public.

Quoted 9 lines
Quoted 3 lines

What if the fence
from arm() is on a timeline currently being used by the device? The memory can move, and the device then can corrupt memory.

What? No, the seqno is just consumed, but there's nothing attached to it, the previous job on this timeline (N-1) is still valid, and the next one will have a seqno of N+1, which will force an implicit dep on N-1 on the same timeline. That's all.

Ok.

Quoted 10 lines
Quoted 3 lines

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

I don't think it's safer, it's just the semantics that have been defined by drm_sched/dma_fence and that we keep forcing ourselves into. I'd rather have a well defined dma_fence state that says "that's it, I'm exposed, you have to signal me now", than this half-enforced arm()+push() model.

So what is the suggestion here — move the asserts I have from arm() to something like begin_push()? We could add a dma-fence state toggle there as well if we can get that part merged into dma-fence. Or should we just drop the asserts/lockdep checks between arm() and push() completely? I’m open to either approach here.

Quoted 39 lines
Quoted 33 lines

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

Quoted 5 lines

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

Quoted 10 lines

That's a valid point, but you're not really enforcing things at compile/run-time it's just "don't do this/that" in the docs. If you encode the is_active() state at the dma_fence level, properly change the fence state anytime it's about to be added to a public container, and make it so an active fence that's released without being signalled triggers a WARN_ON(), you've achieved more. Once you've done that, you can also relax the rule that says that "an armed fence has to be signalled" to "a fence that's active has to be signalled". With this, the pre-alloc for intra-batch deps in your drm_dep_job::deps xarray is no longer required, because you would be able to store inactive fences

I wouldn’t go that far or say it’s that simple. This would require a fairly large refactor of Xe’s VM bind pipeline to call arm() earlier, and I’m not even sure it would be possible. Between arm() and push(), the seqno critical section still remains and requires locking, in particulay the tricky case is kernel binds (e.g., page fault handling) which use the same queue. Multiple threads can issue kernel binds concurrently, as our page fault handler is multi-threaded, similar to the CPU page fault handler, so critical section between arm() and push() is very late in the pipeline tightly protected by a lock.

Quoted 19 lines

there, as long as they become active before the job is pushed.

Quoted 12 lines
Quoted 11 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Quoted 4 lines

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

Quoted 5 lines

That's exactly my point: as soon as you want to insert the fence to a public container, you have to make it "active", so it will never be rolled back to the previous entry in the resv. Similarly, a wait/add_callback() on an inactive fence should be rejected.

This is bit bigger dma-fence / treewide level change but in general I believe this is a good idea.

Matt

Quoted 41 lines
Quoted 27 lines

287 void dma_resv_add_fence(struct dma_resv *obj, struct dma_fence *fence,
288 enum dma_resv_usage usage)
289 {

290         struct dma_resv_list *fobj;
291         struct dma_fence *old;
292         unsigned int i, count;

293
294 dma_fence_get(fence);
295
296 dma_resv_assert_held(obj);
297
298 /* Drivers should not add containers here, instead add each fence
299 * individually.
300 */
301 WARN_ON(dma_fence_is_container(fence));
302
303 fobj = dma_resv_fences_list(obj);
304 count = fobj->num_fences;
305
306 for (i = 0; i < count; ++i) {
307 enum dma_resv_usage old_usage;
308
309 dma_resv_list_entry(fobj, i, obj, &old, &old_usage);
310 if ((old->context == fence->context && old_usage >= usage &&
311 dma_fence_is_later_or_same(fence, old)) ||
312 dma_fence_is_signaled(old)) {

Quoted 7 lines
313                         dma_resv_list_set(fobj, i, fence, usage);
314                         dma_fence_put(old);
315                         return;

316 }
317 }

Imagine syncobjs have similar squashing, but I don't know that offhand.

Same goes for syncobjs.

Regards,

Boris

author maintainer
#43 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 286 lines · Matthew Brost
Quoted 226 lines · Boris Brezillon

Hi Matthew,

Quoted 221 lines · On Sun, 22 Mar 2026 21:50:07 -0700 Matthew Brost <[email protected]>…
Quoted 202 lines
Quoted 202 lines
Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().    

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);    

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

Quoted 7 lines
Quoted 2 lines

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code.

It's not just about the number of lines of code added to the core to deal with that case, but also complexity of the API that results from these various modes.

Fair enough.

Quoted 10 lines
Quoted 5 lines

I figured that if we’re going to invest the time
and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

That's key here. My gut feeling is that we have so much deferred already that adding one more work to the workqueue is not going to hurt in term of scheduling overhead (no context switch if it's scheduled on the same workqueue). Job cleanup is just the phase

Signaling of fences in many drivers occurs in hard IRQ context rather than in a work queue. I agree that if you are signaling fences from a work queue, the overhead of another work item is minimal.

Quoted 2 lines

following the job_done() event, which also requires a deferred work to check progress on the queue anyway. And if you move the entirety of the

Yes, I see Panthor signals fences from a work queue by looking at the seqnos, but again, in many drivers this flow is IRQ-driven for fence signaling latency reasons.

Quoted 11 lines

job cleanup to job_release() instead of doing part of it in drm_dep_job_fini(), it makes for simpler design, where jobs are just cleaned up when their refcnt drops to zero.

IMHO, that's exactly the kind of premature optimization that led us to where we are with drm_sched: we think we need the optimization so we add the complexity upfront without actual numbers to back this theory (like, real GPU workloads to lead to actual differences in term on power consumption, speed, ...), and the complexity just piles up as you keep adding more and more of those flags.

Fair enough. Let me measure the CPU utilization to get some data here. It’s not a huge deal to drop this—as I said, it’s a minimal change.

Quoted 12 lines
Quoted 11 lines
Quoted 5 lines

just defer the cleanup (AKA the stuff you currently have in _release()) so everything is always cleaned up in a thread context. Yes, there's scheduling overhead and extra latency, but it's also simpler, because there's just one path. So, if the latency and the overhead is not

This isn’t bolted on—it’s a built-in feature throughout. I can assure you that either mode works. I’ll likely add a debug Kconfig option to Xe that toggles DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE on each queue creation for CI runs, to ensure both paths work reliably and receive continuous testing.

Quoted 8 lines
Quoted 5 lines

I'm not claiming this doesn't work, I just want to make sure we're not taking the same path drm_sched took with those premature optimizations. If you have numbers that prove the extra power-consumption or the fact the extra latency makes a difference in practice, that's a different story, but otherwise, I still think it's preferable to start with a

+1, will drop this unless I have solid numbers to back this up.

Ok, getting stats is easier than I thought...

./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions /home/mbrost/xe/source/drivers.gpu.i915.igt-gpu-tools/build/tests/xe_exec_threads --r threads-basic

This test creates one thread per engine instance (7 instances this BMG device) and submits 1k exec IOCTLs per thread, each performing a DW write. Each exec IOCTL typically does not have unsignaled input dependencies.

With IRQ putting of jobs off + no bypass (drm_dep_queue_flags = 0):

             8,449      context-switches
               412      cpu-migrations
          2,531.43 msec task-clock
     1,847,846,588      cpu_atom/cycles/
     1,847,856,947      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       460,744,020      cpu_core/instructions/

With IRQ putting of jobs off + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED):

             8,655      context-switches
               229      cpu-migrations
          2,571.33 msec task-clock
       855,900,607      cpu_atom/cycles/
       855,900,272      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       403,651,469      cpu_core/instructions/

With IRQ putting of jobs on + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED | DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

             5,361      context-switches
               169      cpu-migrations
          2,577.44 msec task-clock
       685,769,153      cpu_atom/cycles/
       685,768,407      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       321,336,297      cpu_core/instructions/

Yes, this is a very sythetic test case and results are bit noisy but it seems to point to this entire design to being worthwhile. I'd really like to get customer input (e.g., Google androidOS) here to but is completely out of my control at this point in time.

Matt

Quoted 452 lines
Quoted 17 lines

smaller scope/simpler design, and add optimized cleanup path when we have a proof it makes a difference.

Quoted 14 lines
Quoted 13 lines

proven to be a problem (and it rarely is for cleanup operations), I'm still convinced this makes for an easier design to just defer the cleanup all the time.

Quoted 4 lines

If we droppped that, then yes
this could change. The reason the if statement currently is user is building a job and need to abort prior to calling arm() (e.g., a memory allocation fails) via a drm_dep_job_put.

But even in that context, it could still be deferred and work just fine, no?

Quoted 9 lines
Quoted 2 lines

A work item context switch is thousands, if not tens of thousands, of cycles.

We won't have a full context switch just caused by the cleanup in the normal execution case though. The context switch, you already have it to check progress on the job queue anyway, so adding an extra job cleanup is pretty cheap at this point. I'm not saying free either, there's still the extra work insertion, the dequeuing, etc.

See above. I believe really changes if a driver signals fences in an IRQ context or a work queue.

Quoted 10 lines
Quoted 9 lines

If your job release is only ~20 instructions, this is a massive imbalance and an overall huge waste. Jobs are lightweight objects—they should really be thought of as an extension of fences. Fence release must be IRQ-safe per the documentation, so it follows that jobs can opt in to the same release rules.

In contrast, queues are heavyweight objects, typically with associated memory that also needs to be released. Here, a work item absolutely makes sense—hence the design in DRM dep.

Quoted 8 lines

And that's kinda my point: a job being reported as done will cause a work item to be scheduled to check progress on the queue, so you're already paying the price of a context switch anyway. At this point, all you'll gain by fast-tracking the job cleanup and allowing for IRQ-safe cleanups is just latency. If you have numbers/workloads saying otherwise, I'm fine reconsidering the extra complexity, but I'd to see those first.

Yep, agree on getting data here.

Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Quoted 8 lines
Quoted 2 lines

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

Quoted 5 lines

You can do driver-side book-keeping after all jobs have been successfully initialized, which include arming their fences. The key turning point is when you start exposing those armed fences, not when you arm them. See below.

There is still the seqno critical section which starts at arm() and closed at push() or drop of fence.

Quoted 172 lines
Quoted 5 lines

In Xe, we artificially enforce this rule through our own usage of DRM sched, but in DRM dep this is now an API-level contract. That allows drivers to embrace this semantic directly, simplifying their designs.

[...]

Quoted 164 lines
Quoted 163 lines
Quoted 152 lines
Quoted 128 lines
Quoted 128 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +126 -0
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+	struct drm_dep_queue *q = job->q;
+
+	WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+	drm_dep_job_get(job);
+
+	if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+		drm_dep_queue_push_job(q, job);
+		dma_fence_end_signalling(job->signalling_cookie);
+		drm_dep_queue_push_job_end(job->q);
+		return;
+	}
+
+	scoped_guard(mutex, &q->sched.lock) {
+		if (drm_dep_queue_can_job_bypass(q, job))
+			drm_dep_queue_run_job(q, job);
+		else
+			drm_dep_queue_push_job(q, job);
+	}
+
+	dma_fence_end_signalling(job->signalling_cookie);
+	drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
Quoted 8 lines
Quoted 4 lines

+ * for_each_mmu(mmu) {
+ * // Swap sentinel for bind job's finished fence
+ * drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ * dma_fence_get(bind_job->finished));

Just FYI, Panthor doesn't have this {begin,end}_signalling() in the submit path. If we were to add it, it would be around the panthor_submit_ctx_push_jobs() call, which might seem broken. In

Yes, I noticed that. I put XXX comment in my port [1] around this.

[1] https://patchwork.freedesktop.org/patch/711952/?series=163245&rev=1

Quoted 10 lines

practice I don't think it is because we don't expose fences to the outside world until all jobs have been pushed. So what happens is that a job depending on a previous job in the same batch-submit has the armed-but-not-yet-pushed fence in its deps, and that's the only place where this fence is present. If something fails on a subsequent job preparation in the next batch submit, the rollback logic will just drop the jobs on the floor, and release the armed-but-not-pushed-fence, meaning we're not leaking a fence that will never be signalled. I'm in no way saying this design is sane, just trying to explain why it's currently safe and works fine.

Quoted 2 lines

Yep, I think would be better have no failure points between arm and push which again I do my best to enforce via lockdep/warnings.

I'm still not entirely convinced by that. To me _arm() is not quite the moment you make your fence public, and I'm not sure the extra complexity added for intra-batch dependencies (one job in a SUBMIT depending on a previous job in the same SUBMIT) is justified, because what really matters is not that we leave dangling/unsignalled dma_fence objects around, the problem is when you do so on an object that has been exposed publicly (syncobj, dma_resv, sync_file, ...).

Quoted 9 lines
Quoted 6 lines

Let me give you an example of why a failure between arm() and push() is a huge problem:

arm()
dma_resv_install(fence_from_arm)
fail

That's not what Panthor does. What we do is:

That's good as it would be a bug, just using this as a possible hazard.

Quoted 22 lines
	for_each_job_in_batch() {
		ret = faillible_stuf()
		if (ret)
			goto rollback;

		arm(job)
	}

	// Nothing can fail after this point

	for_each_job_in_batch() {
		update_resvs(job->done_fence);
		push(job)
	}

	update_submit_syncobjs();

As you can see, an armed job doesn't mean the job fence is public, it only becomes public after we've updated the resv of the BOs that might be touched by this job.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

How does one unwind this? Signal the fence from arm()?

That, or we just ignore the fact it's not been signalled. If the job that created the fence has never been submitted, and the fence has vanished before hitting any public container, it doesn't matter.

Ah, this is actually a source of my confusion. I thought the dma-fence API would complain if you made a fence disappear before it was signaled, but it looks like it only complains when the fence is unsignaled and callbacks are attached, i.e. once it has been made public.

Quoted 9 lines
Quoted 3 lines

What if the fence
from arm() is on a timeline currently being used by the device? The memory can move, and the device then can corrupt memory.

What? No, the seqno is just consumed, but there's nothing attached to it, the previous job on this timeline (N-1) is still valid, and the next one will have a seqno of N+1, which will force an implicit dep on N-1 on the same timeline. That's all.

Ok.

Quoted 10 lines
Quoted 3 lines

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

I don't think it's safer, it's just the semantics that have been defined by drm_sched/dma_fence and that we keep forcing ourselves into. I'd rather have a well defined dma_fence state that says "that's it, I'm exposed, you have to signal me now", than this half-enforced arm()+push() model.

So what is the suggestion here — move the asserts I have from arm() to something like begin_push()? We could add a dma-fence state toggle there as well if we can get that part merged into dma-fence. Or should we just drop the asserts/lockdep checks between arm() and push() completely? I’m open to either approach here.

Quoted 39 lines
Quoted 33 lines

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

Quoted 5 lines

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

Quoted 10 lines

That's a valid point, but you're not really enforcing things at compile/run-time it's just "don't do this/that" in the docs. If you encode the is_active() state at the dma_fence level, properly change the fence state anytime it's about to be added to a public container, and make it so an active fence that's released without being signalled triggers a WARN_ON(), you've achieved more. Once you've done that, you can also relax the rule that says that "an armed fence has to be signalled" to "a fence that's active has to be signalled". With this, the pre-alloc for intra-batch deps in your drm_dep_job::deps xarray is no longer required, because you would be able to store inactive fences

I wouldn’t go that far or say it’s that simple. This would require a fairly large refactor of Xe’s VM bind pipeline to call arm() earlier, and I’m not even sure it would be possible. Between arm() and push(), the seqno critical section still remains and requires locking, in particulay the tricky case is kernel binds (e.g., page fault handling) which use the same queue. Multiple threads can issue kernel binds concurrently, as our page fault handler is multi-threaded, similar to the CPU page fault handler, so critical section between arm() and push() is very late in the pipeline tightly protected by a lock.

Quoted 19 lines

there, as long as they become active before the job is pushed.

Quoted 12 lines
Quoted 11 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Quoted 4 lines

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

Quoted 5 lines

That's exactly my point: as soon as you want to insert the fence to a public container, you have to make it "active", so it will never be rolled back to the previous entry in the resv. Similarly, a wait/add_callback() on an inactive fence should be rejected.

This is bit bigger dma-fence / treewide level change but in general I believe this is a good idea.

Matt

Quoted 41 lines
Quoted 27 lines

287 void dma_resv_add_fence(struct dma_resv *obj, struct dma_fence *fence,
288 enum dma_resv_usage usage)
289 {

290         struct dma_resv_list *fobj;
291         struct dma_fence *old;
292         unsigned int i, count;

293
294 dma_fence_get(fence);
295
296 dma_resv_assert_held(obj);
297
298 /* Drivers should not add containers here, instead add each fence
299 * individually.
300 */
301 WARN_ON(dma_fence_is_container(fence));
302
303 fobj = dma_resv_fences_list(obj);
304 count = fobj->num_fences;
305
306 for (i = 0; i < count; ++i) {
307 enum dma_resv_usage old_usage;
308
309 dma_resv_list_entry(fobj, i, obj, &old, &old_usage);
310 if ((old->context == fence->context && old_usage >= usage &&
311 dma_fence_is_later_or_same(fence, old)) ||
312 dma_fence_is_signaled(old)) {

Quoted 7 lines
313                         dma_resv_list_set(fobj, i, fence, usage);
314                         dma_fence_put(old);
315                         return;

316 }
317 }

Imagine syncobjs have similar squashing, but I don't know that offhand.

Same goes for syncobjs.

Regards,

Boris

maintainer
#44 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 34 lines · On Mon, 23 Mar 2026 11:38:06 -0700 Matthew Brost <[email protected]>…

Ok, getting stats is easier than I thought...

./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions /home/mbrost/xe/source/drivers.gpu.i915.igt-gpu-tools/build/tests/xe_exec_threads --r threads-basic

This test creates one thread per engine instance (7 instances this BMG device) and submits 1k exec IOCTLs per thread, each performing a DW write. Each exec IOCTL typically does not have unsignaled input dependencies.

With IRQ putting of jobs off + no bypass (drm_dep_queue_flags = 0):

             8,449      context-switches
               412      cpu-migrations
          2,531.43 msec task-clock
     1,847,846,588      cpu_atom/cycles/
     1,847,856,947      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       460,744,020      cpu_core/instructions/

With IRQ putting of jobs off + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED):

             8,655      context-switches
               229      cpu-migrations
          2,571.33 msec task-clock
       855,900,607      cpu_atom/cycles/
       855,900,272      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       403,651,469      cpu_core/instructions/

With IRQ putting of jobs on + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED | DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

Quoted 7 lines
             5,361      context-switches
               169      cpu-migrations
          2,577.44 msec task-clock
       685,769,153      cpu_atom/cycles/
       685,768,407      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       321,336,297      cpu_core/instructions/

Thanks for sharing those numbers. For completeness, can you also add the "With IRQ putting of jobs on + no bypass" case?

I'm a bit surprised by the difference in number of context switches given I'd expect the local-CPU to be picked in priority, and so queuing work items on the same wq from another work item to be almost free in term on scheduling. But I guess there's some load-balancing happening when you execute jobs at such a high rate.

Also, I don't know if that's just noise or if it's reproducible, but task-clock seems to be ~40usec lower with the deferred cleanup and no-bypass (higher throughput because you're not blocking the dequeuing of the next job on the cleanup of the previous one, I suspect).

author maintainer
#45 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 45 lines · Boris Brezillon
Quoted 34 lines · On Mon, 23 Mar 2026 11:38:06 -0700 Matthew Brost <[email protected]>…

Ok, getting stats is easier than I thought...

./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions /home/mbrost/xe/source/drivers.gpu.i915.igt-gpu-tools/build/tests/xe_exec_threads --r threads-basic

This test creates one thread per engine instance (7 instances this BMG device) and submits 1k exec IOCTLs per thread, each performing a DW write. Each exec IOCTL typically does not have unsignaled input dependencies.

With IRQ putting of jobs off + no bypass (drm_dep_queue_flags = 0):

             8,449      context-switches
               412      cpu-migrations
          2,531.43 msec task-clock
     1,847,846,588      cpu_atom/cycles/
     1,847,856,947      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       460,744,020      cpu_core/instructions/

With IRQ putting of jobs off + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED):

             8,655      context-switches
               229      cpu-migrations
          2,571.33 msec task-clock
       855,900,607      cpu_atom/cycles/
       855,900,272      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       403,651,469      cpu_core/instructions/

With IRQ putting of jobs on + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED | DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

Quoted 7 lines
             5,361      context-switches
               169      cpu-migrations
          2,577.44 msec task-clock
       685,769,153      cpu_atom/cycles/
       685,768,407      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       321,336,297      cpu_core/instructions/
Quoted 3 lines

Thanks for sharing those numbers. For completeness, can you also add the "With IRQ putting of jobs on + no bypass" case?

Yes, I also will share a DRM sched baseline too + I figured out power can be measured too - initial results confirm what I expected too - less power.

I'm putting together a doc based on running glxgears and another benchmark on top Ubuntu 24.10 + Wayland which has explicit sync (linux-drm-syncobj, behaves like surfface flinger when rendering flag to not pass in fences to draw jobs).

Almost have all the data. Will share here once I have it.

Quoted 10 lines

I'm a bit surprised by the difference in number of context switches given I'd expect the local-CPU to be picked in priority, and so queuing work items on the same wq from another work item to be almost free in term on scheduling. But I guess there's some load-balancing happening when you execute jobs at such a high rate.

Also, I don't know if that's just noise or if it's reproducible, but task-clock seems to be ~40usec lower with the deferred cleanup and no-bypass (higher throughput because you're not blocking the dequeuing of the next job on the cleanup of the previous one, I suspect).

I think that is just noise of what the test is doing in user space - that bounces around a bit.

Matt

Quoted 1 line
author maintainer
#46 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 54 lines · Matthew Brost
Quoted 45 lines · Boris Brezillon
Quoted 34 lines · On Mon, 23 Mar 2026 11:38:06 -0700 Matthew Brost <[email protected]>…

Ok, getting stats is easier than I thought...

./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions /home/mbrost/xe/source/drivers.gpu.i915.igt-gpu-tools/build/tests/xe_exec_threads --r threads-basic

This test creates one thread per engine instance (7 instances this BMG device) and submits 1k exec IOCTLs per thread, each performing a DW write. Each exec IOCTL typically does not have unsignaled input dependencies.

With IRQ putting of jobs off + no bypass (drm_dep_queue_flags = 0):

             8,449      context-switches
               412      cpu-migrations
          2,531.43 msec task-clock
     1,847,846,588      cpu_atom/cycles/
     1,847,856,947      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       460,744,020      cpu_core/instructions/

With IRQ putting of jobs off + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED):

             8,655      context-switches
               229      cpu-migrations
          2,571.33 msec task-clock
       855,900,607      cpu_atom/cycles/
       855,900,272      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       403,651,469      cpu_core/instructions/

With IRQ putting of jobs on + bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED | DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

Quoted 7 lines
             5,361      context-switches
               169      cpu-migrations
          2,577.44 msec task-clock
       685,769,153      cpu_atom/cycles/
       685,768,407      cpu_core/cycles/
   <not supported>      cpu_atom/instructions/
       321,336,297      cpu_core/instructions/
Quoted 3 lines

Thanks for sharing those numbers. For completeness, can you also add the "With IRQ putting of jobs on + no bypass" case?

Yes, I also will share a DRM sched baseline too + I figured out power can be measured too - initial results confirm what I expected too - less power.

Quoted 7 lines

I'm putting together a doc based on running glxgears and another benchmark on top Ubuntu 24.10 + Wayland which has explicit sync (linux-drm-syncobj, behaves like surfface flinger when rendering flag to not pass in fences to draw jobs).

Almost have all the data. Will share here once I have it.

Here are some numbers based on glxgears and weston-simple-egl.

5 configurations tested:
DRM sched
DRM dep (no opt flags)
DRM dep + bypass flag
DRM dep + IRQ-safe flag
DRM dep + bypass + IRQ-safe flags

Each configuration was run 3× on both glxgears and weston-simple-egl. Raptor lake CPU, BMG G21.

Summary:
DRM dep reduces power usage, CPU cycles, and context switches. Enabling both the bypass and IRQ-safe flags further reduces all of these metrics.

I’d say this test case best models something like scrolling on a phone or using a laptop for non-GPU-intensive workloads where the screen still needs to refresh.

I’ve run more intensive benchmarks—glmark2 and Unigine Heaven as well. The results are somewhat noisy between boots, but I think the same conclusion holds.

Raw numbers (bit of a firehouse):

DRM sched:

root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
303 frames in 5.0 seconds = 60.565 FPS
300 frames in 5.0 seconds = 60.000 FPS
301 frames in 5.0 seconds = 60.001 FPS

Performance counter stats for 'system wide':

            71,548        context-switches
             1,466        cpu-migrations
        320,440.96 msec   task-clock
     9,140,249,815        cpu_atom/cycles/
     9,140,253,058        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     7,071,794,806        cpu_core/instructions/
            168.76 Joules power/energy-pkg/
             57.78 Joules power/energy-cores/

      20.029126614 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.642 FPS
300 frames in 5.0 seconds = 59.988 FPS
301 frames in 5.0 seconds = 60.001 FPS

Performance counter stats for 'system wide':

            71,720        context-switches
             1,581        cpu-migrations
        320,530.64 msec   task-clock
     8,990,313,521        cpu_atom/cycles/
     8,990,315,400        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,988,827,285        cpu_core/instructions/
            172.15 Joules power/energy-pkg/
             58.33 Joules power/energy-cores/

      20.034862844 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.741 FPS
299 frames in 5.0 seconds = 59.798 FPS
299 frames in 5.0 seconds = 59.799 FPS

Performance counter stats for 'system wide':

            70,871        context-switches
             1,980        cpu-migrations
        320,558.82 msec   task-clock
     8,861,481,467        cpu_atom/cycles/
     8,861,485,448        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,665,294,516        cpu_core/instructions/
            167.82 Joules power/energy-pkg/
             56.97 Joules power/energy-cores/

      20.035713155 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            27,398        context-switches
               678        cpu-migrations
        160,255.17 msec   task-clock
     5,002,546,782        cpu_atom/cycles/
     5,002,549,920        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,498,672,077        cpu_core/instructions/
             93.41 Joules power/energy-pkg/
             23.91 Joules power/energy-cores/

      10.017552274 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            27,322        context-switches
               580        cpu-migrations
        160,307.12 msec   task-clock
     4,783,734,059        cpu_atom/cycles/
     4,783,737,645        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,224,510,206        cpu_core/instructions/
             91.89 Joules power/energy-pkg/
             23.28 Joules power/energy-cores/

      10.020629190 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            27,356        context-switches
               573        cpu-migrations
        160,362.30 msec   task-clock
     5,112,653,847        cpu_atom/cycles/
     5,112,658,503        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,395,873,668        cpu_core/instructions/
             94.40 Joules power/energy-pkg/
             24.58 Joules power/energy-cores/

      10.023979647 seconds time elapsed

No opt (drm_dep_queue_flags = 0):

root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
303 frames in 5.0 seconds = 60.597 FPS
300 frames in 5.0 seconds = 59.989 FPS
297 frames in 5.0 seconds = 59.232 FPS

Performance counter stats for 'system wide':

            66,233        context-switches
             1,820        cpu-migrations
        320,586.39 msec   task-clock
     9,028,164,726        cpu_atom/cycles/
     9,028,178,052        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,541,478,243        cpu_core/instructions/
            178.47 Joules power/energy-pkg/
             44.18 Joules power/energy-cores/

      20.036849235 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.691 FPS
297 frames in 5.0 seconds = 59.393 FPS
300 frames in 5.0 seconds = 59.803 FPS

Performance counter stats for 'system wide':

            68,389        context-switches
             2,034        cpu-migrations
        320,457.18 msec   task-clock
     8,736,092,056        cpu_atom/cycles/
     8,736,096,958        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,511,630,145        cpu_core/instructions/
            183.23 Joules power/energy-pkg/
             47.43 Joules power/energy-cores/

      20.031469459 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
303 frames in 5.0 seconds = 60.458 FPS
299 frames in 5.0 seconds = 59.606 FPS
298 frames in 5.0 seconds = 59.590 FPS

Performance counter stats for 'system wide':

            67,692        context-switches
             1,877        cpu-migrations
        320,524.05 msec   task-clock
     8,837,946,224        cpu_atom/cycles/
     8,837,949,628        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,018,812,170        cpu_core/instructions/
            187.63 Joules power/energy-pkg/
             46.76 Joules power/energy-cores/

      20.034428856 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            27,259        context-switches
               313        cpu-migrations
        160,538.29 msec   task-clock
     5,079,653,975        cpu_atom/cycles/
     5,079,657,432        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,166,877,411        cpu_core/instructions/
             90.72 Joules power/energy-pkg/
             21.70 Joules power/energy-cores/

      10.034716719 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            26,933        context-switches
               449        cpu-migrations
        160,334.74 msec   task-clock
     4,851,027,105        cpu_atom/cycles/
     4,851,054,678        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,042,177,215        cpu_core/instructions/
             87.33 Joules power/energy-pkg/
             21.85 Joules power/energy-cores/

      10.021873082 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            27,101        context-switches
               351        cpu-migrations
        160,333.98 msec   task-clock
     4,903,047,240        cpu_atom/cycles/
     4,903,055,111        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     2,884,284,727        cpu_core/instructions/
             87.68 Joules power/energy-pkg/
             21.36 Joules power/energy-cores/

      10.021938190 seconds time elapsed

Bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED):

root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.718 FPS
299 frames in 5.0 seconds = 59.615 FPS
299 frames in 5.0 seconds = 59.795 FPS

Performance counter stats for 'system wide':

            56,788        context-switches
             2,576        cpu-migrations
        320,610.02 msec   task-clock
     9,056,383,522        cpu_atom/cycles/
     9,056,385,629        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,285,652,796        cpu_core/instructions/
            164.29 Joules power/energy-pkg/
             44.70 Joules power/energy-cores/

      20.041318795 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.734 FPS
300 frames in 5.0 seconds = 59.983 FPS
300 frames in 5.0 seconds = 60.000 FPS

Performance counter stats for 'system wide':

            56,388        context-switches
             2,326        cpu-migrations
        320,581.07 msec   task-clock
     8,789,215,827        cpu_atom/cycles/
     8,789,217,484        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,251,346,200        cpu_core/instructions/
            162.67 Joules power/energy-pkg/
             44.30 Joules power/energy-cores/

      20.037648324 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
305 frames in 5.0 seconds = 60.950 FPS
300 frames in 5.0 seconds = 59.993 FPS
300 frames in 5.0 seconds = 59.806 FPS

Performance counter stats for 'system wide':

            56,167        context-switches
             2,434        cpu-migrations
        320,594.69 msec   task-clock
     8,700,873,664        cpu_atom/cycles/
     8,700,877,150        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,405,556,662        cpu_core/instructions/
            162.55 Joules power/energy-pkg/
             43.33 Joules power/energy-cores/

      20.038448851 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            24,747        context-switches
             1,254        cpu-migrations
        160,543.42 msec   task-clock
     5,047,832,024        cpu_atom/cycles/
     5,047,823,996        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,124,591,155        cpu_core/instructions/
             80.28 Joules power/energy-pkg/
             21.49 Joules power/energy-cores/

      10.034654628 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            24,953        context-switches
               921        cpu-migrations
        160,375.32 msec   task-clock
     5,197,283,835        cpu_atom/cycles/
     5,197,287,623        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,393,363,950        cpu_core/instructions/
             83.36 Joules power/energy-pkg/
             21.92 Joules power/energy-cores/

      10.024899366 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
298 frames in 5 seconds: 59.599998 fps

Performance counter stats for 'system wide':

            24,576        context-switches
               966        cpu-migrations
        160,339.37 msec   task-clock
     4,915,705,971        cpu_atom/cycles/
     4,915,709,503        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     2,968,947,722        cpu_core/instructions/
             79.96 Joules power/energy-pkg/
             21.08 Joules power/energy-cores/

      10.022743041 seconds time elapsed

IRQ (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.643 FPS
298 frames in 5.0 seconds = 59.599 FPS
295 frames in 5.0 seconds = 58.998 FPS

Performance counter stats for 'system wide':

            60,305        context-switches
             1,994        cpu-migrations
        320,528.79 msec   task-clock
     8,518,549,937        cpu_atom/cycles/
     8,518,573,906        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     5,813,890,066        cpu_core/instructions/
            184.52 Joules power/energy-pkg/
             40.79 Joules power/energy-cores/

      20.032795872 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.759 FPS
299 frames in 5.0 seconds = 59.790 FPS
301 frames in 5.0 seconds = 60.003 FPS

Performance counter stats for 'system wide':

            59,401        context-switches
             2,256        cpu-migrations
        320,475.03 msec   task-clock
     8,581,759,828        cpu_atom/cycles/
     8,581,763,986        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,748,269,548        cpu_core/instructions/
            179.76 Joules power/energy-pkg/
             40.66 Joules power/energy-cores/

      20.029861532 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.653 FPS
298 frames in 5.0 seconds = 59.404 FPS
300 frames in 5.0 seconds = 59.990 FPS

Performance counter stats for 'system wide':

            59,381        context-switches
             1,800        cpu-migrations
        320,616.35 msec   task-clock
     8,829,473,025        cpu_atom/cycles/
     8,829,477,019        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,505,926,710        cpu_core/instructions/
            180.38 Joules power/energy-pkg/
             40.86 Joules power/energy-cores/

      20.040016190 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
298 frames in 5 seconds: 59.599998 fps

Performance counter stats for 'system wide':

            27,341        context-switches
               786        cpu-migrations
        160,478.01 msec   task-clock
     4,681,440,843        cpu_atom/cycles/
     4,681,443,905        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     2,969,039,615        cpu_core/instructions/
             91.74 Joules power/energy-pkg/
             20.84 Joules power/energy-cores/

      10.031116623 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            24,626        context-switches
               429        cpu-migrations
        160,367.44 msec   task-clock
     4,828,015,355        cpu_atom/cycles/
     4,828,019,887        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     2,675,419,833        cpu_core/instructions/
             90.35 Joules power/energy-pkg/
             21.10 Joules power/energy-cores/

      10.024476921 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            24,679        context-switches
               340        cpu-migrations
        160,303.90 msec   task-clock
     4,500,129,961        cpu_atom/cycles/
     4,500,132,697        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     2,766,150,592        cpu_core/instructions/
             88.01 Joules power/energy-pkg/
             19.76 Joules power/energy-cores/

      10.019653353 seconds time elapsed

IRQ plus bypass (drm_dep_queue_flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED | DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE):

root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
305 frames in 5.0 seconds = 60.958 FPS
299 frames in 5.0 seconds = 59.607 FPS
299 frames in 5.0 seconds = 59.603 FPS

Performance counter stats for 'system wide':

            46,934        context-switches
             1,558        cpu-migrations
        320,569.83 msec   task-clock
     7,976,414,449        cpu_atom/cycles/
     7,976,417,934        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,126,973,947        cpu_core/instructions/
            178.36 Joules power/energy-pkg/
             40.10 Joules power/energy-cores/

      20.037681420 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
304 frames in 5.0 seconds = 60.696 FPS
299 frames in 5.0 seconds = 59.616 FPS
299 frames in 5.0 seconds = 59.781 FPS

Performance counter stats for 'system wide':

            47,691        context-switches
             1,994        cpu-migrations
        320,602.83 msec   task-clock
     8,270,567,663        cpu_atom/cycles/
     8,270,572,484        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     4,361,204,861        cpu_core/instructions/
            181.56 Joules power/energy-pkg/
             40.16 Joules power/energy-cores/

      20.038511163 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 20s glxgears
Running synchronized to the vertical refresh.  The framerate should be
approximately the same as the monitor refresh rate.
305 frames in 5.0 seconds = 60.911 FPS
298 frames in 5.0 seconds = 59.597 FPS
300 frames in 5.0 seconds = 59.803 FPS

Performance counter stats for 'system wide':

            47,129        context-switches
             1,921        cpu-migrations
        320,491.09 msec   task-clock
     8,054,513,204        cpu_atom/cycles/
     8,054,518,711        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     6,131,796,639        cpu_core/instructions/
            178.54 Joules power/energy-pkg/
             40.08 Joules power/energy-cores/

      20.032444923 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            21,991        context-switches
               286        cpu-migrations
        160,343.73 msec   task-clock
     4,497,475,288        cpu_atom/cycles/
     4,497,477,011        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,042,007,163        cpu_core/instructions/
             89.14 Joules power/energy-pkg/
             20.09 Joules power/energy-cores/

      10.021642254 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
300 frames in 5 seconds: 60.000000 fps

Performance counter stats for 'system wide':

            22,366        context-switches
               225        cpu-migrations
        160,386.68 msec   task-clock
     4,398,432,348        cpu_atom/cycles/
     4,398,435,205        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,086,156,274        cpu_core/instructions/
             89.07 Joules power/energy-pkg/
             19.68 Joules power/energy-cores/

      10.024827902 seconds time elapsed
root@DUT6235BMGFRD:mbrost# ./perf stat -a -e context-switches,cpu-migrations,task-clock,cycles,instructions,power/energy-pkg/,power/energy-cores/ timeout 10 weston-simple-egl -f
Using config: r8g8b8a8
has EGL_EXT_buffer_age and EGL_EXT_swap_buffers_with_damage
has EGL_EXT_surface_compression
299 frames in 5 seconds: 59.799999 fps

Performance counter stats for 'system wide':

            22,515        context-switches
               286        cpu-migrations
        160,481.91 msec   task-clock
     4,447,740,222        cpu_atom/cycles/
     4,447,743,314        cpu_core/cycles/
   <not supported>        cpu_atom/instructions/
     3,217,285,071        cpu_core/instructions/
             90.15 Joules power/energy-pkg/
             19.65 Joules power/energy-cores/

      10.029135743 seconds time elapsed

Matt

Quoted 17 lines
Quoted 10 lines

I'm a bit surprised by the difference in number of context switches given I'd expect the local-CPU to be picked in priority, and so queuing work items on the same wq from another work item to be almost free in term on scheduling. But I guess there's some load-balancing happening when you execute jobs at such a high rate.

Also, I don't know if that's just noise or if it's reproducible, but task-clock seems to be ~40usec lower with the deferred cleanup and no-bypass (higher throughput because you're not blocking the dequeuing of the next job on the cleanup of the previous one, I suspect).

I think that is just noise of what the test is doing in user space - that bounces around a bit.

Matt

Quoted 1 line
maintainer
#47 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

In reply to Matthew Brost
Quoted 243 lines · On Mon, 23 Mar 2026 10:08:53 -0700 Matthew Brost <[email protected]>…
Quoted 226 lines · Boris Brezillon

Hi Matthew,

Quoted 221 lines · On Sun, 22 Mar 2026 21:50:07 -0700 Matthew Brost <[email protected]>…
Quoted 202 lines
Quoted 202 lines
Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().      

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);      

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

Quoted 7 lines
Quoted 2 lines

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code.

It's not just about the number of lines of code added to the core to deal with that case, but also complexity of the API that results from these various modes.

Fair enough.

Quoted 6 lines
Quoted 5 lines

I figured that if we’re going to invest the time
and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

Quoted 8 lines
Quoted 4 lines

That's key here. My gut feeling is that we have so much deferred already that adding one more work to the workqueue is not going to hurt in term of scheduling overhead (no context switch if it's scheduled on the same workqueue). Job cleanup is just the phase

Signaling of fences in many drivers occurs in hard IRQ context rather than in a work queue. I agree that if you are signaling fences from a work queue, the overhead of another work item is minimal.

I'm talking about the drm_dep_queue_run_job_queue() call which in turn calls queue_work() when a job gets reported as done. That, I think, is the most likely path, isn't it?

Quoted 7 lines
Quoted 3 lines

following the job_done() event, which also requires a deferred work to check progress on the queue anyway. And if you move the entirety of the

Yes, I see Panthor signals fences from a work queue by looking at the seqnos, but again, in many drivers this flow is IRQ-driven for fence signaling latency reasons.

I'm not talking about the signalling of the done_fence, but the work that's used to check progress on a job queue (drm_dep_queue_run_job_queue()).

Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 7 lines
Quoted 4 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

Quoted 8 lines
Quoted 5 lines

You can do driver-side book-keeping after all jobs have been successfully initialized, which include arming their fences. The key turning point is when you start exposing those armed fences, not when you arm them. See below.

There is still the seqno critical section which starts at arm() and closed at push() or drop of fence.

That's orthogonal to the rule that says nothing after _arm() can fail, I think. To guarantee proper job ordering, you need extra locking (at the moment, we rely on the VM resv lock to serialize this in Panthor).

Quoted 11 lines
Quoted 10 lines
Quoted 3 lines

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

I don't think it's safer, it's just the semantics that have been defined by drm_sched/dma_fence and that we keep forcing ourselves into. I'd rather have a well defined dma_fence state that says "that's it, I'm exposed, you have to signal me now", than this half-enforced arm()+push() model.

Quoted 5 lines

So what is the suggestion here — move the asserts I have from arm() to something like begin_push()? We could add a dma-fence state toggle there as well if we can get that part merged into dma-fence. Or should we just drop the asserts/lockdep checks between arm() and push() completely? I’m open to either approach here.

If we can have that INACTIVE flag added, and the associated dma_fence_init[64]_inactive() variants, I would say, we call dma_fence_init[64]_inactive() in _arm(), and we call dma_fence_set_active() in _push(). It'd still be valuable to have some sort of delimitation for the submission through some block-like macro with an associated context to which we can attach states and allow for more (optional?) runtime-checks.

Quoted 51 lines
Quoted 40 lines
Quoted 34 lines

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

Quoted 5 lines

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

Quoted 10 lines

That's a valid point, but you're not really enforcing things at compile/run-time it's just "don't do this/that" in the docs. If you encode the is_active() state at the dma_fence level, properly change the fence state anytime it's about to be added to a public container, and make it so an active fence that's released without being signalled triggers a WARN_ON(), you've achieved more. Once you've done that, you can also relax the rule that says that "an armed fence has to be signalled" to "a fence that's active has to be signalled". With this, the pre-alloc for intra-batch deps in your drm_dep_job::deps xarray is no longer required, because you would be able to store inactive fences

Quoted 9 lines

I wouldn’t go that far or say it’s that simple. This would require a fairly large refactor of Xe’s VM bind pipeline to call arm() earlier, and I’m not even sure it would be possible. Between arm() and push(), the seqno critical section still remains and requires locking, in particulay the tricky case is kernel binds (e.g., page fault handling) which use the same queue. Multiple threads can issue kernel binds concurrently, as our page fault handler is multi-threaded, similar to the CPU page fault handler, so critical section between arm() and push() is very late in the pipeline tightly protected by a lock.

This sounds like a different issue to me. That's the constraint that says _arm() and _push() ordering needs to be preserved to guarantee that jobs are properly ordered on the job queue. But that's orthogonal to the rule that says nothing between _arm() and _push() on a given job can fail. Let's take the Panthor case as an example:

	for_each_job_in_batch() {
		// This acquires the VM resv lock, and all BO locks
		// Because queues target a specific VM and all jobs
		// in the a SUBMIT must target the same VM, this
		// guarantees that seqno allocation happening further
		// down (when _arm() is called) won't be interleaved
		// with other concurrent submissions to the same queues.
		lock_and_prepare_resvs()

		<--- Seqno critical section starts here
	}
 	for_each_job_in_batch() {
		// If something fails here, we drop all the jobs that
		// are part of this SUBMIT, and the resv locks are
		// released as part of the rollback. This means we
		// consumed but didn't use the seqnos, thus creating
		// a whole on the timeline, which is armless, as long
		// as those seqnos are not recycled.
		ret = faillible_stuf()

if (ret)
goto rollback;

arm(job)
}

// Nothing can fail after this point

 	for_each_job_in_batch() {
		// resv locks are released here, unblocking other
		// concurrent submissions

update_resvs(job->done_fence)

<--- Seqno critical section ends here in case of success

push(job)
}

	update_submit_syncobjs();

rollback:
unlock_resvs()
<--- Seqno critical section ends here in case of success
...

How wide your critical seqno section is is up to each driver, really.

Quoted 20 lines
Quoted 20 lines

there, as long as they become active before the job is pushed.

Quoted 16 lines
Quoted 11 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

Quoted 8 lines
Quoted 5 lines

That's exactly my point: as soon as you want to insert the fence to a public container, you have to make it "active", so it will never be rolled back to the previous entry in the resv. Similarly, a wait/add_callback() on an inactive fence should be rejected.

This is bit bigger dma-fence / treewide level change but in general I believe this is a good idea.

I agree it's a bit more work. It implies patching containers to reject insertion when the INACTIVE flag is set. If we keep !INACTIVE as the default (__dma_fence_init(INACTIVE) being an opt-in), fence emitters can be moved to this model progressively though.

author maintainer
#48 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

Quoted 251 lines · Boris Brezillon
Quoted 248 lines · On Mon, 23 Mar 2026 10:08:53 -0700 Matthew Brost <[email protected]>…
Quoted 226 lines · Boris Brezillon

Hi Matthew,

Quoted 221 lines · On Sun, 22 Mar 2026 21:50:07 -0700 Matthew Brost <[email protected]>…
Quoted 202 lines
Quoted 202 lines
Quoted 178 lines
Quoted 178 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +172 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+		     const struct drm_dep_job_init_args *args)
+{
+	if (unlikely(!args->credits)) {
+		pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+		return -EINVAL;
+	}
+
+	memset(job, 0, sizeof(*job));
+
+	job->dfence = drm_dep_fence_alloc();
+	if (!job->dfence)
+		return -ENOMEM;
+
+	job->ops = args->ops;
+	job->q = drm_dep_queue_get(args->q);
+	job->credits = args->credits;
+
+	kref_init(&job->refcount);
+	xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+	INIT_LIST_HEAD(&job->pending_link);
+
+	return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+	struct dma_fence *fence;
+	unsigned long index;
+
+	xa_for_each(&job->dependencies, index, fence) {
+		if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+			continue;
+		dma_fence_put(fence);
+	}
+	xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+	bool armed = drm_dep_fence_is_armed(job->dfence);
+
+	WARN_ON(!list_empty(&job->pending_link));
+
+	drm_dep_fence_cleanup(job->dfence);
+	job->dfence = NULL;
+
+	/*
+	 * Armed jobs have their dependencies drained by
Quoted 8 lines

drivers/gpu/drm/dep/drm_dep_job.c line 173

Quoted 1 line
drivers/gpu/drm/dep/drm_dep_job.c new file +1 -0
+	 * drm_dep_job_drop_dependencies() in process context after run_job().      

Just want to clear the confusion and make sure I get this right at the same time. To me, "process context" means a user thread entering some syscall(). What you call "process context" is more a "thread context" to me. I'm actually almost certain it's always a kernel thread (a workqueue worker thread to be accurate) that executes the drop_deps() after a run_job().

Some of context comments likely could be cleaned up. 'process context' here either in user context (bypass path) or run job work item.

Quoted 11 lines

drivers/gpu/drm/dep/drm_dep_job.c line 177

Quoted 5 lines
drivers/gpu/drm/dep/drm_dep_job.c new file +4 -0
+	 * Skip here to avoid calling xa_destroy() from IRQ context.
+	 */
+	if (!armed)
+		drm_dep_job_drop_dependencies(job);      

Why do we need to make a difference here. Can't we just assume that the hole drm_dep_job_fini() call is unsafe in atomic context, and have a work item embedded in the job to defer its destruction when _put() is called in a context where the destruction is not allowed?

Quoted 7 lines
Quoted 2 lines

We already touched on this, but the design currently allows the last job put from dma-fence signaling path (IRQ).

It's not much about the last _put and more about what happens in the _release() you pass to kref_put(). My point being, if you assume something in _release() is not safe to be done in an atomic context, and _put() is assumed to be called from any context, you might as well

No. DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE indicates that the entire job put (including release) is IRQ-safe. If the documentation isn’t clear, I can clean that up. Some of my comments here [1] try to explain this further.

Setting DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE makes a job analogous to a dma-fence whose release must be IRQ-safe, so there is precedent for this. I didn’t want to unilaterally require that all job releases be IRQ-safe, as that would conflict with existing DRM scheduler jobs—hence the flag.

Quoted 7 lines
Quoted 2 lines

The difference between non-IRQ-safe and IRQ-safe job release is only about 12 lines of code.

It's not just about the number of lines of code added to the core to deal with that case, but also complexity of the API that results from these various modes.

Fair enough.

Quoted 10 lines
Quoted 5 lines

I figured that if we’re going to invest the time
and effort to replace DRM sched, we should aim for the best possible implementation. Any driver can opt-in here and immediately get less CPU ultization and power savings. I will try to figure out how to measure this and get some number here.

That's key here. My gut feeling is that we have so much deferred already that adding one more work to the workqueue is not going to hurt in term of scheduling overhead (no context switch if it's scheduled on the same workqueue). Job cleanup is just the phase

Quoted 8 lines
Quoted 3 lines

Signaling of fences in many drivers occurs in hard IRQ context rather than in a work queue. I agree that if you are signaling fences from a work queue, the overhead of another work item is minimal.

I'm talking about the drm_dep_queue_run_job_queue() call which in turn calls queue_work() when a job gets reported as done. That, I think, is the most likely path, isn't it?

Oh, I think I understand what you are getting at...

drm_dep_queue_run_job_queue() is called less blindly in DRM dep compared to DRM sched. When a fence signals, drm_dep_queue_run_job_queue() only kicks a worker if the SPSC queue is non-empty and the signal may have freed up the credits needed to call ->run_job(). This could likely be optimized further to skip the work item if the run_job path is blocked on a dependency, though I’d need to think through the races.

With the way credits work in Xe (we have many per queue), it is quite likely this won’t be triggered. Therefore, the extra worker kick to put the job incurs an otherwise unnecessary context switch. If drm_dep_queue_run_job_queue() always kicked a work item, the additional work item would indeed be less costly.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

following the job_done() event, which also requires a deferred work to check progress on the queue anyway. And if you move the entirety of the

Quoted 8 lines
Quoted 3 lines

Yes, I see Panthor signals fences from a work queue by looking at the seqnos, but again, in many drivers this flow is IRQ-driven for fence signaling latency reasons.

I'm not talking about the signalling of the done_fence, but the work that's used to check progress on a job queue (drm_dep_queue_run_job_queue()).

See above.

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Once arm() is called there is a guarnette the run_job path is called either via bypass or run job work item.

Quoted 8 lines
Quoted 2 lines

Sure.

Let’s not gloss over this—this is actually a huge difference from DRM sched. One of the biggest problems I found with DRM sched is that if you call arm(), run_job() may or may not be called. Without this guarantee, you can’t do driver-side bookkeeping in arm() that is later released in run_job(), which would otherwise simplify the driver design.

You can do driver-side book-keeping after all jobs have been successfully initialized, which include arming their fences. The key turning point is when you start exposing those armed fences, not when you arm them. See below.

Quoted 7 lines
Quoted 2 lines

There is still the seqno critical section which starts at arm() and closed at push() or drop of fence.

That's orthogonal to the rule that says nothing after _arm() can fail, I think. To guarantee proper job ordering, you need extra locking (at the moment, we rely on the VM resv lock to serialize this in Panthor).

Yes, you need a lock. The VM resv lock is typically how we handle this as well, but of course in Xe it gets more complicated for kernel-issued binds, which share a queue across VMs, or in compute cases where we do not take resv locks in the exec ioctl paths.

Quoted 18 lines
Quoted 12 lines
Quoted 11 lines
Quoted 4 lines

In my opinion, it’s best and safest to enforce a no-failure policy between arm() and push().

I don't think it's safer, it's just the semantics that have been defined by drm_sched/dma_fence and that we keep forcing ourselves into. I'd rather have a well defined dma_fence state that says "that's it, I'm exposed, you have to signal me now", than this half-enforced arm()+push() model.

Quoted 5 lines

So what is the suggestion here — move the asserts I have from arm() to something like begin_push()? We could add a dma-fence state toggle there as well if we can get that part merged into dma-fence. Or should we just drop the asserts/lockdep checks between arm() and push() completely? I’m open to either approach here.

Quoted 4 lines

If we can have that INACTIVE flag added, and the associated dma_fence_init[64]_inactive() variants, I would say, we call dma_fence_init[64]_inactive() in _arm(), and we call dma_fence_set_active() in _push(). It'd still be valuable to have some

I think it is a valid use case to add an armed fence to dma-resv or sync objs before calling push(), though. Xe doesn’t do this, nor does Panthor, but IIRC other drivers do, and I believe that is completely valid. Of course, as discussed, once that is done it becomes the no-failure point. Hence my reasoing was make arm() the no failure point...

How about I add a make_active() call? I can start by moving the lockdep checks I have in place from arm() to make_active(), and we can incorporate the dma-fence suggestions you mentioned into that logic in a follow-up, since that is a broader change + would need buy from dma-resv maintainers.

Quoted 91 lines

sort of delimitation for the submission through some block-like macro with an associated context to which we can attach states and allow for more (optional?) runtime-checks.

Quoted 51 lines
Quoted 40 lines
Quoted 34 lines

FWIW, this came up while I was reviewing AMDXDNA’s DRM scheduler usage, which had the exact issue I described above. I pointed it out and got a reply saying, “well, this is an API issue, right?”—and they were correct, it is an API issue.

Quoted 19 lines
Quoted 8 lines
Quoted 6 lines

In general, I wonder if we should distinguish between "armed" and "publicly exposed" to help deal with this intra-batch dep thing without resorting to reservation and other tricks like that.

I'm not exactly sure what you suggesting but always open to ideas.

Right now _arm() is what does the dma_fence_init(). But there's an extra step between initializing the fence object and making it visible to the outside world. In order for the dep to be added to the job, you need the fence to be initialized, but that's not quite external visibility, because the job is still very much a driver object, and if something fails, the rollback mechanism makes it so all the deps are dropped on the floor along the job that's being destroyed. So we won't really wait on this fence that's never going to be signalled.

Quoted 7 lines

I see what's appealing in pretending that _arm() == externally-visible, but it's also forcing us to do extra pre-alloc (or other pre-init) operations that would otherwise not be required in the submit path. Not a hill I'm willing to die on, but I just thought I'd mention the fact I find it weird that we put extra constraints on ourselves that are not strictly needed, just because we fail to properly flag the dma_fence visibility transitions.

Quoted 5 lines

See the dma-resv example above. I’m not willing to die on this hill either, but again, in my opinion, for safety and as an API-level contract, enforcing arm() as a no-failure point makes sense. It prevents drivers from doing anything dangerous like the dma-resv example, which is an extremely subtle bug.

Quoted 10 lines

That's a valid point, but you're not really enforcing things at compile/run-time it's just "don't do this/that" in the docs. If you encode the is_active() state at the dma_fence level, properly change the fence state anytime it's about to be added to a public container, and make it so an active fence that's released without being signalled triggers a WARN_ON(), you've achieved more. Once you've done that, you can also relax the rule that says that "an armed fence has to be signalled" to "a fence that's active has to be signalled". With this, the pre-alloc for intra-batch deps in your drm_dep_job::deps xarray is no longer required, because you would be able to store inactive fences

Quoted 9 lines

I wouldn’t go that far or say it’s that simple. This would require a fairly large refactor of Xe’s VM bind pipeline to call arm() earlier, and I’m not even sure it would be possible. Between arm() and push(), the seqno critical section still remains and requires locking, in particulay the tricky case is kernel binds (e.g., page fault handling) which use the same queue. Multiple threads can issue kernel binds concurrently, as our page fault handler is multi-threaded, similar to the CPU page fault handler, so critical section between arm() and push() is very late in the pipeline tightly protected by a lock.

This sounds like a different issue to me. That's the constraint that says _arm() and _push() ordering needs to be preserved to guarantee that jobs are properly ordered on the job queue. But that's orthogonal to the rule that says nothing between _arm() and _push() on a given job can fail. Let's take the Panthor case as an example:

	for_each_job_in_batch() {
		// This acquires the VM resv lock, and all BO locks
		// Because queues target a specific VM and all jobs
		// in the a SUBMIT must target the same VM, this
		// guarantees that seqno allocation happening further
		// down (when _arm() is called) won't be interleaved
		// with other concurrent submissions to the same queues.
		lock_and_prepare_resvs()

		<--- Seqno critical section starts here
	}
 	for_each_job_in_batch() {
		// If something fails here, we drop all the jobs that
		// are part of this SUBMIT, and the resv locks are
		// released as part of the rollback. This means we
		// consumed but didn't use the seqnos, thus creating
		// a whole on the timeline, which is armless, as long
		// as those seqnos are not recycled.
		ret = faillible_stuf()

Review

Quoted 9 lines
(quoted diff) +0 -0
 		if (ret)
 			goto rollback;
 
 		arm(job)
 	}
 
 	// Nothing can fail after this point
 
 	for_each_job_in_batch() {

With my suggestion above...

		make_active(job);
Quoted 9 lines
		// resv locks are released here, unblocking other
		// concurrent submissions

update_resvs(job->done_fence)

<--- Seqno critical section ends here in case of success

(quoted diff) +0 -0
 		push(job)
 	}
Quoted 9 lines
	update_submit_syncobjs();

rollback:
unlock_resvs()
<--- Seqno critical section ends here in case of success
...

How wide your critical seqno section is is up to each driver, really.

I agree with this. The same logic follows: by adding make_active(), a driver can decide the critical non-failing section.

Another thing I could add is an option for a driver to register a lockdep class with a DRM dep queue that asserts the lock is held in arm(), make_active(), and push(). GPUVM and GPUSVM have similar interfaces, for example in functions that must be protected by a driver-side lock.

Quoted 26 lines
Quoted 26 lines
Quoted 25 lines

there, as long as they become active before the job is pushed.

Quoted 12 lines
Quoted 11 lines

On the rust side it would be directly described through the type system (see the Visibility attribute in Daniel's branch[1]). On C side, this could take the form of a new DMA_FENCE_FLAG_INACTIVE (or whichever name you want to give it). Any operation pushing the fence to public container (dma_resv, syncobj, sync_file, ...) would be rejected when that flag is set. At _push() time, we'd clear that flag with a dma_fence_set_active() helper, which would reflect the fact the fence can now be observed and exposed to the outside world.

Quoted 4 lines

Timeline squashing is problematic due to the DMA_FENCE_FLAG_INACTIVE flag. When adding a fence to dma-resv, fences that belong to the same timeline are immediately squashed. A later transition of the fence state completely breaks this behavior.

That's exactly my point: as soon as you want to insert the fence to a public container, you have to make it "active", so it will never be rolled back to the previous entry in the resv. Similarly, a wait/add_callback() on an inactive fence should be rejected.

Quoted 7 lines
Quoted 2 lines

This is bit bigger dma-fence / treewide level change but in general I believe this is a good idea.

I agree it's a bit more work. It implies patching containers to reject insertion when the INACTIVE flag is set. If we keep !INACTIVE as the default (__dma_fence_init(INACTIVE) being an opt-in), fence emitters can be moved to this model progressively though.

See above. I think we should start with the make_active() split and then see if we can get dma-resv/dma-fence updated with this semantic.

Matt

#49 lore raw

Re: [RFC PATCH 02/12] drm/dep: Add DRM dependency queue layer

In reply to Matthew Brost
Quoted 82 lines · Matthew Brost

External email: Use caution opening links or attachments

Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that drm_gpu_scheduler is no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendered drm_gpu_scheduler unmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.

Add drm_dep, a lightweight GPU submission queue intended as a replacement for drm_gpu_scheduler for firmware-managed GPU schedulers (e.g. Xe, Panthor, AMDXDNA, PVR, Nouveau, Nova). Unlike drm_gpu_scheduler, which separates the scheduler (drm_gpu_scheduler) from the queue (drm_sched_entity) into two objects requiring external coordination, drm_dep merges both roles into a single struct drm_dep_queue. This eliminates the N:1 entity-to-scheduler mapping that is unnecessary for firmware schedulers which manage their own run-lists internally.

Unlike drm_gpu_scheduler, which relies on external locking and lifetime management by the driver, drm_dep uses reference counting (kref) on both queues and jobs to guarantee object lifetime safety. A job holds a queue reference from init until its last put, and the queue holds a job reference from dispatch until the put_job worker runs. This makes use-after-free impossible even when completion arrives from IRQ context or concurrent teardown is in flight.

The core objects are:

struct drm_dep_queue - a per-context submission queue owning an
ordered submit workqueue, a TDR timeout workqueue, an SPSC job queue, and a pending-job list. Reference counted; drivers can embed it and provide a .release vfunc for RCU-safe teardown.

struct drm_dep_job - a single unit of GPU work. Drivers embed this
and provide a .release vfunc. Jobs carry an xarray of input dma_fence dependencies and produce a drm_dep_fence as their finished fence.

struct drm_dep_fence - a dma_fence subclass wrapping an optional
parent hardware fence. The finished fence is armed (sequence number assigned) before submission and signals when the hardware fence signals (or immediately on synchronous completion).

Job lifecycle:
1. drm_dep_job_init() - allocate and initialise; job acquires a queue reference.
2. drm_dep_job_add_dependency() and friends - register input fences; duplicates from the same context are deduplicated.
3. drm_dep_job_arm() - assign sequence number, obtain finished fence.
4. drm_dep_job_push() - submit to queue.

Submission paths under queue lock:
- Bypass path: if DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the SPSC queue is empty, no dependencies are pending, and credits are available, the job is dispatched inline on the calling thread.
- Queued path: job is pushed onto the SPSC queue and the run_job worker is kicked. The worker resolves remaining dependencies (installing wakeup callbacks for unresolved fences) before calling ops->run_job().

Credit-based throttling prevents hardware overflow: each job declares a credit cost at init time; dispatch is deferred until sufficient credits are available.

Timeout Detection and Recovery (TDR): a per-queue delayed work item fires when the head pending job exceeds q->job.timeout jiffies, calling ops->timedout_job(). drm_dep_queue_trigger_timeout() forces immediate expiry for device teardown.

IRQ-safe completion: queues flagged DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE allow drm_dep_job_done() to be called from hardirq context (e.g. a dma_fence callback). Dependency cleanup is deferred to process context after ops->run_job() returns to avoid calling xa_destroy() from IRQ.

Zombie-state guard: workers use kref_get_unless_zero() on entry and bail immediately if the queue refcount has already reached zero and async teardown is in flight, preventing use-after-free.

Quoted 6 lines

Teardown is always deferred to a module-private workqueue (dep_free_wq) so that destroy_workqueue() is never called from within one of the queue's own workers. Each queue holds a drm_dev_get() reference on its owning struct drm_device, released as the final step of teardown via drm_dev_put(). This prevents the driver module from being unloaded while any queue is still alive without requiring a separate drain API.

Considering workload scheduling for FW-based drivers was supposed to be simpler than legacy hardware scheduling, replacing the N:1 entity-to-scheduler mapping with this unified, refcounted drm_dep_queue makes a lot of sense.

One architectural question regarding TTM-based drivers: does drm_dep intend to handle buffer evictions and page-table (PT) update related events ? If yes, we might also need something like 'eviction fences' to halt execution, evict BOs, update the PT, and resume. Since the firmware manages the actual hardware queue, do you envision drm_dep_queue providing generic hooks/flags to temporarily pause job dispatching (or inject high-priority PT update jobs) when a memory manager needs to perform an eviction? It would be good to ensure a provision for this type of pipeline stall, as any TTM-based or dynamic-VM driver will inevitably need it.

- Shashank

Quoted 3,516 lines
Cc Boris Brezillon <[email protected]>
Cc Tvrtko Ursulin <[email protected]>
Cc Rodrigo Vivi <[email protected]>
Cc Thomas Hellström <[email protected]>
Cc Christian König <[email protected]>
Cc Danilo Krummrich <[email protected]>
Cc David Airlie <[email protected]>
Cc Maarten Lankhorst <[email protected]>
Cc Maxime Ripard <[email protected]>
Cc Philipp Stanner <[email protected]>
Cc Simona Vetter <[email protected]>
Cc Sumit Semwal <[email protected]>
Cc Thomas Zimmermann <[email protected]>
Assisted-by GitHub Copilot:claude-sonnet-4.6
---
  drivers/gpu/drm/Kconfig             |    4 +
  drivers/gpu/drm/Makefile            |    1 +
  drivers/gpu/drm/dep/Makefile        |    5 +
  drivers/gpu/drm/dep/drm_dep_fence.c |  406 +++++++
  drivers/gpu/drm/dep/drm_dep_fence.h |   25 +
  drivers/gpu/drm/dep/drm_dep_job.c   |  675 +++++++++++
  drivers/gpu/drm/dep/drm_dep_job.h   |   13 +
  drivers/gpu/drm/dep/drm_dep_queue.c | 1647 +++++++++++++++++++++++++++
  drivers/gpu/drm/dep/drm_dep_queue.h |   31 +
  include/drm/drm_dep.h               |  597 ++++++++++
  10 files changed, 3404 insertions(+)

create mode 100644 drivers/gpu/drm/dep/Makefile create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h create mode 100644 include/drm/drm_dep.h

drivers/gpu/drm/Kconfig +4 -0
index 5386248e75b6..834f6e210551 100644
@@ -276,6 +276,10 @@ config DRM_SCHED
         tristate
         depends on DRM
 
+config DRM_DEP
+       tristate
+       depends on DRM
+
  # Separate option as not all DRM drivers use it
  config DRM_PANEL_BACKLIGHT_QUIRKS
         tristate
drivers/gpu/drm/Makefile +1 -0
index e97faabcd783..1ad87cc0e545 100644
@@ -173,6 +173,7 @@ obj-y                       += clients/
  obj-y                  += display/
  obj-$(CONFIG_DRM_TTM)  += ttm/
  obj-$(CONFIG_DRM_SCHED)        += scheduler/
+obj-$(CONFIG_DRM_DEP)  += dep/
  obj-$(CONFIG_DRM_RADEON)+= radeon/
  obj-$(CONFIG_DRM_AMDGPU)+= amd/amdgpu/
  obj-$(CONFIG_DRM_AMDGPU)+= amd/amdxcp/
drivers/gpu/drm/dep/Makefile new file +5 -0
new file mode 100644
index 000000000000..335f1af46a7b
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+drm_dep-y := drm_dep_queue.o drm_dep_job.o drm_dep_fence.o
+
+obj-$(CONFIG_DRM_DEP) += drm_dep.o
drivers/gpu/drm/dep/drm_dep_fence.c new file +406 -0
new file mode 100644
index 000000000000..ae05b9077772
@@ -0,0 +1,406 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency fence
+ *
+ * Each struct drm_dep_job has an associated struct drm_dep_fence that
+ * provides a single dma_fence (@finished) signalled when the hardware
+ * completes the job.
+ *
+ * The hardware fence returned by &drm_dep_queue_ops.run_job is stored as
+ * @parent. @finished is chained to @parent via drm_dep_job_done_cb() and
+ * is signalled once @parent signals (or immediately if run_job() returns
+ * NULL or an error).
+ *
+ * Drivers should expose @finished as the out-fence for GPU work since it is
+ * valid from the moment drm_dep_job_arm() returns, whereas the hardware fence
+ * could be a compound fence, which is disallowed when installed into
+ * drm_syncobjs or dma-resv.
+ *
+ * The fence uses the kernel's inline spinlock (NULL passed to dma_fence_init())
+ * so no separate lock allocation is required.
+ *
+ * Deadline propagation is supported: if a consumer sets a deadline via
+ * dma_fence_set_deadline(), it is forwarded to @parent when @parent is set.
+ * If @parent has not been set yet the deadline is stored in @deadline and
+ * forwarded at that point.
+ *
+ * Memory management: drm_dep_fence objects are allocated with kzalloc() and
+ * freed via kfree_rcu() once the fence is released, ensuring safety with
+ * RCU-protected fence accesses.
+ */
+
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include "drm_dep_fence.h"
+
+/**
+ * DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT - a fence deadline hint has been set
+ *
+ * Set by the deadline callback on the finished fence to indicate a deadline
+ * has been set which may need to be propagated to the parent hardware fence.
+ */
+#define DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT    (DMA_FENCE_FLAG_USER_BITS + 1)
+
+/**
+ * struct drm_dep_fence - fence tracking the completion of a dep job
+ *
+ * Contains a single dma_fence (@finished) that is signalled when the
+ * hardware completes the job. The fence uses the kernel's inline_lock
+ * (no external spinlock required).
+ *
+ * This struct is private to the drm_dep module; external code interacts
+ * through the accessor functions declared in drm_dep_fence.h.
+ */
+struct drm_dep_fence {
+       /**
+        * @finished: signalled when the job completes on hardware.
+        *
+        * Drivers should use this fence as the out-fence for a job since it
+        * is available immediately upon drm_dep_job_arm().
+        */
+       struct dma_fence finished;
+
+       /**
+        * @deadline: deadline set on @finished which potentially needs to be
+        * propagated to @parent.
+        */
+       ktime_t deadline;
+
+       /**
+        * @parent: The hardware fence returned by &drm_dep_queue_ops.run_job.
+        *
+        * @finished is signaled once @parent is signaled. The initial store is
+        * performed via smp_store_release to synchronize with deadline handling.
+        *
+        * All readers must access this under the fence lock and take a reference to
+        * it, as @parent is set to NULL under the fence lock when the drm_dep_fence
+        * signals, and this drop also releases its internal reference.
+        */
+       struct dma_fence *parent;
+
+       /**
+        * @q: the queue this fence belongs to.
+        */
+       struct drm_dep_queue *q;
+};
+
+static const struct dma_fence_ops drm_dep_fence_ops;
+
+/**
+ * to_drm_dep_fence() - cast a dma_fence to its enclosing drm_dep_fence
+ * @f: dma_fence to cast
+ *
+ * Context: No context requirements (inline helper).
+ * Return: pointer to the enclosing &drm_dep_fence.
+ */
+static struct drm_dep_fence *to_drm_dep_fence(struct dma_fence *f)
+{
+       return container_of(f, struct drm_dep_fence, finished);
+}
+
+/**
+ * drm_dep_fence_set_parent() - store the hardware fence and propagate
+ *   any deadline
+ * @dfence: dep fence
+ * @parent: hardware fence returned by &drm_dep_queue_ops.run_job, or NULL/error
+ *
+ * Stores @parent on @dfence under smp_store_release() so that a concurrent
+ * drm_dep_fence_set_deadline() call sees the parent before checking the
+ * deadline bit. If a deadline has already been set on @dfence->finished it is
+ * forwarded to @parent immediately. Does nothing if @parent is NULL or an
+ * error pointer.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_set_parent(struct drm_dep_fence *dfence,
+                             struct dma_fence *parent)
+{
+       if (IS_ERR_OR_NULL(parent))
+               return;
+
+       /*
+        * smp_store_release() to ensure a thread racing us in
+        * drm_dep_fence_set_deadline() sees the parent set before
+        * it calls test_bit(HAS_DEADLINE_BIT).
+        */
+       smp_store_release(&dfence->parent, dma_fence_get(parent));
+       if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT,
+                    &dfence->finished.flags))
+               dma_fence_set_deadline(parent, dfence->deadline);
+}
+
+/**
+ * drm_dep_fence_finished() - signal the finished fence with a result
+ * @dfence: dep fence to signal
+ * @result: error code to set, or 0 for success
+ *
+ * Sets the fence error to @result if non-zero, then signals
+ * @dfence->finished. Also removes parent visibility under the fence lock
+ * and drops the parent reference. Dropping the parent here allows the
+ * DRM dep fence to be completely decoupled from the DRM dep module.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_fence_finished(struct drm_dep_fence *dfence, int result)
+{
+       struct dma_fence *parent;
+       unsigned long flags;
+
+       dma_fence_lock_irqsave(&dfence->finished, flags);
+       if (result)
+               dma_fence_set_error(&dfence->finished, result);
+       dma_fence_signal_locked(&dfence->finished);
+       parent = dfence->parent;
+       dfence->parent = NULL;
+       dma_fence_unlock_irqrestore(&dfence->finished, flags);
+
+       dma_fence_put(parent);
+}
+
+static const char *drm_dep_fence_get_driver_name(struct dma_fence *fence)
+{
+       return "drm_dep";
+}
+
+static const char *drm_dep_fence_get_timeline_name(struct dma_fence *f)
+{
+       struct drm_dep_fence *dfence = to_drm_dep_fence(f);
+
+       return dfence->q->name;
+}
+
+/**
+ * drm_dep_fence_get_parent() - get a reference to the parent hardware fence
+ * @dfence: dep fence to query
+ *
+ * Returns a new reference to @dfence->parent, or NULL if the parent has
+ * already been cleared (i.e. @dfence->finished has signalled and the parent
+ * reference was dropped under the fence lock).
+ *
+ * Uses smp_load_acquire() to pair with the smp_store_release() in
+ * drm_dep_fence_set_parent(), ensuring that if we race a concurrent
+ * drm_dep_fence_set_parent() call we observe the parent pointer only after
+ * the store is fully visible — before set_parent() tests
+ * %DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT.
+ *
+ * Caller must hold the fence lock on @dfence->finished.
+ *
+ * Context: Any context, fence lock on @dfence->finished must be held.
+ * Return: a new reference to the parent fence, or NULL.
+ */
+static struct dma_fence *drm_dep_fence_get_parent(struct drm_dep_fence *dfence)
+{
+       dma_fence_assert_held(&dfence->finished);
+
+       return dma_fence_get(smp_load_acquire(&dfence->parent));
+}
+
+/**
+ * drm_dep_fence_set_deadline() - dma_fence_ops deadline callback
+ * @f: fence on which the deadline is being set
+ * @deadline: the deadline hint to apply
+ *
+ * Stores the earliest deadline under the fence lock, then propagates
+ * it to the parent hardware fence via smp_load_acquire() to race
+ * safely with drm_dep_fence_set_parent().
+ *
+ * Context: Any context.
+ */
+static void drm_dep_fence_set_deadline(struct dma_fence *f, ktime_t deadline)
+{
+       struct drm_dep_fence *dfence = to_drm_dep_fence(f);
+       struct dma_fence *parent;
+       unsigned long flags;
+
+       dma_fence_lock_irqsave(f, flags);
+
+       /* If we already have an earlier deadline, keep it: */
+       if (test_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags) &&
+           ktime_before(dfence->deadline, deadline)) {
+               dma_fence_unlock_irqrestore(f, flags);
+               return;
+       }
+
+       dfence->deadline = deadline;
+       set_bit(DRM_DEP_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags);
+
+       parent = drm_dep_fence_get_parent(dfence);
+       dma_fence_unlock_irqrestore(f, flags);
+
+       if (parent)
+               dma_fence_set_deadline(parent, deadline);
+
+       dma_fence_put(parent);
+}
+
+static const struct dma_fence_ops drm_dep_fence_ops = {
+       .get_driver_name = drm_dep_fence_get_driver_name,
+       .get_timeline_name = drm_dep_fence_get_timeline_name,
+       .set_deadline = drm_dep_fence_set_deadline,
+};
+
+/**
+ * drm_dep_fence_alloc() - allocate a dep fence
+ *
+ * Allocates a &drm_dep_fence with kzalloc() without initialising the
+ * dma_fence. Call drm_dep_fence_init() to fully initialise it.
+ *
+ * Context: Process context.
+ * Return: new &drm_dep_fence on success, NULL on allocation failure.
+ */
+struct drm_dep_fence *drm_dep_fence_alloc(void)
+{
+       return kzalloc_obj(struct drm_dep_fence);
+}
+
+/**
+ * drm_dep_fence_init() - initialise the dma_fence inside a dep fence
+ * @dfence: dep fence to initialise
+ * @q: queue the owning job belongs to
+ *
+ * Initialises @dfence->finished using the context and sequence number from @q.
+ * Passes NULL as the lock so the fence uses its inline spinlock.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_init(struct drm_dep_fence *dfence, struct drm_dep_queue *q)
+{
+       u32 seq = ++q->fence.seqno;
+
+       /*
+        * XXX: Inline fence hazard: currently all expected users of DRM dep
+        * hardware fences have a unique lockdep class. If that ever changes,
+        * we will need to assign a unique lockdep class here so lockdep knows
+        * this fence is allowed to nest with driver hardware fences.
+        */
+
+       dfence->q = q;
+       dma_fence_init(&dfence->finished, &drm_dep_fence_ops,
+                      NULL, q->fence.context, seq);
+}
+
+/**
+ * drm_dep_fence_cleanup() - release a dep fence at job teardown
+ * @dfence: dep fence to clean up
+ *
+ * Called from drm_dep_job_fini(). If the dep fence was armed (refcount > 0)
+ * it is released via dma_fence_put() and will be freed by the RCU release
+ * callback once all waiters have dropped their references. If it was never
+ * armed it is freed directly with kfree().
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_cleanup(struct drm_dep_fence *dfence)
+{
+       if (drm_dep_fence_is_armed(dfence))
+               dma_fence_put(&dfence->finished);
+       else
+               kfree(dfence);
+}
+
+/**
+ * drm_dep_fence_is_armed() - check whether the fence has been armed
+ * @dfence: dep fence to check
+ *
+ * Returns true if drm_dep_job_arm() has been called, i.e. @dfence->finished
+ * has been initialised and its reference count is non-zero.  Used by
+ * assertions to enforce correct job lifecycle ordering (arm before push,
+ * add_dependency before arm).
+ *
+ * Context: Any context.
+ * Return: true if the fence is armed, false otherwise.
+ */
+bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence)
+{
+       return !!kref_read(&dfence->finished.refcount);
+}
+
+/**
+ * drm_dep_fence_is_finished() - test whether the finished fence has signalled
+ * @dfence: dep fence to check
+ *
+ * Uses dma_fence_test_signaled_flag() to read %DMA_FENCE_FLAG_SIGNALED_BIT
+ * directly without invoking the fence's ->signaled() callback or triggering
+ * any signalling side-effects.
+ *
+ * Context: Any context.
+ * Return: true if @dfence->finished has been signalled, false otherwise.
+ */
+bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence)
+{
+       return dma_fence_test_signaled_flag(&dfence->finished);
+}
+
+/**
+ * drm_dep_fence_is_complete() - test whether the job has completed
+ * @dfence: dep fence to check
+ *
+ * Takes the fence lock on @dfence->finished and calls
+ * drm_dep_fence_get_parent() to safely obtain a reference to the parent
+ * hardware fence — or NULL if the parent has already been cleared after
+ * signalling.  Calls dma_fence_is_signaled() on @parent outside the lock,
+ * which may invoke the fence's ->signaled() callback and trigger signalling
+ * side-effects if the fence has completed but the signalled flag has not yet
+ * been set.  The finished fence is tested via dma_fence_test_signaled_flag(),
+ * without side-effects.
+ *
+ * May only be called on a stopped queue (see drm_dep_queue_is_stopped()).
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if the job is complete, false otherwise.
+ */
+bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence)
+{
+       struct dma_fence *parent;
+       unsigned long flags;
+       bool complete;
+
+       dma_fence_lock_irqsave(&dfence->finished, flags);
+       parent = drm_dep_fence_get_parent(dfence);
+       dma_fence_unlock_irqrestore(&dfence->finished, flags);
+
+       complete = (parent && dma_fence_is_signaled(parent)) ||
+               dma_fence_test_signaled_flag(&dfence->finished);
+
+       dma_fence_put(parent);
+
+       return complete;
+}
+
+/**
+ * drm_dep_fence_to_dma() - return the finished dma_fence for a dep fence
+ * @dfence: dep fence to query
+ *
+ * No reference is taken; the caller must hold its own reference to the owning
+ * &drm_dep_job for the duration of the access.
+ *
+ * Context: Any context.
+ * Return: the finished &dma_fence.
+ */
+struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence)
+{
+       return &dfence->finished;
+}
+
+/**
+ * drm_dep_fence_done() - signal the finished fence on job completion
+ * @dfence: dep fence to signal
+ * @result: job error code, or 0 on success
+ *
+ * Gets a temporary reference to @dfence->finished to guard against a racing
+ * last-put, signals the fence with @result, then drops the temporary
+ * reference. Called from drm_dep_job_done() in the queue core when a
+ * hardware completion callback fires or when run_job() returns immediately.
+ *
+ * Context: Any context.
+ */
+void drm_dep_fence_done(struct drm_dep_fence *dfence, int result)
+{
+       dma_fence_get(&dfence->finished);
+       drm_dep_fence_finished(dfence, result);
+       dma_fence_put(&dfence->finished);
+}
drivers/gpu/drm/dep/drm_dep_fence.h new file +25 -0
new file mode 100644
index 000000000000..65a1582f858b
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_FENCE_H_
+#define _DRM_DEP_FENCE_H_
+
+#include <linux/dma-fence.h>
+
+struct drm_dep_fence;
+struct drm_dep_queue;
+
+struct drm_dep_fence *drm_dep_fence_alloc(void);
+void drm_dep_fence_init(struct drm_dep_fence *dfence, struct drm_dep_queue *q);
+void drm_dep_fence_cleanup(struct drm_dep_fence *dfence);
+void drm_dep_fence_set_parent(struct drm_dep_fence *dfence,
+                             struct dma_fence *parent);
+void drm_dep_fence_done(struct drm_dep_fence *dfence, int result);
+bool drm_dep_fence_is_armed(struct drm_dep_fence *dfence);
+bool drm_dep_fence_is_finished(struct drm_dep_fence *dfence);
+bool drm_dep_fence_is_complete(struct drm_dep_fence *dfence);
+struct dma_fence *drm_dep_fence_to_dma(struct drm_dep_fence *dfence);
+
+#endif /* _DRM_DEP_FENCE_H_ */
drivers/gpu/drm/dep/drm_dep_job.c new file +675 -0
new file mode 100644
index 000000000000..2d012b29a5fc
@@ -0,0 +1,675 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency job
+ *
+ * A struct drm_dep_job represents a single unit of GPU work associated with
+ * a struct drm_dep_queue. The lifecycle of a job is:
+ *
+ * 1. **Allocation**: the driver allocates memory for the job (typically by
+ *    embedding struct drm_dep_job in a larger structure) and calls
+ *    drm_dep_job_init() to initialise it. On success the job holds one
+ *    kref reference and a reference to its queue.
+ *
+ * 2. **Dependency collection**: the driver calls drm_dep_job_add_dependency(),
+ *    drm_dep_job_add_syncobj_dependency(), drm_dep_job_add_resv_dependencies(),
+ *    or drm_dep_job_add_implicit_dependencies() to register dma_fence objects
+ *    that must be signalled before the job can run. Duplicate fences from the
+ *    same fence context are deduplicated automatically.
+ *
+ * 3. **Arming**: drm_dep_job_arm() initialises the job's finished fence,
+ *    consuming a sequence number from the queue. After arming,
+ *    drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ *    userspace or used as a dependency by other jobs.
+ *
+ * 4. **Submission**: drm_dep_job_push() submits the job to the queue. The
+ *    queue takes a reference that it holds until the job's finished fence
+ *    signals and the job is freed by the put_job worker.
+ *
+ * 5. **Completion**: when the job's hardware work finishes its finished fence
+ *    is signalled and drm_dep_job_put() is called by the queue. The driver
+ *    must release any driver-private resources in &drm_dep_job_ops.release.
+ *
+ * Reference counting uses drm_dep_job_get() / drm_dep_job_put(). The
+ * internal drm_dep_job_fini() tears down the dependency xarray and fence
+ * objects before the driver's release callback is invoked.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/slab.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_file.h>
+#include <drm/drm_gem.h>
+#include <drm/drm_syncobj.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/**
+ * drm_dep_job_init() - initialise a dep job
+ * @job: dep job to initialise
+ * @args: initialisation arguments
+ *
+ * Initialises @job with the queue, ops and credit count from @args.  Acquires
+ * a reference to @args->q via drm_dep_queue_get(); this reference is held for
+ * the lifetime of the job and released by drm_dep_job_release() when the last
+ * job reference is dropped.
+ *
+ * Resources are released automatically when the last reference is dropped
+ * via drm_dep_job_put(), which must be called to release the job; drivers
+ * must not free the job directly.
+ *
+ * Context: Process context. Allocates memory with GFP_KERNEL.
+ * Return: 0 on success, -%EINVAL if credits is 0,
+ *   -%ENOMEM on fence allocation failure.
+ */
+int drm_dep_job_init(struct drm_dep_job *job,
+                    const struct drm_dep_job_init_args *args)
+{
+       if (unlikely(!args->credits)) {
+               pr_err("drm_dep: %s: credits cannot be 0\n", __func__);
+               return -EINVAL;
+       }
+
+       memset(job, 0, sizeof(*job));
+
+       job->dfence = drm_dep_fence_alloc();
+       if (!job->dfence)
+               return -ENOMEM;
+
+       job->ops = args->ops;
+       job->q = drm_dep_queue_get(args->q);
+       job->credits = args->credits;
+
+       kref_init(&job->refcount);
+       xa_init_flags(&job->dependencies, XA_FLAGS_ALLOC);
+       INIT_LIST_HEAD(&job->pending_link);
+
+       return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_init);
+
+/**
+ * drm_dep_job_drop_dependencies() - release all input dependency fences
+ * @job: dep job whose dependency xarray to drain
+ *
+ * Walks @job->dependencies, puts each fence, and destroys the xarray.
+ * Any slots still holding a %DRM_DEP_JOB_FENCE_PREALLOC sentinel —
+ * i.e. slots that were pre-allocated but never replaced — are silently
+ * skipped; the sentinel carries no reference.  Called from
+ * drm_dep_queue_run_job() in process context immediately after
+ * @ops->run_job() returns, before the final drm_dep_job_put().  Releasing
+ * dependencies here — while still in process context — avoids calling
+ * xa_destroy() from IRQ context if the job's last reference is later
+ * dropped from a dma_fence callback.
+ *
+ * Context: Process context.
+ */
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job)
+{
+       struct dma_fence *fence;
+       unsigned long index;
+
+       xa_for_each(&job->dependencies, index, fence) {
+               if (unlikely(fence == DRM_DEP_JOB_FENCE_PREALLOC))
+                       continue;
+               dma_fence_put(fence);
+       }
+       xa_destroy(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_fini() - clean up a dep job
+ * @job: dep job to clean up
+ *
+ * Cleans up the dep fence and drops the queue reference held by @job.
+ *
+ * If the job was never armed (e.g. init failed before drm_dep_job_arm()),
+ * the dependency xarray is also released here.  For armed jobs the xarray
+ * has already been drained by drm_dep_job_drop_dependencies() in process
+ * context immediately after run_job(), so it is left untouched to avoid
+ * calling xa_destroy() from IRQ context.
+ *
+ * Warns if @job is still linked on the queue's pending list, which would
+ * indicate a bug in the teardown ordering.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_fini(struct drm_dep_job *job)
+{
+       bool armed = drm_dep_fence_is_armed(job->dfence);
+
+       WARN_ON(!list_empty(&job->pending_link));
+
+       drm_dep_fence_cleanup(job->dfence);
+       job->dfence = NULL;
+
+       /*
+        * Armed jobs have their dependencies drained by
+        * drm_dep_job_drop_dependencies() in process context after run_job().
+        * Skip here to avoid calling xa_destroy() from IRQ context.
+        */
+       if (!armed)
+               drm_dep_job_drop_dependencies(job);
+}
+
+/**
+ * drm_dep_job_get() - acquire a reference to a dep job
+ * @job: dep job to acquire a reference on, or NULL
+ *
+ * Context: Any context.
+ * Return: @job with an additional reference held, or NULL if @job is NULL.
+ */
+struct drm_dep_job *drm_dep_job_get(struct drm_dep_job *job)
+{
+       if (job)
+               kref_get(&job->refcount);
+       return job;
+}
+EXPORT_SYMBOL(drm_dep_job_get);
+
+/**
+ * drm_dep_job_release() - kref release callback for a dep job
+ * @kref: kref embedded in the dep job
+ *
+ * Calls drm_dep_job_fini(), then invokes &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree().  Finally, releases the queue reference
+ * that was acquired by drm_dep_job_init() via drm_dep_queue_put().  The
+ * queue put is performed last to ensure no queue state is accessed after
+ * the job memory is freed.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+static void drm_dep_job_release(struct kref *kref)
+{
+       struct drm_dep_job *job =
+               container_of(kref, struct drm_dep_job, refcount);
+       struct drm_dep_queue *q = job->q;
+
+       drm_dep_job_fini(job);
+
+       if (job->ops && job->ops->release)
+               job->ops->release(job);
+       else
+               kfree(job);
+
+       drm_dep_queue_put(q);
+}
+
+/**
+ * drm_dep_job_put() - release a reference to a dep job
+ * @job: dep job to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_job_ops.release if set,
+ * otherwise frees @job with kfree(). Does nothing if @job is NULL.
+ *
+ * Context: Any context if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set on the
+ *   job's queue; otherwise process context only, as the release callback may
+ *   sleep.
+ */
+void drm_dep_job_put(struct drm_dep_job *job)
+{
+       if (job)
+               kref_put(&job->refcount, drm_dep_job_release);
+}
+EXPORT_SYMBOL(drm_dep_job_put);
+
+/**
+ * drm_dep_job_arm() - arm a dep job for submission
+ * @job: dep job to arm
+ *
+ * Initialises the finished fence on @job->dfence, assigning
+ * it a sequence number from the job's queue. Must be called after
+ * drm_dep_job_init() and before drm_dep_job_push(). Once armed,
+ * drm_dep_job_finished_fence() returns a valid fence that may be passed to
+ * userspace or used as a dependency by other jobs.
+ *
+ * Begins the DMA fence signalling path via dma_fence_begin_signalling().
+ * After this point, memory allocations that could trigger reclaim are
+ * forbidden; lockdep enforces this. arm() must always be paired with
+ * drm_dep_job_push(); lockdep also enforces this pairing.
+ *
+ * Warns if the job has already been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_arm(struct drm_dep_job *job)
+{
+       drm_dep_queue_push_job_begin(job->q);
+       WARN_ON(drm_dep_fence_is_armed(job->dfence));
+       drm_dep_fence_init(job->dfence, job->q);
+       job->signalling_cookie = dma_fence_begin_signalling();
+}
+EXPORT_SYMBOL(drm_dep_job_arm);
+
+/**
+ * drm_dep_job_push() - submit a job to its queue for execution
+ * @job: dep job to push
+ *
+ * Submits @job to the queue it was initialised with. Must be called after
+ * drm_dep_job_arm(). Acquires a reference on @job on behalf of the queue,
+ * held until the queue is fully done with it. The reference is released
+ * directly in the finished-fence dma_fence callback for queues with
+ * %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE (where drm_dep_job_done() may run
+ * from hardirq context), or via the put_job work item on the submit
+ * workqueue otherwise.
+ *
+ * Ends the DMA fence signalling path begun by drm_dep_job_arm() via
+ * dma_fence_end_signalling(). This must be paired with arm(); lockdep
+ * enforces the pairing.
+ *
+ * Once pushed, &drm_dep_queue_ops.run_job is guaranteed to be called for
+ * @job exactly once, even if the queue is killed or torn down before the
+ * job reaches the head of the queue. Drivers can use this guarantee to
+ * perform bookkeeping cleanup; the actual backend operation should be
+ * skipped when drm_dep_queue_is_killed() returns true.
+ *
+ * If the queue does not support the bypass path, the job is pushed directly
+ * onto the SPSC submission queue via drm_dep_queue_push_job() without holding
+ * @q->sched.lock. Otherwise, @q->sched.lock is taken and the job is either
+ * run immediately via drm_dep_queue_run_job() if it qualifies for bypass, or
+ * enqueued via drm_dep_queue_push_job() for dispatch by the run_job work item.
+ *
+ * Warns if the job has not been armed.
+ *
+ * Context: Process context if %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set
+ *   (takes @q->sched.lock, a mutex); any context otherwise. DMA fence signaling
+ *   path.
+ */
+void drm_dep_job_push(struct drm_dep_job *job)
+{
+       struct drm_dep_queue *q = job->q;
+
+       WARN_ON(!drm_dep_fence_is_armed(job->dfence));
+
+       drm_dep_job_get(job);
+
+       if (!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED)) {
+               drm_dep_queue_push_job(q, job);
+               dma_fence_end_signalling(job->signalling_cookie);
+               drm_dep_queue_push_job_end(job->q);
+               return;
+       }
+
+       scoped_guard(mutex, &q->sched.lock) {
+               if (drm_dep_queue_can_job_bypass(q, job))
+                       drm_dep_queue_run_job(q, job);
+               else
+                       drm_dep_queue_push_job(q, job);
+       }
+
+       dma_fence_end_signalling(job->signalling_cookie);
+       drm_dep_queue_push_job_end(job->q);
+}
+EXPORT_SYMBOL(drm_dep_job_push);
+
+/**
+ * drm_dep_job_add_dependency() - adds the fence as a job dependency
+ * @job: dep job to add the dependencies to
+ * @fence: the dma_fence to add to the list of dependencies, or
+ *         %DRM_DEP_JOB_FENCE_PREALLOC to reserve a slot for later.
+ *
+ * Note that @fence is consumed in both the success and error cases (except
+ * when @fence is %DRM_DEP_JOB_FENCE_PREALLOC, which carries no reference).
+ *
+ * Signalled fences and fences belonging to the same queue as @job (i.e. where
+ * fence->context matches the queue's finished fence context) are silently
+ * dropped; the job need not wait on its own queue's output.
+ *
+ * Warns if the job has already been armed (dependencies must be added before
+ * drm_dep_job_arm()).
+ *
+ * **Pre-allocation pattern**
+ *
+ * When multiple jobs across different queues must be prepared and submitted
+ * together in a single atomic commit — for example, where job A's finished
+ * fence is an input dependency of job B — all jobs must be armed and pushed
+ * within a single dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * region.  Once that region has started no memory allocation is permitted.
+ *
+ * To handle this, pass %DRM_DEP_JOB_FENCE_PREALLOC during the preparation
+ * phase (before arming any job, while GFP_KERNEL allocation is still allowed)
+ * to pre-allocate a slot in @job->dependencies.  The slot index assigned by
+ * the underlying xarray must be tracked by the caller separately (e.g. it is
+ * always index 0 when the dependency array is empty, as Xe relies on).
+ * After all jobs have been armed and the finished fences are available, call
+ * drm_dep_job_replace_dependency() with that index and the real fence.
+ * drm_dep_job_replace_dependency() uses GFP_NOWAIT internally and may be
+ * called from atomic or signalling context.
+ *
+ * The sentinel slot is never skipped by the signalled-fence fast-path,
+ * ensuring a slot is always allocated even when the real fence is not yet
+ * known.
+ *
+ * **Example: bind job feeding TLB invalidation jobs**
+ *
+ * Consider a GPU with separate queues for page-table bind operations and for
+ * TLB invalidation.  A single atomic commit must:
+ *
+ *  1. Run a bind job that modifies page tables.
+ *  2. Run one TLB-invalidation job per MMU that depends on the bind
+ *     completing, so stale translations are flushed before the engines
+ *     continue.
+ *
+ * Because all jobs must be armed and pushed inside a signalling region (where
+ * GFP_KERNEL is forbidden), pre-allocate slots before entering the region::
+ *
+ *   // Phase 1 — process context, GFP_KERNEL allowed
+ *   drm_dep_job_init(bind_job, bind_queue, ops);
+ *   for_each_mmu(mmu) {
+ *       drm_dep_job_init(tlb_job[mmu], tlb_queue[mmu], ops);
+ *       // Pre-allocate slot at index 0; real fence not available yet
+ *       drm_dep_job_add_dependency(tlb_job[mmu], DRM_DEP_JOB_FENCE_PREALLOC);
+ *   }
+ *
+ *   // Phase 2 — inside signalling region, no GFP_KERNEL
+ *   dma_fence_begin_signalling();
+ *   drm_dep_job_arm(bind_job);
+ *   for_each_mmu(mmu) {
+ *       // Swap sentinel for bind job's finished fence
+ *       drm_dep_job_replace_dependency(tlb_job[mmu], 0,
+ *                                      dma_fence_get(bind_job->finished));
+ *       drm_dep_job_arm(tlb_job[mmu]);
+ *   }
+ *   drm_dep_job_push(bind_job);
+ *   for_each_mmu(mmu)
+ *       drm_dep_job_push(tlb_job[mmu]);
+ *   dma_fence_end_signalling();
+ *
+ * Context: Process context. May allocate memory with GFP_KERNEL.
+ * Return: If fence == DRM_DEP_JOB_FENCE_PREALLOC index of allocation on
+ * success, else 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_dependency(struct drm_dep_job *job, struct dma_fence *fence)
+{
+       struct drm_dep_queue *q = job->q;
+       struct dma_fence *entry;
+       unsigned long index;
+       u32 id = 0;
+       int ret;
+
+       WARN_ON(drm_dep_fence_is_armed(job->dfence));
+       might_alloc(GFP_KERNEL);
+
+       if (!fence)
+               return 0;
+
+       if (fence == DRM_DEP_JOB_FENCE_PREALLOC)
+               goto add_fence;
+
+       /*
+        * Ignore signalled fences or fences from our own queue — finished
+        * fences use q->fence.context.
+        */
+       if (dma_fence_test_signaled_flag(fence) ||
+           fence->context == q->fence.context) {
+               dma_fence_put(fence);
+               return 0;
+       }
+
+       /* Deduplicate if we already depend on a fence from the same context.
+        * This lets the size of the array of deps scale with the number of
+        * engines involved, rather than the number of BOs.
+        */
+       xa_for_each(&job->dependencies, index, entry) {
+               if (entry == DRM_DEP_JOB_FENCE_PREALLOC ||
+                   entry->context != fence->context)
+                       continue;
+
+               if (dma_fence_is_later(fence, entry)) {
+                       dma_fence_put(entry);
+                       xa_store(&job->dependencies, index, fence, GFP_KERNEL);
+               } else {
+                       dma_fence_put(fence);
+               }
+               return 0;
+       }
+
+add_fence:
+       ret = xa_alloc(&job->dependencies, &id, fence, xa_limit_32b,
+                      GFP_KERNEL);
+       if (ret != 0) {
+               if (fence != DRM_DEP_JOB_FENCE_PREALLOC)
+                       dma_fence_put(fence);
+               return ret;
+       }
+
+       return (fence == DRM_DEP_JOB_FENCE_PREALLOC) ? id : 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_dependency);
+
+/**
+ * drm_dep_job_replace_dependency() - replace a pre-allocated dependency slot
+ * @job: dep job to update
+ * @index: xarray index of the slot to replace, as returned when the sentinel
+ *         was originally inserted via drm_dep_job_add_dependency()
+ * @fence: the real dma_fence to store; its reference is always consumed
+ *
+ * Replaces the %DRM_DEP_JOB_FENCE_PREALLOC sentinel at @index in
+ * @job->dependencies with @fence.  The slot must have been pre-allocated by
+ * passing %DRM_DEP_JOB_FENCE_PREALLOC to drm_dep_job_add_dependency(); the
+ * existing entry is asserted to be the sentinel.
+ *
+ * This is the second half of the pre-allocation pattern described in
+ * drm_dep_job_add_dependency().  It is intended to be called inside a
+ * dma_fence_begin_signalling() / dma_fence_end_signalling() region where
+ * memory allocation with GFP_KERNEL is forbidden.  It uses GFP_NOWAIT
+ * internally so it is safe to call from atomic or signalling context, but
+ * since the slot has been pre-allocated no actual memory allocation occurs.
+ *
+ * If @fence is already signalled the slot is erased rather than storing a
+ * redundant dependency.  The successful store is asserted — if the store
+ * fails it indicates a programming error (slot index out of range or
+ * concurrent modification).
+ *
+ * Must be called before drm_dep_job_arm(). @fence is consumed in all cases.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_job_replace_dependency(struct drm_dep_job *job, u32 index,
+                                   struct dma_fence *fence)
+{
+       WARN_ON(xa_load(&job->dependencies, index) !=
+               DRM_DEP_JOB_FENCE_PREALLOC);
+
+       if (dma_fence_test_signaled_flag(fence)) {
+               xa_erase(&job->dependencies, index);
+               dma_fence_put(fence);
+               return;
+       }
+
+       if (WARN_ON(xa_is_err(xa_store(&job->dependencies, index, fence,
+                                      GFP_NOWAIT)))) {
+               dma_fence_put(fence);
+               return;
+       }
+}
+EXPORT_SYMBOL(drm_dep_job_replace_dependency);
+
+/**
+ * drm_dep_job_add_syncobj_dependency() - adds a syncobj's fence as a
+ *   job dependency
+ * @job: dep job to add the dependencies to
+ * @file: drm file private pointer
+ * @handle: syncobj handle to lookup
+ * @point: timeline point
+ *
+ * This adds the fence matching the given syncobj to @job.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_syncobj_dependency(struct drm_dep_job *job,
+                                      struct drm_file *file, u32 handle,
+                                      u32 point)
+{
+       struct dma_fence *fence;
+       int ret;
+
+       ret = drm_syncobj_find_fence(file, handle, point, 0, &fence);
+       if (ret)
+               return ret;
+
+       return drm_dep_job_add_dependency(job, fence);
+}
+EXPORT_SYMBOL(drm_dep_job_add_syncobj_dependency);
+
+/**
+ * drm_dep_job_add_resv_dependencies() - add all fences from the resv to the job
+ * @job: dep job to add the dependencies to
+ * @resv: the dma_resv object to get the fences from
+ * @usage: the dma_resv_usage to use to filter the fences
+ *
+ * This adds all fences matching the given usage from @resv to @job.
+ * Must be called with the @resv lock held.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_resv_dependencies(struct drm_dep_job *job,
+                                     struct dma_resv *resv,
+                                     enum dma_resv_usage usage)
+{
+       struct dma_resv_iter cursor;
+       struct dma_fence *fence;
+       int ret;
+
+       dma_resv_assert_held(resv);
+
+       dma_resv_for_each_fence(&cursor, resv, usage, fence) {
+               /*
+                * As drm_dep_job_add_dependency always consumes the fence
+                * reference (even when it fails), and dma_resv_for_each_fence
+                * is not obtaining one, we need to grab one before calling.
+                */
+               ret = drm_dep_job_add_dependency(job, dma_fence_get(fence));
+               if (ret)
+                       return ret;
+       }
+       return 0;
+}
+EXPORT_SYMBOL(drm_dep_job_add_resv_dependencies);
+
+/**
+ * drm_dep_job_add_implicit_dependencies() - adds implicit dependencies
+ *   as job dependencies
+ * @job: dep job to add the dependencies to
+ * @obj: the gem object to add new dependencies from.
+ * @write: whether the job might write the object (so we need to depend on
+ * shared fences in the reservation object).
+ *
+ * This should be called after drm_gem_lock_reservations() on your array of
+ * GEM objects used in the job but before updating the reservations with your
+ * own fences.
+ *
+ * Context: Process context.
+ * Return: 0 on success, or a negative error code.
+ */
+int drm_dep_job_add_implicit_dependencies(struct drm_dep_job *job,
+                                         struct drm_gem_object *obj,
+                                         bool write)
+{
+       return drm_dep_job_add_resv_dependencies(job, obj->resv,
+                                                dma_resv_usage_rw(write));
+}
+EXPORT_SYMBOL(drm_dep_job_add_implicit_dependencies);
+
+/**
+ * drm_dep_job_is_signaled() - check whether a dep job has completed
+ * @job: dep job to check
+ *
+ * Determines whether @job has signalled. The queue should be stopped before
+ * calling this to obtain a stable snapshot of state. Both the parent hardware
+ * fence and the finished software fence are checked.
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if the job is signalled, false otherwise.
+ */
+bool drm_dep_job_is_signaled(struct drm_dep_job *job)
+{
+       WARN_ON(!drm_dep_queue_is_stopped(job->q));
+       return drm_dep_fence_is_complete(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_is_signaled);
+
+/**
+ * drm_dep_job_is_finished() - test whether a dep job's finished fence has signalled
+ * @job: dep job to check
+ *
+ * Tests whether the job's software finished fence has been signalled, using
+ * dma_fence_test_signaled_flag() to avoid any signalling side-effects. Unlike
+ * drm_dep_job_is_signaled(), this does not require the queue to be stopped and
+ * does not check the parent hardware fence — it is a lightweight test of the
+ * finished fence only.
+ *
+ * Context: Any context.
+ * Return: true if the job's finished fence has been signalled, false otherwise.
+ */
+bool drm_dep_job_is_finished(struct drm_dep_job *job)
+{
+       return drm_dep_fence_is_finished(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_is_finished);
+
+/**
+ * drm_dep_job_invalidate_job() - increment the invalidation count for a job
+ * @job: dep job to invalidate
+ * @threshold: threshold above which the job is considered invalidated
+ *
+ * Increments @job->invalidate_count and returns true if it exceeds @threshold,
+ * indicating the job should be considered hung and discarded. The queue must
+ * be stopped before calling this function.
+ *
+ * Context: Process context. The queue must be stopped before calling this.
+ * Return: true if @job->invalidate_count exceeds @threshold, false otherwise.
+ */
+bool drm_dep_job_invalidate_job(struct drm_dep_job *job, int threshold)
+{
+       WARN_ON(!drm_dep_queue_is_stopped(job->q));
+       return ++job->invalidate_count > threshold;
+}
+EXPORT_SYMBOL(drm_dep_job_invalidate_job);
+
+/**
+ * drm_dep_job_finished_fence() - return the finished fence for a job
+ * @job: dep job to query
+ *
+ * No reference is taken on the returned fence; the caller must hold its own
+ * reference to @job for the duration of any access.
+ *
+ * Context: Any context.
+ * Return: the finished &dma_fence for @job.
+ */
+struct dma_fence *drm_dep_job_finished_fence(struct drm_dep_job *job)
+{
+       return drm_dep_fence_to_dma(job->dfence);
+}
+EXPORT_SYMBOL(drm_dep_job_finished_fence);
drivers/gpu/drm/dep/drm_dep_job.h new file +13 -0
new file mode 100644
index 000000000000..35c61d258fa1
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_JOB_H_
+#define _DRM_DEP_JOB_H_
+
+struct drm_dep_queue;
+
+void drm_dep_job_drop_dependencies(struct drm_dep_job *job);
+
+#endif /* _DRM_DEP_JOB_H_ */
drivers/gpu/drm/dep/drm_dep_queue.c new file +1647 -0
new file mode 100644
index 000000000000..dac02d0d22c4
@@ -0,0 +1,1647 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+/**
+ * DOC: DRM dependency queue
+ *
+ * The drm_dep subsystem provides a lightweight GPU submission queue that
+ * combines the roles of drm_gpu_scheduler and drm_sched_entity into a
+ * single object (struct drm_dep_queue). Each queue owns its own ordered
+ * submit workqueue, timeout workqueue, and TDR delayed-work.
+ *
+ * **Job lifecycle**
+ *
+ * 1. Allocate and initialise a job with drm_dep_job_init().
+ * 2. Add dependency fences with drm_dep_job_add_dependency() and friends.
+ * 3. Arm the job with drm_dep_job_arm() to obtain its out-fences.
+ * 4. Submit with drm_dep_job_push().
+ *
+ * **Submission paths**
+ *
+ * drm_dep_job_push() decides between two paths under @q->sched.lock:
+ *
+ * - **Bypass path** (drm_dep_queue_can_job_bypass()): if
+ *   %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set, the queue is not stopped,
+ *   the SPSC queue is empty, the job has no dependency fences, and credits
+ *   are available, the job is submitted inline on the calling thread without
+ *   touching the submit workqueue.
+ *
+ * - **Queued path** (drm_dep_queue_push_job()): the job is pushed onto an
+ *   SPSC queue and the run_job worker is kicked. The run_job worker pops the
+ *   job, resolves any remaining dependency fences (installing wakeup
+ *   callbacks for unresolved ones), and calls drm_dep_queue_run_job().
+ *
+ * **Running a job**
+ *
+ * drm_dep_queue_run_job() accounts credits, appends the job to the pending
+ * list (starting the TDR timer only when the list was previously empty),
+ * calls @ops->run_job(), stores the returned hardware fence as the parent
+ * of the job's dep fence, then installs a callback on it. When the hardware
+ * fence fires (or the job completes synchronously), drm_dep_job_done()
+ * signals the finished fence, returns credits, and kicks the put_job worker
+ * to free the job.
+ *
+ * **Timeout detection and recovery (TDR)**
+ *
+ * A delayed work item fires when a job on the pending list takes longer than
+ * @q->job.timeout jiffies. It calls @ops->timedout_job() and acts on the
+ * returned status (%DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED or
+ * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB).
+ * drm_dep_queue_trigger_timeout() forces the timer to fire immediately (without
+ * changing the stored timeout), for example during device teardown.
+ *
+ * **Reference counting**
+ *
+ * Jobs and queues are both reference counted.
+ *
+ * A job holds a reference to its queue from drm_dep_job_init() until
+ * drm_dep_job_put() drops the job's last reference and its release callback
+ * runs. This ensures the queue remains valid for the entire lifetime of any
+ * job that was submitted to it.
+ *
+ * The queue holds its own reference to a job for as long as the job is
+ * internally tracked: from the moment the job is added to the pending list
+ * in drm_dep_queue_run_job() until drm_dep_job_done() kicks the put_job
+ * worker, which calls drm_dep_job_put() to release that reference.
+ *
+ * **Hazard: use-after-free from within a worker**
+ *
+ * Because a job holds a queue reference, drm_dep_job_put() dropping the last
+ * job reference will also drop a queue reference via the job's release path.
+ * If that happens to be the last queue reference, drm_dep_queue_fini() can be
+ * called, which queues @q->free_work on dep_free_wq and returns immediately.
+ * free_work calls disable_work_sync() / disable_delayed_work_sync() on the
+ * queue's own workers before destroying its workqueues, so in practice a
+ * running worker always completes before the queue memory is freed.
+ *
+ * However, there is a secondary hazard: a worker can be queued while the
+ * queue is in a "zombie" state — refcount has already reached zero and async
+ * teardown is in flight, but the work item has not yet been disabled by
+ * free_work.  To guard against this every worker uses
+ * drm_dep_queue_get_unless_zero() at entry; if the refcount is already zero
+ * the worker bails immediately without touching the queue state.
+ *
+ * Because all actual teardown (disable_*_sync, destroy_workqueue) runs on
+ * dep_free_wq — which is independent of the queue's own submit/timeout
+ * workqueues — there is no deadlock risk.  Each queue holds a drm_dev_get()
+ * reference on its owning &drm_device, which is released as the last step of
+ * teardown.  This ensures the driver module cannot be unloaded while any queue
+ * is still alive.
+ */
+
+#include <linux/dma-resv.h>
+#include <linux/kref.h>
+#include <linux/module.h>
+#include <linux/overflow.h>
+#include <linux/slab.h>
+#include <linux/wait.h>
+#include <linux/workqueue.h>
+#include <drm/drm_dep.h>
+#include <drm/drm_drv.h>
+#include <drm/drm_print.h>
+#include "drm_dep_fence.h"
+#include "drm_dep_job.h"
+#include "drm_dep_queue.h"
+
+/*
+ * Dedicated workqueue for deferred drm_dep_queue teardown.  Using a
+ * module-private WQ instead of system_percpu_wq keeps teardown isolated
+ * from unrelated kernel subsystems.
+ */
+static struct workqueue_struct *dep_free_wq;
+
+/**
+ * drm_dep_queue_flags_set() - set a flag on the queue under sched.lock
+ * @q: dep queue
+ * @flag: flag to set (one of &enum drm_dep_queue_flags)
+ *
+ * Sets @flag in @q->sched.flags. Must be called with @q->sched.lock
+ * held; the lockdep assertion enforces this.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_flags_set(struct drm_dep_queue *q,
+                                   enum drm_dep_queue_flags flag)
+{
+       lockdep_assert_held(&q->sched.lock);
+       q->sched.flags |= flag;
+}
+
+/**
+ * drm_dep_queue_flags_clear() - clear a flag on the queue under sched.lock
+ * @q: dep queue
+ * @flag: flag to clear (one of &enum drm_dep_queue_flags)
+ *
+ * Clears @flag in @q->sched.flags. Must be called with @q->sched.lock
+ * held; the lockdep assertion enforces this.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_flags_clear(struct drm_dep_queue *q,
+                                     enum drm_dep_queue_flags flag)
+{
+       lockdep_assert_held(&q->sched.lock);
+       q->sched.flags &= ~flag;
+}
+
+/**
+ * drm_dep_queue_has_credits() - check whether the queue has enough credits
+ * @q: dep queue
+ * @job: job requesting credits
+ *
+ * Checks whether the queue has enough available credits to dispatch
+ * @job. If @job->credits exceeds the queue's credit limit, it is
+ * clamped with a WARN.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if available credits >= @job->credits, false otherwise.
+ */
+static bool drm_dep_queue_has_credits(struct drm_dep_queue *q,
+                                     struct drm_dep_job *job)
+{
+       u32 available;
+
+       lockdep_assert_held(&q->sched.lock);
+
+       if (job->credits > q->credit.limit) {
+               drm_warn(q->drm,
+                        "Jobs may not exceed the credit limit, truncate.\n");
+               job->credits = q->credit.limit;
+       }
+
+       WARN_ON(check_sub_overflow(q->credit.limit,
+                                  atomic_read(&q->credit.count),
+                                  &available));
+
+       return available >= job->credits;
+}
+
+/**
+ * drm_dep_queue_run_job_queue() - kick the run-job worker
+ * @q: dep queue
+ *
+ * Queues @q->sched.run_job on @q->sched.submit_wq unless the queue is stopped
+ * or the job queue is empty.  The empty-queue check avoids queueing a work item
+ * that would immediately return with nothing to do.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_run_job_queue(struct drm_dep_queue *q)
+{
+       if (!drm_dep_queue_is_stopped(q) && spsc_queue_count(&q->job.queue))
+               queue_work(q->sched.submit_wq, &q->sched.run_job);
+}
+
+/**
+ * drm_dep_queue_put_job_queue() - kick the put-job worker
+ * @q: dep queue
+ *
+ * Queues @q->sched.put_job on @q->sched.submit_wq unless the queue
+ * is stopped.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_put_job_queue(struct drm_dep_queue *q)
+{
+       if (!drm_dep_queue_is_stopped(q))
+               queue_work(q->sched.submit_wq, &q->sched.put_job);
+}
+
+/**
+ * drm_queue_start_timeout() - arm or re-arm the TDR delayed work
+ * @q: dep queue
+ *
+ * Arms the TDR delayed work with @q->job.timeout. No-op if
+ * @q->ops->timedout_job is NULL, the timeout is MAX_SCHEDULE_TIMEOUT,
+ * or the pending list is empty.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_queue_start_timeout(struct drm_dep_queue *q)
+{
+       lockdep_assert_held(&q->job.lock);
+
+       if (!q->ops->timedout_job ||
+           q->job.timeout == MAX_SCHEDULE_TIMEOUT ||
+           list_empty(&q->job.pending))
+               return;
+
+       mod_delayed_work(q->sched.timeout_wq, &q->sched.tdr, q->job.timeout);
+}
+
+/**
+ * drm_queue_start_timeout_unlocked() - arm TDR, acquiring job.lock
+ * @q: dep queue
+ *
+ * Acquires @q->job.lock with irq and calls
+ * drm_queue_start_timeout().
+ *
+ * Context: Process context (workqueue).
+ */
+static void drm_queue_start_timeout_unlocked(struct drm_dep_queue *q)
+{
+       guard(spinlock_irq)(&q->job.lock);
+       drm_queue_start_timeout(q);
+}
+
+/**
+ * drm_dep_queue_remove_dependency() - clear the active dependency and wake
+ *   the run-job worker
+ * @q: dep queue
+ * @f: the dependency fence being removed
+ *
+ * Stores @f into @q->dep.removed_fence via smp_store_release() so that the
+ * run-job worker can drop the reference to it in drm_dep_queue_is_ready(),
+ * paired with smp_load_acquire().  Clears @q->dep.fence and kicks the
+ * run-job worker.
+ *
+ * The fence reference is not dropped here; it is deferred to the run-job
+ * worker via @q->dep.removed_fence to keep this path suitable dma_fence
+ * callback removal in drm_dep_queue_kill().
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_remove_dependency(struct drm_dep_queue *q,
+                                           struct dma_fence *f)
+{
+       /* removed_fence must be visible to the reader before &q->dep.fence */
+       smp_store_release(&q->dep.removed_fence, f);
+
+       WRITE_ONCE(q->dep.fence, NULL);
+       drm_dep_queue_run_job_queue(q);
+}
+
+/**
+ * drm_dep_queue_wakeup() - dma_fence callback to wake the run-job worker
+ * @f: the signalled dependency fence
+ * @cb: callback embedded in the dep queue
+ *
+ * Called from dma_fence_signal() when the active dependency fence signals.
+ * Delegates to drm_dep_queue_remove_dependency() to clear @q->dep.fence and
+ * kick the run-job worker.  The fence reference is not dropped here; it is
+ * deferred to the run-job worker via @q->dep.removed_fence.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+       struct drm_dep_queue *q =
+               container_of(cb, struct drm_dep_queue, dep.cb);
+
+       drm_dep_queue_remove_dependency(q, f);
+}
+
+/**
+ * drm_dep_queue_is_ready() - check whether the queue has a dispatchable job
+ * @q: dep queue
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if SPSC queue non-empty and no dep fence pending,
+ *   false otherwise.
+ */
+static bool drm_dep_queue_is_ready(struct drm_dep_queue *q)
+{
+       lockdep_assert_held(&q->sched.lock);
+
+       if (!spsc_queue_count(&q->job.queue))
+               return false;
+
+       if (READ_ONCE(q->dep.fence))
+               return false;
+
+       /* Paired with smp_store_release in drm_dep_queue_remove_dependency() */
+       dma_fence_put(smp_load_acquire(&q->dep.removed_fence));
+
+       q->dep.removed_fence = NULL;
+
+       return true;
+}
+
+/**
+ * drm_dep_queue_is_killed() - check whether a dep queue has been killed
+ * @q: dep queue to check
+ *
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_KILLED is set on @q, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_killed(struct drm_dep_queue *q)
+{
+       return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_KILLED);
+}
+EXPORT_SYMBOL(drm_dep_queue_is_killed);
+
+/**
+ * drm_dep_queue_is_initialized() - check whether a dep queue has been initialized
+ * @q: dep queue to check
+ *
+ * A queue is considered initialized once its ops pointer has been set by a
+ * successful call to drm_dep_queue_init().  Drivers that embed a
+ * &drm_dep_queue inside a larger structure may call this before attempting any
+ * other queue operation to confirm that initialization has taken place.
+ * drm_dep_queue_put() must be called if this function returns true to drop the
+ * initialization reference from drm_dep_queue_init().
+ *
+ * Return: true if @q has been initialized, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_initialized(struct drm_dep_queue *q)
+{
+       return !!q->ops;
+}
+EXPORT_SYMBOL(drm_dep_queue_is_initialized);
+
+/**
+ * drm_dep_queue_set_stopped() - pre-mark a queue as stopped before first use
+ * @q: dep queue to mark
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED directly on @q without going through the
+ * normal drm_dep_queue_stop() path.  This is only valid during the driver-side
+ * queue initialisation sequence — i.e. after drm_dep_queue_init() returns but
+ * before the queue is made visible to other threads (e.g. before it is added
+ * to any lookup structures).  Using this after the queue is live is a driver
+ * bug; use drm_dep_queue_stop() instead.
+ *
+ * Context: Process context, queue not yet visible to other threads.
+ */
+void drm_dep_queue_set_stopped(struct drm_dep_queue *q)
+{
+       q->sched.flags |= DRM_DEP_QUEUE_FLAGS_STOPPED;
+}
+EXPORT_SYMBOL(drm_dep_queue_set_stopped);
+
+/**
+ * drm_dep_queue_refcount() - read the current reference count of a queue
+ * @q: dep queue to query
+ *
+ * Returns the instantaneous kref value.  The count may change immediately
+ * after this call; callers must not make safety decisions based solely on
+ * the returned value.  Intended for diagnostic snapshots and debugfs output.
+ *
+ * Context: Any context.
+ * Return: current reference count.
+ */
+unsigned int drm_dep_queue_refcount(const struct drm_dep_queue *q)
+{
+       return kref_read(&q->refcount);
+}
+EXPORT_SYMBOL(drm_dep_queue_refcount);
+
+/**
+ * drm_dep_queue_timeout() - read the per-job TDR timeout for a queue
+ * @q: dep queue to query
+ *
+ * Returns the per-job timeout in jiffies as set at init time.
+ * %MAX_SCHEDULE_TIMEOUT means no timeout is configured.
+ *
+ * Context: Any context.
+ * Return: timeout in jiffies.
+ */
+long drm_dep_queue_timeout(const struct drm_dep_queue *q)
+{
+       return q->job.timeout;
+}
+EXPORT_SYMBOL(drm_dep_queue_timeout);
+
+/**
+ * drm_dep_queue_is_job_put_irq_safe() - test whether job-put from IRQ is allowed
+ * @q: dep queue
+ *
+ * Context: Any context.
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set,
+ *   false otherwise.
+ */
+static bool drm_dep_queue_is_job_put_irq_safe(const struct drm_dep_queue *q)
+{
+       return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE);
+}
+
+/**
+ * drm_dep_queue_job_dependency() - get next unresolved dep fence
+ * @q: dep queue
+ * @job: job whose dependencies to advance
+ *
+ * Returns NULL immediately if the queue has been killed via
+ * drm_dep_queue_kill(), bypassing all dependency waits so that jobs
+ * drain through run_job as quickly as possible.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: next unresolved &dma_fence with a new reference, or NULL
+ *   when all dependencies have been consumed (or the queue is killed).
+ */
+static struct dma_fence *
+drm_dep_queue_job_dependency(struct drm_dep_queue *q,
+                            struct drm_dep_job *job)
+{
+       struct dma_fence *f;
+
+       lockdep_assert_held(&q->sched.lock);
+
+       if (drm_dep_queue_is_killed(q))
+               return NULL;
+
+       f = xa_load(&job->dependencies, job->last_dependency);
+       if (f) {
+               job->last_dependency++;
+               if (WARN_ON(DRM_DEP_JOB_FENCE_PREALLOC == f))
+                       return dma_fence_get_stub();
+               return dma_fence_get(f);
+       }
+
+       return NULL;
+}
+
+/**
+ * drm_dep_queue_add_dep_cb() - install wakeup callback on dep fence
+ * @q: dep queue
+ * @job: job whose dependency fence is stored in @q->dep.fence
+ *
+ * Installs a wakeup callback on @q->dep.fence. Returns true if the
+ * callback was installed (the queue must wait), false if the fence is
+ * already signalled or is a self-fence from the same queue context.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: true if callback installed, false if fence already done.
+ */
+static bool drm_dep_queue_add_dep_cb(struct drm_dep_queue *q,
+                                    struct drm_dep_job *job)
+{
+       struct dma_fence *fence = q->dep.fence;
+
+       lockdep_assert_held(&q->sched.lock);
+
+       if (WARN_ON(fence->context == q->fence.context)) {
+               dma_fence_put(q->dep.fence);
+               q->dep.fence = NULL;
+               return false;
+       }
+
+       if (!dma_fence_add_callback(q->dep.fence, &q->dep.cb,
+                                   drm_dep_queue_wakeup))
+               return true;
+
+       dma_fence_put(q->dep.fence);
+       q->dep.fence = NULL;
+
+       return false;
+}
+
+/**
+ * drm_dep_queue_pop_job() - pop a dispatchable job from the SPSC queue
+ * @q: dep queue
+ *
+ * Peeks at the head of the SPSC queue and drains all resolved
+ * dependencies. If a dependency is still pending, installs a wakeup
+ * callback and returns NULL. On success pops the job and returns it.
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ * Return: next dispatchable job, or NULL if a dep is still pending.
+ */
+static struct drm_dep_job *drm_dep_queue_pop_job(struct drm_dep_queue *q)
+{
+       struct spsc_node *node;
+       struct drm_dep_job *job;
+
+       lockdep_assert_held(&q->sched.lock);
+
+       node = spsc_queue_peek(&q->job.queue);
+       if (!node)
+               return NULL;
+
+       job = container_of(node, struct drm_dep_job, queue_node);
+
+       while ((q->dep.fence = drm_dep_queue_job_dependency(q, job))) {
+               if (drm_dep_queue_add_dep_cb(q, job))
+                       return NULL;
+       }
+
+       spsc_queue_pop(&q->job.queue);
+
+       return job;
+}
+
+/*
+ * drm_dep_queue_get_unless_zero() - try to acquire a queue reference
+ *
+ * Workers use this instead of drm_dep_queue_get() to guard against the zombie
+ * state: the queue's refcount has already reached zero (async teardown is in
+ * flight) but a work item was queued before free_work had a chance to cancel
+ * it.  If kref_get_unless_zero() fails the caller must bail immediately.
+ *
+ * Context: Any context.
+ * Returns true if the reference was acquired, false if the queue is zombie.
+ */
+bool drm_dep_queue_get_unless_zero(struct drm_dep_queue *q)
+{
+       return kref_get_unless_zero(&q->refcount);
+}
+EXPORT_SYMBOL(drm_dep_queue_get_unless_zero);
+
+/**
+ * drm_dep_queue_run_job_work() - run-job worker
+ * @work: work item embedded in the dep queue
+ *
+ * Acquires @q->sched.lock, checks stopped state, queue readiness and
+ * available credits, pops the next job via drm_dep_queue_pop_job(),
+ * dispatches it via drm_dep_queue_run_job(), then re-kicks itself.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_run_job_work(struct work_struct *work)
+{
+       struct drm_dep_queue *q =
+               container_of(work, struct drm_dep_queue, sched.run_job);
+       struct spsc_node *node;
+       struct drm_dep_job *job;
+       bool cookie = dma_fence_begin_signalling();
+
+       /* Bail if queue is zombie (refcount already zero, teardown in flight). */
+       if (!drm_dep_queue_get_unless_zero(q)) {
+               dma_fence_end_signalling(cookie);
+               return;
+       }
+
+       mutex_lock(&q->sched.lock);
+
+       if (drm_dep_queue_is_stopped(q))
+               goto put_queue;
+
+       if (!drm_dep_queue_is_ready(q))
+               goto put_queue;
+
+       /* Peek to check credits before committing to pop and dep resolution */
+       node = spsc_queue_peek(&q->job.queue);
+       if (!node)
+               goto put_queue;
+
+       job = container_of(node, struct drm_dep_job, queue_node);
+       if (!drm_dep_queue_has_credits(q, job))
+               goto put_queue;
+
+       job = drm_dep_queue_pop_job(q);
+       if (!job)
+               goto put_queue;
+
+       drm_dep_queue_run_job(q, job);
+       drm_dep_queue_run_job_queue(q);
+
+put_queue:
+       mutex_unlock(&q->sched.lock);
+       drm_dep_queue_put(q);
+       dma_fence_end_signalling(cookie);
+}
+
+/*
+ * drm_dep_queue_remove_job() - unlink a job from the pending list and reset TDR
+ * @q:   dep queue owning @job
+ * @job: job to remove
+ *
+ * Splices @job out of @q->job.pending, cancels any pending TDR delayed work,
+ * and arms the timeout for the new list head (if any).
+ *
+ * Context: Process context. Must hold @q->sched.lock. DMA fence signaling path.
+ */
+static void drm_dep_queue_remove_job(struct drm_dep_queue *q,
+                                    struct drm_dep_job *job)
+{
+       lockdep_assert_held(&q->job.lock);
+
+       list_del_init(&job->pending_link);
+       cancel_delayed_work(&q->sched.tdr);
+       drm_queue_start_timeout(q);
+}
+
+/**
+ * drm_dep_queue_get_finished_job() - dequeue a finished job
+ * @q: dep queue
+ *
+ * Under @q->job.lock checks the head of the pending list for a
+ * finished dep fence. If found, removes the job from the list,
+ * cancels the TDR, and re-arms it for the new head.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ * Return: the finished &drm_dep_job, or NULL if none is ready.
+ */
+static struct drm_dep_job *
+drm_dep_queue_get_finished_job(struct drm_dep_queue *q)
+{
+       struct drm_dep_job *job;
+
+       guard(spinlock_irq)(&q->job.lock);
+
+       job = list_first_entry_or_null(&q->job.pending, struct drm_dep_job,
+                                      pending_link);
+       if (job && drm_dep_fence_is_finished(job->dfence))
+               drm_dep_queue_remove_job(q, job);
+       else
+               job = NULL;
+
+       return job;
+}
+
+/**
+ * drm_dep_queue_put_job_work() - put-job worker
+ * @work: work item embedded in the dep queue
+ *
+ * Drains all finished jobs by calling drm_dep_job_put() in a loop,
+ * then kicks the run-job worker.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * because workqueue is shared with other items in the fence signaling path.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_put_job_work(struct work_struct *work)
+{
+       struct drm_dep_queue *q =
+               container_of(work, struct drm_dep_queue, sched.put_job);
+       struct drm_dep_job *job;
+       bool cookie = dma_fence_begin_signalling();
+
+       /* Bail if queue is zombie (refcount already zero, teardown in flight). */
+       if (!drm_dep_queue_get_unless_zero(q)) {
+               dma_fence_end_signalling(cookie);
+               return;
+       }
+
+       while ((job = drm_dep_queue_get_finished_job(q)))
+               drm_dep_job_put(job);
+
+       drm_dep_queue_run_job_queue(q);
+
+       drm_dep_queue_put(q);
+       dma_fence_end_signalling(cookie);
+}
+
+/**
+ * drm_dep_queue_tdr_work() - TDR worker
+ * @work: work item embedded in the delayed TDR work
+ *
+ * Removes the head job from the pending list under @q->job.lock,
+ * asserts @q->ops->timedout_job is non-NULL, calls it outside the lock,
+ * requeues the job if %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB, drops the
+ * queue's job reference on %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED, and always
+ * restarts the TDR timer after handling the job (unless @q is stopping).
+ * Any other return value triggers a WARN.
+ *
+ * The TDR is never armed when @q->ops->timedout_job is NULL, so firing
+ * this worker without a timedout_job callback is a driver bug.
+ *
+ * Uses drm_dep_queue_get_unless_zero() at entry and bails immediately if the
+ * queue is in zombie state (refcount already zero, async teardown in flight).
+ *
+ * Wraps execution in dma_fence_begin_signalling() / dma_fence_end_signalling()
+ * because timedout_job() is expected to signal the guilty job's fence as part
+ * of reset.
+ *
+ * Context: Process context (workqueue). DMA fence signaling path.
+ */
+static void drm_dep_queue_tdr_work(struct work_struct *work)
+{
+       struct drm_dep_queue *q =
+               container_of(work, struct drm_dep_queue, sched.tdr.work);
+       struct drm_dep_job *job;
+       bool cookie = dma_fence_begin_signalling();
+
+       /* Bail if queue is zombie (refcount already zero, teardown in flight). */
+       if (!drm_dep_queue_get_unless_zero(q)) {
+               dma_fence_end_signalling(cookie);
+               return;
+       }
+
+       scoped_guard(spinlock_irq, &q->job.lock) {
+               job = list_first_entry_or_null(&q->job.pending,
+                                              struct drm_dep_job,
+                                              pending_link);
+               if (job)
+                       /*
+                        * Remove from pending so it cannot be freed
+                        * concurrently by drm_dep_queue_get_finished_job() or
+                        * .drm_dep_job_done().
+                        */
+                       list_del_init(&job->pending_link);
+       }
+
+       if (job) {
+               enum drm_dep_timedout_stat status;
+
+               if (WARN_ON(!q->ops->timedout_job)) {
+                       drm_dep_job_put(job);
+                       goto out;
+               }
+
+               status = q->ops->timedout_job(job);
+
+               switch (status) {
+               case DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB:
+                       scoped_guard(spinlock_irq, &q->job.lock)
+                               list_add(&job->pending_link, &q->job.pending);
+                       drm_dep_queue_put_job_queue(q);
+                       break;
+               case DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED:
+                       drm_dep_job_put(job);
+                       break;
+               default:
+                       WARN_ON("invalid drm_dep_timedout_stat");
+                       break;
+               }
+       }
+
+out:
+       drm_queue_start_timeout_unlocked(q);
+       drm_dep_queue_put(q);
+       dma_fence_end_signalling(cookie);
+}
+
+/**
+ * drm_dep_alloc_submit_wq() - allocate an ordered submit workqueue
+ * @name: name for the workqueue
+ * @flags: DRM_DEP_QUEUE_FLAGS_* flags
+ *
+ * Allocates an ordered workqueue for job submission with %WQ_MEM_RECLAIM and
+ * %WQ_MEM_WARN_ON_RECLAIM set, ensuring the workqueue is safe to use from
+ * memory reclaim context and properly annotated for lockdep taint tracking.
+ * Adds %WQ_HIGHPRI if %DRM_DEP_QUEUE_FLAGS_HIGHPRI is set. When
+ * CONFIG_LOCKDEP is enabled, uses a dedicated lockdep map for annotation.
+ *
+ * Context: Process context.
+ * Return: the new &workqueue_struct, or NULL on failure.
+ */
+static struct workqueue_struct *
+drm_dep_alloc_submit_wq(const char *name, enum drm_dep_queue_flags flags)
+{
+       unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM;
+
+       if (flags & DRM_DEP_QUEUE_FLAGS_HIGHPRI)
+               wq_flags |= WQ_HIGHPRI;
+
+#if IS_ENABLED(CONFIG_LOCKDEP)
+       static struct lockdep_map map = {
+               .name = "drm_dep_submit_lockdep_map"
+       };
+       return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map);
+#else
+       return alloc_ordered_workqueue(name, wq_flags);
+#endif
+}
+
+/**
+ * drm_dep_alloc_timeout_wq() - allocate an ordered TDR workqueue
+ * @name: name for the workqueue
+ *
+ * Allocates an ordered workqueue for timeout detection and recovery with
+ * %WQ_MEM_RECLAIM and %WQ_MEM_WARN_ON_RECLAIM set, ensuring consistent taint
+ * annotation with the submit workqueue. When CONFIG_LOCKDEP is enabled, uses
+ * a dedicated lockdep map for annotation.
+ *
+ * Context: Process context.
+ * Return: the new &workqueue_struct, or NULL on failure.
+ */
+static struct workqueue_struct *drm_dep_alloc_timeout_wq(const char *name)
+{
+       unsigned int wq_flags = WQ_MEM_RECLAIM | WQ_MEM_WARN_ON_RECLAIM;
+
+#if IS_ENABLED(CONFIG_LOCKDEP)
+       static struct lockdep_map map = {
+               .name = "drm_dep_timeout_lockdep_map"
+       };
+       return alloc_ordered_workqueue_lockdep_map(name, wq_flags, &map);
+#else
+       return alloc_ordered_workqueue(name, wq_flags);
+#endif
+}
+
+/**
+ * drm_dep_queue_init() - initialize a dep queue
+ * @q: dep queue to initialize
+ * @args: initialization arguments
+ *
+ * Initializes all fields of @q from @args. If @args->submit_wq is NULL an
+ * ordered workqueue is allocated and owned by the queue
+ * (%DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ). If @args->timeout_wq is NULL an
+ * ordered workqueue is allocated and owned by the queue
+ * (%DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ). On success the queue holds one kref
+ * reference and drm_dep_queue_put() must be called to drop this reference
+ * (i.e., drivers cannot directly free the queue).
+ *
+ * When CONFIG_LOCKDEP is enabled, @q->sched.lock is primed against the
+ * fs_reclaim pseudo-lock so that lockdep can detect any lock ordering
+ * inversion between @sched.lock and memory reclaim.
+ *
+ * Return: 0 on success, %-EINVAL when @args->credit_limit is zero, @args->ops
+ * is NULL, @args->drm is NULL, @args->ops->run_job is NULL, or when
+ * @args->submit_wq or @args->timeout_wq is non-NULL but was not allocated with
+ * %WQ_MEM_WARN_ON_RECLAIM; %-ENOMEM when workqueue allocation fails.
+ *
+ * Context: Process context. May allocate memory and create workqueues.
+ */
+int drm_dep_queue_init(struct drm_dep_queue *q,
+                      const struct drm_dep_queue_init_args *args)
+{
+       if (!args->credit_limit || !args->drm || !args->ops ||
+           !args->ops->run_job)
+               return -EINVAL;
+
+       if (args->submit_wq && !workqueue_is_reclaim_annotated(args->submit_wq))
+               return -EINVAL;
+
+       if (args->timeout_wq &&
+           !workqueue_is_reclaim_annotated(args->timeout_wq))
+               return -EINVAL;
+
+       memset(q, 0, sizeof(*q));
+
+       q->name = args->name;
+       q->drm = args->drm;
+       q->credit.limit = args->credit_limit;
+       q->job.timeout = args->timeout ? args->timeout : MAX_SCHEDULE_TIMEOUT;
+
+       init_rcu_head(&q->rcu);
+       INIT_LIST_HEAD(&q->job.pending);
+       spin_lock_init(&q->job.lock);
+       spsc_queue_init(&q->job.queue);
+
+       mutex_init(&q->sched.lock);
+       if (IS_ENABLED(CONFIG_LOCKDEP)) {
+               fs_reclaim_acquire(GFP_KERNEL);
+               might_lock(&q->sched.lock);
+               fs_reclaim_release(GFP_KERNEL);
+       }
+
+       if (args->submit_wq) {
+               q->sched.submit_wq = args->submit_wq;
+       } else {
+               q->sched.submit_wq = drm_dep_alloc_submit_wq(args->name ?: "drm_dep",
+                                                            args->flags);
+               if (!q->sched.submit_wq)
+                       return -ENOMEM;
+
+               q->sched.flags |= DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ;
+       }
+
+       if (args->timeout_wq) {
+               q->sched.timeout_wq = args->timeout_wq;
+       } else {
+               q->sched.timeout_wq = drm_dep_alloc_timeout_wq(args->name ?: "drm_dep");
+               if (!q->sched.timeout_wq)
+                       goto err_submit_wq;
+
+               q->sched.flags |= DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ;
+       }
+
+       q->sched.flags |= args->flags &
+               ~(DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ |
+                 DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ);
+
+       INIT_DELAYED_WORK(&q->sched.tdr, drm_dep_queue_tdr_work);
+       INIT_WORK(&q->sched.run_job, drm_dep_queue_run_job_work);
+       INIT_WORK(&q->sched.put_job, drm_dep_queue_put_job_work);
+
+       q->fence.context = dma_fence_context_alloc(1);
+
+       kref_init(&q->refcount);
+       q->ops = args->ops;
+       drm_dev_get(q->drm);
+
+       return 0;
+
+err_submit_wq:
+       if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ)
+               destroy_workqueue(q->sched.submit_wq);
+       mutex_destroy(&q->sched.lock);
+
+       return -ENOMEM;
+}
+EXPORT_SYMBOL(drm_dep_queue_init);
+
+#if IS_ENABLED(CONFIG_PROVE_LOCKING)
+/**
+ * drm_dep_queue_push_job_begin() - mark the start of an arm/push critical section
+ * @q: dep queue the job belongs to
+ *
+ * Called at the start of drm_dep_job_arm() and warns if the push context is
+ * already owned by another task, which would indicate concurrent arm/push on
+ * the same queue.
+ *
+ * No-op when CONFIG_PROVE_LOCKING is disabled.
+ *
+ * Context: Process context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job_begin(struct drm_dep_queue *q)
+{
+       WARN_ON(q->job.push.owner);
+       q->job.push.owner = current;
+}
+
+/**
+ * drm_dep_queue_push_job_end() - mark the end of an arm/push critical section
+ * @q: dep queue the job belongs to
+ *
+ * Called at the end of drm_dep_job_push() and warns if the push context is not
+ * owned by the current task, which would indicate a mismatched begin/end pair
+ * or a push from the wrong thread.
+ *
+ * No-op when CONFIG_PROVE_LOCKING is disabled.
+ *
+ * Context: Process context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job_end(struct drm_dep_queue *q)
+{
+       WARN_ON(q->job.push.owner != current);
+       q->job.push.owner = NULL;
+}
+#endif
+
+/**
+ * drm_dep_queue_assert_teardown_invariants() - assert teardown invariants
+ * @q: dep queue being torn down
+ *
+ * Warns if the pending-job list, the SPSC submission queue, or the credit
+ * counter is non-zero when called, or if the queue still has a non-zero
+ * reference count.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_queue_assert_teardown_invariants(struct drm_dep_queue *q)
+{
+       WARN_ON(!list_empty(&q->job.pending));
+       WARN_ON(spsc_queue_count(&q->job.queue));
+       WARN_ON(atomic_read(&q->credit.count));
+       WARN_ON(drm_dep_queue_refcount(q));
+}
+
+/**
+ * drm_dep_queue_release() - final internal cleanup of a dep queue
+ * @q: dep queue to clean up
+ *
+ * Asserts teardown invariants and destroys internal resources allocated by
+ * drm_dep_queue_init() that cannot be torn down earlier in the teardown
+ * sequence.  Currently this destroys @q->sched.lock.
+ *
+ * Drivers that implement &drm_dep_queue_ops.release **must** call this
+ * function after removing @q from any internal bookkeeping (e.g. lookup
+ * tables or lists) but before freeing the memory that contains @q.  When
+ * &drm_dep_queue_ops.release is NULL, drm_dep follows the default teardown
+ * path and calls this function automatically.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_release(struct drm_dep_queue *q)
+{
+       drm_dep_queue_assert_teardown_invariants(q);
+       mutex_destroy(&q->sched.lock);
+}
+EXPORT_SYMBOL(drm_dep_queue_release);
+
+/**
+ * drm_dep_queue_free() - final cleanup of a dep queue
+ * @q: dep queue to free
+ *
+ * Invokes &drm_dep_queue_ops.release if set, in which case the driver is
+ * responsible for calling drm_dep_queue_release() and freeing @q itself.
+ * If &drm_dep_queue_ops.release is NULL, calls drm_dep_queue_release()
+ * and then frees @q with kfree_rcu().
+ *
+ * In either case, releases the drm_dev_get() reference taken at init time
+ * via drm_dev_put(), allowing the owning &drm_device to be unloaded once
+ * all queues have been freed.
+ *
+ * Context: Process context (workqueue), reclaim safe.
+ */
+static void drm_dep_queue_free(struct drm_dep_queue *q)
+{
+       struct drm_device *drm = q->drm;
+
+       if (q->ops->release) {
+               q->ops->release(q);
+       } else {
+               drm_dep_queue_release(q);
+               kfree_rcu(q, rcu);
+       }
+       drm_dev_put(drm);
+}
+
+/**
+ * drm_dep_queue_free_work() - deferred queue teardown worker
+ * @work: free_work item embedded in the dep queue
+ *
+ * Runs on dep_free_wq. Disables all work items synchronously
+ * (preventing re-queue and waiting for in-flight instances),
+ * destroys any owned workqueues, then calls drm_dep_queue_free().
+ * Running on dep_free_wq ensures destroy_workqueue() is never
+ * called from within one of the queue's own workers (deadlock)
+ * and disable_*_sync() cannot deadlock either.
+ *
+ * Context: Process context (workqueue), reclaim safe.
+ */
+static void drm_dep_queue_free_work(struct work_struct *work)
+{
+       struct drm_dep_queue *q =
+               container_of(work, struct drm_dep_queue, free_work);
+
+       drm_dep_queue_assert_teardown_invariants(q);
+
+       disable_delayed_work_sync(&q->sched.tdr);
+       disable_work_sync(&q->sched.run_job);
+       disable_work_sync(&q->sched.put_job);
+
+       if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ)
+               destroy_workqueue(q->sched.timeout_wq);
+
+       if (q->sched.flags & DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ)
+               destroy_workqueue(q->sched.submit_wq);
+
+       drm_dep_queue_free(q);
+}
+
+/**
+ * drm_dep_queue_fini() - tear down a dep queue
+ * @q: dep queue to tear down
+ *
+ * Asserts teardown invariants  and nitiates teardown of @q by queuing the
+ * deferred free work onto tht module-private dep_free_wq workqueue.  The work
+ * item disables any pending TDR and run/put-job work synchronously, destroys
+ * any workqueues that were allocated by drm_dep_queue_init(), and then releases
+ * the queue memory.
+ *
+ * Running teardown from dep_free_wq ensures that destroy_workqueue() is never
+ * called from within one of the queue's own workers (e.g. via
+ * drm_dep_queue_put()), which would deadlock.
+ *
+ * Drivers can wait for all outstanding deferred work to complete by waiting
+ * for the last drm_dev_put() reference on their &drm_device, which is
+ * released as the final step of each queue's teardown.
+ *
+ * Drivers that implement &drm_dep_queue_ops.fini **must** call this
+ * function after removing @q from any device bookkeeping but before freeing the
+ * memory that contains @q.  When &drm_dep_queue_ops.fini is NULL, drm_dep
+ * follows the default teardown path and calls this function automatically.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_fini(struct drm_dep_queue *q)
+{
+       drm_dep_queue_assert_teardown_invariants(q);
+
+       INIT_WORK(&q->free_work, drm_dep_queue_free_work);
+       queue_work(dep_free_wq, &q->free_work);
+}
+EXPORT_SYMBOL(drm_dep_queue_fini);
+
+/**
+ * drm_dep_queue_get() - acquire a reference to a dep queue
+ * @q: dep queue to acquire a reference on, or NULL
+ *
+ * Return: @q with an additional reference held, or NULL if @q is NULL.
+ *
+ * Context: Any context.
+ */
+struct drm_dep_queue *drm_dep_queue_get(struct drm_dep_queue *q)
+{
+       if (q)
+               kref_get(&q->refcount);
+       return q;
+}
+EXPORT_SYMBOL(drm_dep_queue_get);
+
+/**
+ * __drm_dep_queue_release() - kref release callback for a dep queue
+ * @kref: kref embedded in the dep queue
+ *
+ * Calls &drm_dep_queue_ops.fini if set, otherwise calls
+ * drm_dep_queue_fini() to initiate deferred teardown.
+ *
+ * Context: Any context.
+ */
+static void __drm_dep_queue_release(struct kref *kref)
+{
+       struct drm_dep_queue *q =
+               container_of(kref, struct drm_dep_queue, refcount);
+
+       if (q->ops->fini)
+               q->ops->fini(q);
+       else
+               drm_dep_queue_fini(q);
+}
+
+/**
+ * drm_dep_queue_put() - release a reference to a dep queue
+ * @q: dep queue to release a reference on, or NULL
+ *
+ * When the last reference is dropped, calls &drm_dep_queue_ops.fini if set,
+ * otherwise calls drm_dep_queue_fini(). Final memory release is handled by
+ * &drm_dep_queue_ops.release (which must call drm_dep_queue_release()) if set,
+ * or drm_dep_queue_release() followed by kfree_rcu() otherwise.
+ * Does nothing if @q is NULL.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_put(struct drm_dep_queue *q)
+{
+       if (q)
+               kref_put(&q->refcount, __drm_dep_queue_release);
+}
+EXPORT_SYMBOL(drm_dep_queue_put);
+
+/**
+ * drm_dep_queue_stop() - stop a dep queue from processing new jobs
+ * @q: dep queue to stop
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex)
+ * and @q->job.lock (spinlock_irq), making the flag safe to test from finished
+ * fenced signaling context. Then cancels any in-flight run_job and put_job work
+ * items. Once stopped, the bypass path and the submit workqueue will not
+ * dispatch further jobs nor will any jobs be removed from the pending list.
+ * Call drm_dep_queue_start() to resume processing.
+ *
+ * Context: Process context. Waits for in-flight workers to complete.
+ */
+void drm_dep_queue_stop(struct drm_dep_queue *q)
+{
+       scoped_guard(mutex, &q->sched.lock) {
+               scoped_guard(spinlock_irq, &q->job.lock)
+                       drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_STOPPED);
+       }
+       cancel_work_sync(&q->sched.run_job);
+       cancel_work_sync(&q->sched.put_job);
+}
+EXPORT_SYMBOL(drm_dep_queue_stop);
+
+/**
+ * drm_dep_queue_start() - resume a stopped dep queue
+ * @q: dep queue to start
+ *
+ * Clears %DRM_DEP_QUEUE_FLAGS_STOPPED on @q under both @q->sched.lock (mutex)
+ * and @q->job.lock (spinlock_irq), making the flag safe to test from IRQ
+ * context. Then re-queues the run_job and put_job work items so that any jobs
+ * pending since the queue was stopped are processed. Must only be called after
+ * drm_dep_queue_stop().
+ *
+ * Context: Process context.
+ */
+void drm_dep_queue_start(struct drm_dep_queue *q)
+{
+       scoped_guard(mutex, &q->sched.lock) {
+               scoped_guard(spinlock_irq, &q->job.lock)
+                       drm_dep_queue_flags_clear(q, DRM_DEP_QUEUE_FLAGS_STOPPED);
+       }
+       drm_dep_queue_run_job_queue(q);
+       drm_dep_queue_put_job_queue(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_start);
+
+/**
+ * drm_dep_queue_trigger_timeout() - trigger the TDR immediately for
+ *   all pending jobs
+ * @q: dep queue to trigger timeout on
+ *
+ * Sets @q->job.timeout to 1 and arms the TDR delayed work with a one-jiffy
+ * delay, causing it to fire almost immediately without hot-spinning at zero
+ * delay. This is used to force-expire any pendind jobs on the queue, for
+ * example when the device is being torn down or has encountered an
+ * unrecoverable error.
+ *
+ * It is suggested that when this function is used, the first timedout_job call
+ * causes the driver to kick the queue off the hardware and signal all pending
+ * job fences. Subsequent calls continue to signal all pending job fences.
+ *
+ * Has no effect if the pending list is empty.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_trigger_timeout(struct drm_dep_queue *q)
+{
+       guard(spinlock_irqsave)(&q->job.lock);
+       q->job.timeout = 1;
+       drm_queue_start_timeout(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_trigger_timeout);
+
+/**
+ * drm_dep_queue_cancel_tdr_sync() - cancel any pending TDR and wait
+ *   for it to finish
+ * @q: dep queue whose TDR to cancel
+ *
+ * Cancels the TDR delayed work item if it has not yet started, and waits for
+ * it to complete if it is already running.  After this call returns, the TDR
+ * worker is guaranteed not to be executing and will not fire again until
+ * explicitly rearmed (e.g. via drm_dep_queue_resume_timeout() or by a new
+ * job being submitted).
+ *
+ * Useful during error recovery or queue teardown when the caller needs to
+ * know that no timeout handling races with its own reset logic.
+ *
+ * Context: Process context. May sleep waiting for the TDR worker to finish.
+ */
+void drm_dep_queue_cancel_tdr_sync(struct drm_dep_queue *q)
+{
+       cancel_delayed_work_sync(&q->sched.tdr);
+}
+EXPORT_SYMBOL(drm_dep_queue_cancel_tdr_sync);
+
+/**
+ * drm_dep_queue_resume_timeout() - restart the TDR timer with the
+ *   configured timeout
+ * @q: dep queue to resume the timeout for
+ *
+ * Restarts the TDR delayed work using @q->job.timeout. Called after device
+ * recovery to give pending jobs a fresh full timeout window. Has no effect
+ * if the pending list is empty.
+ *
+ * Context: Any context.
+ */
+void drm_dep_queue_resume_timeout(struct drm_dep_queue *q)
+{
+       drm_queue_start_timeout_unlocked(q);
+}
+EXPORT_SYMBOL(drm_dep_queue_resume_timeout);
+
+/**
+ * drm_dep_queue_is_stopped() - check whether a dep queue is stopped
+ * @q: dep queue to check
+ *
+ * Return: true if %DRM_DEP_QUEUE_FLAGS_STOPPED is set on @q, false otherwise.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_is_stopped(struct drm_dep_queue *q)
+{
+       return !!(q->sched.flags & DRM_DEP_QUEUE_FLAGS_STOPPED);
+}
+EXPORT_SYMBOL(drm_dep_queue_is_stopped);
+
+/**
+ * drm_dep_queue_kill() - kill a dep queue and flush all pending jobs
+ * @q: dep queue to kill
+ *
+ * Sets %DRM_DEP_QUEUE_FLAGS_KILLED on @q under @q->sched.lock.  If a
+ * dependency fence is currently being waited on, its callback is removed and
+ * the run-job worker is kicked immediately so that the blocked job drains
+ * without waiting.
+ *
+ * Once killed, drm_dep_queue_job_dependency() returns NULL for all jobs,
+ * bypassing dependency waits so that every queued job drains through
+ * &drm_dep_queue_ops.run_job without blocking.
+ *
+ * The &drm_dep_queue_ops.run_job callback is guaranteed to be called for every
+ * job that was pushed before or after drm_dep_queue_kill(), even during queue
+ * teardown.  Drivers should use this guarantee to perform any necessary
+ * bookkeeping cleanup without executing the actual backend operation when the
+ * queue is killed.
+ *
+ * Unlike drm_dep_queue_stop(), killing is one-way: there is no corresponding
+ * start function.
+ *
+ * **Driver safety requirement**
+ *
+ * drm_dep_queue_kill() must only be called once the driver can guarantee that
+ * no job in the queue will touch memory associated with any of its fences
+ * (i.e., the queue has been removed from the device and will never be put back
+ * on).
+ *
+ * Context: Process context.
+ */
+void drm_dep_queue_kill(struct drm_dep_queue *q)
+{
+       scoped_guard(mutex, &q->sched.lock) {
+               struct dma_fence *fence;
+
+               drm_dep_queue_flags_set(q, DRM_DEP_QUEUE_FLAGS_KILLED);
+
+               /*
+                * Holding &q->sched.lock guarantees that the run-job work item
+                * cannot drop its reference to q->dep.fence concurrently, so
+                * reading q->dep.fence here is safe.
+                */
+               fence = READ_ONCE(q->dep.fence);
+               if (fence && dma_fence_remove_callback(fence, &q->dep.cb))
+                       drm_dep_queue_remove_dependency(q, fence);
+       }
+}
+EXPORT_SYMBOL(drm_dep_queue_kill);
+
+/**
+ * drm_dep_queue_submit_wq() - retrieve the submit workqueue of a dep queue
+ * @q: dep queue whose workqueue to retrieve
+ *
+ * Drivers may use this to queue their own work items alongside the queue's
+ * internal run-job and put-job workers — for example to process incoming
+ * messages in the same serialisation domain.
+ *
+ * Prefer drm_dep_queue_work_enqueue() when the only need is to enqueue a
+ * work item, as it additionally checks the stopped state.  Use this accessor
+ * when the workqueue itself is required (e.g. for alloc_ordered_workqueue
+ * replacement or drain_workqueue calls).
+ *
+ * Context: Any context.
+ * Return: the &workqueue_struct used by @q for job submission.
+ */
+struct workqueue_struct *drm_dep_queue_submit_wq(struct drm_dep_queue *q)
+{
+       return q->sched.submit_wq;
+}
+EXPORT_SYMBOL(drm_dep_queue_submit_wq);
+
+/**
+ * drm_dep_queue_timeout_wq() - retrieve the timeout workqueue of a dep queue
+ * @q: dep queue whose workqueue to retrieve
+ *
+ * Returns the workqueue used by @q to run TDR (timeout detection and recovery)
+ * work.  Drivers may use this to queue their own timeout-domain work items, or
+ * to call drain_workqueue() when tearing down and needing to ensure all pending
+ * timeout callbacks have completed before proceeding.
+ *
+ * Context: Any context.
+ * Return: the &workqueue_struct used by @q for TDR work.
+ */
+struct workqueue_struct *drm_dep_queue_timeout_wq(struct drm_dep_queue *q)
+{
+       return q->sched.timeout_wq;
+}
+EXPORT_SYMBOL(drm_dep_queue_timeout_wq);
+
+/**
+ * drm_dep_queue_work_enqueue() - queue work on the dep queue's submit workqueue
+ * @q: dep queue to enqueue work on
+ * @work: work item to enqueue
+ *
+ * Queues @work on @q->sched.submit_wq if the queue is not stopped.  This
+ * allows drivers to schedule custom work items that run serialised with the
+ * queue's own run-job and put-job workers.
+ *
+ * Return: true if the work was queued, false if the queue is stopped or the
+ * work item was already pending.
+ *
+ * Context: Any context.
+ */
+bool drm_dep_queue_work_enqueue(struct drm_dep_queue *q,
+                               struct work_struct *work)
+{
+       if (drm_dep_queue_is_stopped(q))
+               return false;
+
+       return queue_work(q->sched.submit_wq, work);
+}
+EXPORT_SYMBOL(drm_dep_queue_work_enqueue);
+
+/**
+ * drm_dep_queue_can_job_bypass() - test whether a job can skip the SPSC queue
+ * @q: dep queue
+ * @job: job to test
+ *
+ * A job may bypass the submit workqueue and run inline on the calling thread
+ * if all of the following hold:
+ *
+ *  - %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED is set on the queue
+ *  - the queue is not stopped
+ *  - the SPSC submission queue is empty (no other jobs waiting)
+ *  - the queue has enough credits for @job
+ *  - @job has no unresolved dependency fences
+ *
+ * Must be called under @q->sched.lock.
+ *
+ * Context: Process context. Must hold @q->sched.lock (a mutex).
+ * Return: true if the job may be run inline, false otherwise.
+ */
+bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q,
+                                 struct drm_dep_job *job)
+{
+       lockdep_assert_held(&q->sched.lock);
+
+       return q->sched.flags & DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED &&
+               !drm_dep_queue_is_stopped(q) &&
+               !spsc_queue_count(&q->job.queue) &&
+               drm_dep_queue_has_credits(q, job) &&
+               xa_empty(&job->dependencies);
+}
+
+/**
+ * drm_dep_job_done() - mark a job as complete
+ * @job: the job that finished
+ * @result: error code to propagate, or 0 for success
+ *
+ * Subtracts @job->credits from the queue credit counter, then signals the
+ * job's dep fence with @result.
+ *
+ * When %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE is set (IRQ-safe path), a
+ * temporary extra reference is taken on @job before signalling the fence.
+ * This prevents a concurrent put-job worker — which may be woken by timeouts or
+ * queue starting — from freeing the job while this function still holds a
+ * pointer to it.  The extra reference is released at the end of the function.
+ *
+ * After signalling, the IRQ-safe path removes the job from the pending list
+ * under @q->job.lock, provided the queue is not stopped.  Removal is skipped
+ * when the queue is stopped so that drm_dep_queue_for_each_pending_job() can
+ * iterate the list without racing with the completion path.  On successful
+ * removal, kicks the run-job worker so the next queued job can be dispatched
+ * immediately, then drops the job reference.  If the job was already removed
+ * by TDR, or removal was skipped because the queue is stopped, kicks the
+ * put-job worker instead to allow the deferred put to complete.
+ *
+ * Context: Any context.
+ */
+static void drm_dep_job_done(struct drm_dep_job *job, int result)
+{
+       struct drm_dep_queue *q = job->q;
+       bool irq_safe = drm_dep_queue_is_job_put_irq_safe(q), removed = false;
+
+       /*
+        * Local ref to ensure the put worker—which may be woken by external
+        * forces (TDR, driver-side queue starting)—doesn't free the job behind
+        * this function's back after drm_dep_fence_done() while it is still on
+        * the pending list.
+        */
+       if (irq_safe)
+               drm_dep_job_get(job);
+
+       atomic_sub(job->credits, &q->credit.count);
+       drm_dep_fence_done(job->dfence, result);
+
+       /* Only safe to touch job after fence signal if we have a local ref. */
+
+       if (irq_safe) {
+               scoped_guard(spinlock_irqsave, &q->job.lock) {
+                       removed = !list_empty(&job->pending_link) &&
+                               !drm_dep_queue_is_stopped(q);
+
+                       /* Guard against TDR operating on job */
+                       if (removed)
+                               drm_dep_queue_remove_job(q, job);
+               }
+       }
+
+       if (removed) {
+               drm_dep_queue_run_job_queue(q);
+               drm_dep_job_put(job);
+       } else {
+               drm_dep_queue_put_job_queue(q);
+       }
+
+       if (irq_safe)
+               drm_dep_job_put(job);
+}
+
+/**
+ * drm_dep_job_done_cb() - dma_fence callback to complete a job
+ * @f: the hardware fence that signalled
+ * @cb: fence callback embedded in the dep job
+ *
+ * Extracts the job from @cb and calls drm_dep_job_done() with
+ * @f->error as the result.
+ *
+ * Context: Any context, but with IRQ disabled. May not sleep.
+ */
+static void drm_dep_job_done_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+       struct drm_dep_job *job = container_of(cb, struct drm_dep_job, cb);
+
+       drm_dep_job_done(job, f->error);
+}
+
+/**
+ * drm_dep_queue_run_job() - submit a job to hardware and set up
+ *   completion tracking
+ * @q: dep queue
+ * @job: job to run
+ *
+ * Accounts @job->credits against the queue, appends the job to the pending
+ * list, then calls @q->ops->run_job(). The TDR timer is started only when
+ * @job is the first entry on the pending list; subsequent jobs added while
+ * a TDR is already in flight do not reset the timer (which would otherwise
+ * extend the deadline for the already-running head job). Stores the returned
+ * hardware fence as the parent of the job's dep fence, then installs
+ * drm_dep_job_done_cb() on it. If the hardware fence is already signalled
+ * (%-ENOENT from dma_fence_add_callback()) or run_job() returns NULL/error,
+ * the job is completed immediately. Must be called under @q->sched.lock.
+ *
+ * Context: Process context. Must hold @q->sched.lock (a mutex). DMA fence
+ * signaling path.
+ */
+void drm_dep_queue_run_job(struct drm_dep_queue *q, struct drm_dep_job *job)
+{
+       struct dma_fence *fence;
+       int r;
+
+       lockdep_assert_held(&q->sched.lock);
+
+       drm_dep_job_get(job);
+       atomic_add(job->credits, &q->credit.count);
+
+       scoped_guard(spinlock_irq, &q->job.lock) {
+               bool first = list_empty(&q->job.pending);
+
+               list_add_tail(&job->pending_link, &q->job.pending);
+               if (first)
+                       drm_queue_start_timeout(q);
+       }
+
+       fence = q->ops->run_job(job);
+       drm_dep_fence_set_parent(job->dfence, fence);
+
+       if (!IS_ERR_OR_NULL(fence)) {
+               r = dma_fence_add_callback(fence, &job->cb,
+                                          drm_dep_job_done_cb);
+               if (r == -ENOENT)
+                       drm_dep_job_done(job, fence->error);
+               else if (r)
+                       drm_err(q->drm, "fence add callback failed (%d)\n", r);
+               dma_fence_put(fence);
+       } else {
+               drm_dep_job_done(job, IS_ERR(fence) ? PTR_ERR(fence) : 0);
+       }
+
+       /*
+        * Drop all input dependency fences now, in process context, before the
+        * final job put. Once the job is on the pending list its last reference
+        * may be dropped from a dma_fence callback (IRQ context), where calling
+        * xa_destroy() would be unsafe.
+        */
+       drm_dep_job_drop_dependencies(job);
+       drm_dep_job_put(job);
+}
+
+/**
+ * drm_dep_queue_push_job() - enqueue a job on the SPSC submission queue
+ * @q: dep queue
+ * @job: job to push
+ *
+ * Pushes @job onto the SPSC queue. If the queue was previously empty
+ * (i.e. this is the first pending job), kicks the run_job worker so it
+ * processes the job promptly without waiting for the next wakeup.
+ * May be called with or without @q->sched.lock held.
+ *
+ * Context: Any context. DMA fence signaling path.
+ */
+void drm_dep_queue_push_job(struct drm_dep_queue *q, struct drm_dep_job *job)
+{
+       /*
+        * spsc_queue_push() returns true if the queue was previously empty,
+        * i.e. this is the first pending job. Kick the run_job worker so it
+        * picks it up without waiting for the next wakeup.
+        */
+       if (spsc_queue_push(&q->job.queue, &job->queue_node))
+               drm_dep_queue_run_job_queue(q);
+}
+
+/**
+ * drm_dep_init() - module initialiser
+ *
+ * Allocates the module-private dep_free_wq unbound workqueue used for
+ * deferred queue teardown.
+ *
+ * Return: 0 on success, %-ENOMEM if workqueue allocation fails.
+ */
+static int __init drm_dep_init(void)
+{
+       dep_free_wq = alloc_workqueue("drm_dep_free", WQ_UNBOUND, 0);
+       if (!dep_free_wq)
+               return -ENOMEM;
+
+       return 0;
+}
+
+/**
+ * drm_dep_exit() - module exit
+ *
+ * Destroys the module-private dep_free_wq workqueue.
+ */
+static void __exit drm_dep_exit(void)
+{
+       destroy_workqueue(dep_free_wq);
+       dep_free_wq = NULL;
+}
+
+module_init(drm_dep_init);
+module_exit(drm_dep_exit);
+
+MODULE_DESCRIPTION("DRM dependency queue");
+MODULE_LICENSE("Dual MIT/GPL");
drivers/gpu/drm/dep/drm_dep_queue.h new file +31 -0
new file mode 100644
index 000000000000..e5c217a3fab5
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_QUEUE_H_
+#define _DRM_DEP_QUEUE_H_
+
+#include <linux/types.h>
+
+struct drm_dep_job;
+struct drm_dep_queue;
+
+bool drm_dep_queue_can_job_bypass(struct drm_dep_queue *q,
+                                 struct drm_dep_job *job);
+void drm_dep_queue_run_job(struct drm_dep_queue *q, struct drm_dep_job *job);
+void drm_dep_queue_push_job(struct drm_dep_queue *q, struct drm_dep_job *job);
+
+#if IS_ENABLED(CONFIG_PROVE_LOCKING)
+void drm_dep_queue_push_job_begin(struct drm_dep_queue *q);
+void drm_dep_queue_push_job_end(struct drm_dep_queue *q);
+#else
+static inline void drm_dep_queue_push_job_begin(struct drm_dep_queue *q)
+{
+}
+static inline void drm_dep_queue_push_job_end(struct drm_dep_queue *q)
+{
+}
+#endif
+
+#endif /* _DRM_DEP_QUEUE_H_ */
include/drm/drm_dep.h new file +597 -1
new file mode 100644
index 000000000000..615926584506
@@ -0,0 +1,597 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright 2015 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Copyright © 2026 Intel Corporation
+ */
+
+#ifndef _DRM_DEP_H_
+#define _DRM_DEP_H_
+
+#include <drm/spsc_queue.h>
+#include <linux/dma-fence.h>
+#include <linux/xarray.h>
+#include <linux/workqueue.h>
+
+enum dma_resv_usage;
+struct dma_resv;
+struct drm_dep_fence;
+struct drm_dep_job;
+struct drm_dep_queue;
+struct drm_file;
+struct drm_gem_object;
+
+/**
+ * enum drm_dep_timedout_stat - return value of &drm_dep_queue_ops.timedout_job
+ * @DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED: driver signaled the job's finished
+ *   fence during reset; drm_dep may safely drop its reference to the job.
+ * @DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB: timeout was a false alarm; reinsert the
+ *   job at the head of the pending list so it can complete normally.
+ */
+enum drm_dep_timedout_stat {
+       DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED,
+       DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB,
+};
+
+/**
+ * struct drm_dep_queue_ops - driver callbacks for a dep queue
+ */
+struct drm_dep_queue_ops {
+       /**
+        * @run_job: submit the job to hardware. Returns the hardware completion
+        * fence (with a reference held for the scheduler), or NULL/ERR_PTR on
+        * synchronous completion or error.
+        */
+       struct dma_fence *(*run_job)(struct drm_dep_job *job);
+
+       /**
+        * @timedout_job: called when the TDR fires for the head job. Must stop
+        * the hardware, then return %DRM_DEP_TIMEDOUT_STAT_JOB_SIGNALED if the
+        * job's fence was signalled during reset, or
+        * %DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB if the timeout was spurious or
+        * signalling was otherwise delayed, and the job should be re-inserted
+        * at the head of the pending list. Any other value triggers a WARN.
+        */
+       enum drm_dep_timedout_stat (*timedout_job)(struct drm_dep_job *job);
+
+       /**
+        * @release: called when the last kref on the queue is dropped and
+        * drm_dep_queue_fini() has completed.  The driver is responsible for
+        * removing @q from any internal bookkeeping, calling
+        * drm_dep_queue_release(), and then freeing the memory containing @q
+        * (e.g. via kfree_rcu() using @q->rcu).  If NULL, drm_dep calls
+        * drm_dep_queue_release() and frees @q automatically via kfree_rcu().
+        * Use this when the queue is embedded in a larger structure.
+        */
+       void (*release)(struct drm_dep_queue *q);
+
+       /**
+        * @fini: if set, called instead of drm_dep_queue_fini() when the last
+        * kref is dropped. The driver is responsible for calling
+        * drm_dep_queue_fini() itself after it is done with the queue. Use this
+        * when additional teardown logic must run before fini (e.g., cleanup
+        * firmware resources associated with the queue).
+        */
+       void (*fini)(struct drm_dep_queue *q);
+};
+
+/**
+ * enum drm_dep_queue_flags - flags for &drm_dep_queue and
+ *   &drm_dep_queue_init_args
+ *
+ * Flags are divided into three categories:
+ *
+ * - **Private static**: set internally at init time and never changed.
+ *   Drivers must not read or write these.
+ *   %DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ,
+ *   %DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ.
+ *
+ * - **Public dynamic**: toggled at runtime by drivers via accessors.
+ *   Any modification must be performed under &drm_dep_queue.sched.lock.
+ *   Accessor functions provide unstable reads.
+ *   %DRM_DEP_QUEUE_FLAGS_STOPPED,
+ *   %DRM_DEP_QUEUE_FLAGS_KILLED.
+ *
+ * - **Public static**: supplied by the driver in
+ *   &drm_dep_queue_init_args.flags at queue creation time and not modified
+ *   thereafter.
+ *   %DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED,
+ *   %DRM_DEP_QUEUE_FLAGS_HIGHPRI,
+ *   %DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE.
+ *
+ * @DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ: (private, static) submit workqueue was
+ *   allocated by drm_dep_queue_init() and will be destroyed by
+ *   drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ: (private, static) timeout workqueue
+ *   was allocated by drm_dep_queue_init() and will be destroyed by
+ *   drm_dep_queue_fini().
+ * @DRM_DEP_QUEUE_FLAGS_STOPPED: (public, dynamic) the queue is stopped and
+ *   will not dispatch new jobs or remove jobs from the pending list, dropping
+ *   the drm_dep-owned reference. Set by drm_dep_queue_stop(), cleared by
+ *   drm_dep_queue_start().
+ * @DRM_DEP_QUEUE_FLAGS_KILLED: (public, dynamic) the queue has been killed
+ *   via drm_dep_queue_kill(). Any active dependency wait is cancelled
+ *   immediately.  Jobs continue to flow through run_job for bookkeeping
+ *   cleanup, but dependency waiting is skipped so that queued work drains
+ *   as quickly as possible.
+ * @DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED: (public, static) the queue supports
+ *   the bypass path where eligible jobs skip the SPSC queue and run inline.
+ * @DRM_DEP_QUEUE_FLAGS_HIGHPRI: (public, static) the submit workqueue owned
+ *   by the queue is created with %WQ_HIGHPRI, causing run-job and put-job
+ *   workers to execute at elevated priority. Only privileged clients (e.g.
+ *   drivers managing time-critical or real-time GPU contexts) should request
+ *   this flag; granting it to unprivileged userspace would allow priority
+ *   inversion attacks.
+ *   @drm_dep_queue_init_args.submit_wq is provided.
+ * @DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE: (public, static) when set,
+ *   drm_dep_job_done() may be called from hardirq context (e.g. from a
+ *   hardware-signalled dma_fence callback). drm_dep_job_done() will directly
+ *   dequeue the job and call drm_dep_job_put() without deferring to a
+ *   workqueue. The driver's &drm_dep_job_ops.release callback must therefore
+ *   be safe to invoke from IRQ context.
+ */
+enum drm_dep_queue_flags {
+       DRM_DEP_QUEUE_FLAGS_OWN_SUBMIT_WQ       = BIT(0),
+       DRM_DEP_QUEUE_FLAGS_OWN_TIMEDOUT_WQ     = BIT(1),
+       DRM_DEP_QUEUE_FLAGS_STOPPED             = BIT(2),
+       DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED    = BIT(3),
+       DRM_DEP_QUEUE_FLAGS_HIGHPRI             = BIT(4),
+       DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE    = BIT(5),
+       DRM_DEP_QUEUE_FLAGS_KILLED              = BIT(6),
+};
+
+/**
+ * struct drm_dep_queue - a dependency-tracked GPU submission queue
+ *
+ * Combines the role of &drm_gpu_scheduler and &drm_sched_entity into a single
+ * object.  Each queue owns a submit workqueue (or borrows one), a timeout
+ * workqueue, an SPSC submission queue, and a pending-job list used for TDR.
+ *
+ * Initialise with drm_dep_queue_init(), tear down with drm_dep_queue_fini().
+ * Reference counted via drm_dep_queue_get() / drm_dep_queue_put().
+ *
+ * All fields are **opaque to drivers**.  Do not read or write any field
+ * directly; use the provided helper functions instead.  The sole exception
+ * is @rcu, which drivers may pass to kfree_rcu() when the queue is embedded
+ * inside a larger driver-managed structure and the &drm_dep_queue_ops.release
+ * vfunc performs an RCU-deferred free.
+ */
+struct drm_dep_queue {
+       /** @ops: driver callbacks, set at init time. */
+       const struct drm_dep_queue_ops *ops;
+       /** @name: human-readable name used for workqueue and fence naming. */
+       const char *name;
[… more lines in the raw message]

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

2.34.1

Part of the diff is not shown here. View the raw message.

author maintainer
#50 lore raw

RFC PATCH 3/12 drm/xe: Use WQ_MEM_WARN_ON_RECLAIM on all workqueues in the reclaim path

In reply to Matthew Brost

Mark all workqueues in the reclaim path with WQ_MEM_WARN_ON_RECLAIM to ensure that reclaim rules are properly enforced via lockdep on these workqueues.

---
 drivers/gpu/drm/xe/xe_device.c    | 3 ++-
 drivers/gpu/drm/xe/xe_ggtt.c      | 3 ++-
 drivers/gpu/drm/xe/xe_gt.c        | 3 ++-
 drivers/gpu/drm/xe/xe_guc_ct.c    | 3 ++-
 drivers/gpu/drm/xe/xe_tlb_inval.c | 3 ++-
 5 files changed, 10 insertions(+), 5 deletions(-)
drivers/gpu/drm/xe/xe_device.c +2 -1
index e77a3a3db73d..00b966744d25 100644
@@ -507,7 +507,8 @@ struct xe_device *xe_device_create(struct pci_dev *pdev,
 		return ERR_PTR(err);
 
 	xe->preempt_fence_wq = alloc_ordered_workqueue("xe-preempt-fence-wq",
-						       WQ_MEM_RECLAIM);
+						       WQ_MEM_RECLAIM |
+						       WQ_MEM_WARN_ON_RECLAIM);
 	xe->ordered_wq = alloc_ordered_workqueue("xe-ordered-wq", 0);
 	xe->unordered_wq = alloc_workqueue("xe-unordered-wq", WQ_PERCPU, 0);
 	xe->destroy_wq = alloc_workqueue("xe-destroy-wq", WQ_PERCPU, 0);
drivers/gpu/drm/xe/xe_ggtt.c +2 -1
index 21071b64b09d..017f679ba13f 100644
@@ -434,7 +434,8 @@ int xe_ggtt_init_early(struct xe_ggtt *ggtt)
 	else
 		ggtt->pt_ops = &xelp_pt_ops;
 
-	ggtt->wq = alloc_workqueue("xe-ggtt-wq", WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+	ggtt->wq = alloc_workqueue("xe-ggtt-wq", WQ_MEM_RECLAIM | WQ_PERCPU |
+				   WQ_MEM_WARN_ON_RECLAIM, 0);
 	if (!ggtt->wq)
 		return -ENOMEM;
 
drivers/gpu/drm/xe/xe_gt.c +2 -1
index bae895fa066a..79f4128fe325 100644
@@ -84,7 +84,8 @@ struct xe_gt *xe_gt_alloc(struct xe_tile *tile)
 		ordered_wq = tile->primary_gt->ordered_wq;
 	else
 		ordered_wq = drmm_alloc_ordered_workqueue(drm, "gt-ordered-wq",
-							  WQ_MEM_RECLAIM);
+							  WQ_MEM_RECLAIM |
+							  WQ_MEM_WARN_ON_RECLAIM);
 	if (IS_ERR(ordered_wq))
 		return ERR_CAST(ordered_wq);
 
drivers/gpu/drm/xe/xe_guc_ct.c +2 -1
index 3b1c03743f83..7ec34ac00f59 100644
@@ -325,7 +325,8 @@ int xe_guc_ct_init_noalloc(struct xe_guc_ct *ct)
 
 	primelockdep(ct);
 
-	ct->g2h_wq = alloc_ordered_workqueue("xe-g2h-wq", WQ_MEM_RECLAIM);
+	ct->g2h_wq = alloc_ordered_workqueue("xe-g2h-wq", WQ_MEM_RECLAIM |
+					     WQ_MEM_WARN_ON_RECLAIM);
 	if (!ct->g2h_wq)
 		return -ENOMEM;
 
drivers/gpu/drm/xe/xe_tlb_inval.c +2 -1
index 10dcd4abb00f..10dff2d8cf45 100644
@@ -154,7 +154,8 @@ int xe_gt_tlb_inval_init_early(struct xe_gt *gt)
 
 	tlb_inval->job_wq = drmm_alloc_ordered_workqueue(&xe->drm,
 							 "gt-tbl-inval-job-wq",
-							 WQ_MEM_RECLAIM);
+							 WQ_MEM_RECLAIM |
+							 WQ_MEM_WARN_ON_RECLAIM);
 	if (IS_ERR(tlb_inval->job_wq))
 		return PTR_ERR(tlb_inval->job_wq);
 
Signature · 1 line
2.34.1
author maintainer
#51 lore raw

RFC PATCH 4/12 drm/xe: Issue GGTT invalidation under lock in ggtt_node_remove

In reply to Matthew Brost

Async work (e.g., GuC queue teardowns) can call ggtt_node_remove, so the operation must be performed under the GGTT lock to ensure the GGTT online check remains stable. GGTT insertion and removal are heavyweight operations (e.g., queue create/destroy), so the additional serialization cost is negligible compared to ensuring correctness.

Fixes 4f3a998a173b ("drm/xe: Open-code GGTT MMIO access protection")
---
 drivers/gpu/drm/xe/xe_ggtt.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/gpu/drm/xe/xe_ggtt.c +2 -7
index 017f679ba13f..353b324f7301 100644
@@ -482,15 +482,10 @@ static void ggtt_node_remove(struct xe_ggtt_node *node)
 		xe_ggtt_clear(ggtt, xe_ggtt_node_addr(node), xe_ggtt_node_size(node));
 	drm_mm_remove_node(&node->base);
 	node->base.size = 0;
-	mutex_unlock(&ggtt->lock);
-
-	if (!bound)
-		goto free_node;
-
-	if (node->invalidate_on_remove)
+	if (bound && node->invalidate_on_remove)
 		xe_ggtt_invalidate(ggtt);
+	mutex_unlock(&ggtt->lock);
 
-free_node:
 	ggtt_node_fini(node);
 }
 
Signature · 1 line
2.34.1
Reviewed-by on v1 4/12
#52 lore raw

RE: [RFC PATCH 04/12] drm/xe: Issue GGTT invalidation under lock in ggtt_node_remove

Quoted 11 lines

-----Original Message-----
From: Intel-xe <[email protected]> On Behalf Of Matthew Brost
Sent: Monday, March 16, 2026 10:03 AM
To: [email protected]
Cc: [email protected]
Subject: [RFC PATCH 04/12] drm/xe: Issue GGTT invalidation under lock in ggtt_node_remove

Quoted 8 lines

Async work (e.g., GuC queue teardowns) can call ggtt_node_remove, so the operation must be performed under the GGTT lock to ensure the GGTT online check remains stable. GGTT insertion and removal are heavyweight operations (e.g., queue create/destroy), so the additional serialization cost is negligible compared to ensuring correctness.

Fixes 4f3a998a173b ("drm/xe: Open-code GGTT MMIO access protection")

HI, please take care of nitpick here "Signed-off-by" before merging.

Quoted 13 lines
---
 drivers/gpu/drm/xe/xe_ggtt.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
drivers/gpu/drm/xe/xe_ggtt.c +0 -0
index 017f679ba13f..353b324f7301 100644
@@ -482,15 +482,10 @@ static void ggtt_node_remove(struct xe_ggtt_node
 *node)
 		xe_ggtt_clear(ggtt, xe_ggtt_node_addr(node),
 xe_ggtt_node_size(node));
 	drm_mm_remove_node(&node->base);

drivers/gpu/drm/xe/xe_ggtt.c static void ggtt_node_remove(struct xe_ggtt_node · line 489

Quoted 10 lines
drivers/gpu/drm/xe/xe_ggtt.c +2 -6
 	node->base.size = 0;
-	mutex_unlock(&ggtt->lock);
-
-	if (!bound)
-		goto free_node;
-
-	if (node->invalidate_on_remove)
+	if (bound && node->invalidate_on_remove)
 		xe_ggtt_invalidate(ggtt);
+	mutex_unlock(&ggtt->lock);

Looks good to me,

Dnyaneshwar

Quoted 7 lines
drivers/gpu/drm/xe/xe_ggtt.c +0 -2
-free_node:
 	ggtt_node_fini(node);
 }
 
--

2.34.1

author maintainer
#53 lore raw

RFC PATCH 5/12 drm/xe: Return fence from xe_sched_job_arm and adjust job references

In reply to Matthew Brost

Update xe_sched_job_arm to return a fence that the caller can use to install dma-resv fences, signal syncobjs, or wait for job completion. The caller is responsible for dropping the fence reference when it is finished with it.

Update xe_sched_job_push to take a reference to the job owned by the scheduler. The caller is now responsible for dropping the initial job creation reference.

These semantics better align with DRM dep semantics.

---
 drivers/gpu/drm/xe/tests/xe_migrate.c | 10 ++++++----
 drivers/gpu/drm/xe/xe_exec.c          | 12 ++++++------
 drivers/gpu/drm/xe/xe_gsc.c           |  5 +++--
 drivers/gpu/drm/xe/xe_gsc_submit.c    |  5 +++--
 drivers/gpu/drm/xe/xe_gt.c            |  5 +++--
 drivers/gpu/drm/xe/xe_migrate.c       | 25 +++++++++++++++----------
 drivers/gpu/drm/xe/xe_oa.c            |  5 +++--
 drivers/gpu/drm/xe/xe_pxp_submit.c    | 10 ++++++----
 drivers/gpu/drm/xe/xe_sched_job.c     |  5 +++--
 drivers/gpu/drm/xe/xe_sched_job.h     |  2 +-
 10 files changed, 49 insertions(+), 35 deletions(-)
drivers/gpu/drm/xe/tests/xe_migrate.c +6 -4
index 34e2f0f4631f..257b373fcaec 100644
@@ -50,9 +50,10 @@ static int run_sanity_job(struct xe_migrate *m, struct xe_device *xe,
 		return PTR_ERR(job);
 	}
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	if (sanity_fence_failed(xe, fence, str, test))
 		return -ETIMEDOUT;
@@ -463,10 +464,11 @@ static struct dma_fence *blt_copy(struct xe_tile *tile,
 		xe_sched_job_add_migrate_flush(job, flush_flags);
 
 		mutex_lock(&m->job_mutex);
-		xe_sched_job_arm(job);
 		dma_fence_put(fence);
-		fence = dma_fence_get(&job->drm.s_fence->finished);
+		fence = xe_sched_job_arm(job);
+		dma_fence_get(fence);
 		xe_sched_job_push(job);
+		xe_sched_job_put(job);
 
 		dma_fence_put(m->fence);
 		m->fence = dma_fence_get(fence);
drivers/gpu/drm/xe/xe_exec.c +6 -6
index e05dabfcd43c..0c0ddf500981 100644
@@ -124,6 +124,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 	struct xe_validation_ctx ctx;
 	struct xe_sched_job *job;
 	struct xe_vm *vm;
+	struct dma_fence *fence;
 	bool write_locked;
 	int err = 0;
 	struct xe_hw_engine_group *group;
@@ -320,19 +321,19 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 	 * Point of no return, if we error after this point just set an error on
 	 * the job and let the DRM scheduler / backend clean up the job.
 	 */
-	xe_sched_job_arm(job);
+	fence = xe_sched_job_arm(job);
 	if (!xe_vm_in_lr_mode(vm))
-		drm_gpuvm_resv_add_fence(&vm->gpuvm, exec, &job->drm.s_fence->finished,
+		drm_gpuvm_resv_add_fence(&vm->gpuvm, exec, fence,
 					 DMA_RESV_USAGE_BOOKKEEP,
 					 DMA_RESV_USAGE_BOOKKEEP);
 
 	for (i = 0; i < num_syncs; i++) {
-		xe_sync_entry_signal(&syncs[i], &job->drm.s_fence->finished);
+		xe_sync_entry_signal(&syncs[i], fence);
 		xe_sched_job_init_user_fence(job, &syncs[i]);
 	}
 
 	if (!xe_vm_in_lr_mode(vm))
-		xe_exec_queue_last_fence_set(q, vm, &job->drm.s_fence->finished);
+		xe_exec_queue_last_fence_set(q, vm, fence);
 	xe_sched_job_push(job);
 	xe_vm_reactivate_rebind(vm);
 
@@ -349,8 +350,7 @@ int xe_exec_ioctl(struct drm_device *dev, void *data, struct drm_file *file)
 	if (!xe_vm_in_lr_mode(vm))
 		xe_svm_notifier_unlock(vm);
 err_put_job:
-	if (err)
-		xe_sched_job_put(job);
+	xe_sched_job_put(job);
 err_exec:
 	if (!xe_vm_in_lr_mode(vm))
 		xe_validation_ctx_fini(&ctx);
drivers/gpu/drm/xe/xe_gsc.c +3 -2
index e5c234f3d795..10168c27fc22 100644
@@ -92,9 +92,10 @@ static int emit_gsc_upload(struct xe_gsc *gsc)
 		return PTR_ERR(job);
 	}
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	timeout = dma_fence_wait_timeout(fence, false, HZ);
 	dma_fence_put(fence);
drivers/gpu/drm/xe/xe_gsc_submit.c +3 -2
index 08082b596501..2d506fb66017 100644
@@ -202,9 +202,10 @@ int xe_gsc_pkt_submit_kernel(struct xe_gsc *gsc, u64 addr_in, u32 size_in,
 		return PTR_ERR(job);
 	}
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	timeout = dma_fence_wait_timeout(fence, false, HZ);
 	dma_fence_put(fence);
drivers/gpu/drm/xe/xe_gt.c +3 -2
index 79f4128fe325..10885ba7c2d2 100644
@@ -182,9 +182,10 @@ static int emit_job_sync(struct xe_exec_queue *q, struct xe_bb *bb,
 	if (IS_ERR(job))
 		return PTR_ERR(job);
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	timeout = dma_fence_wait_timeout(fence, false, timeout_jiffies);
 	dma_fence_put(fence);
drivers/gpu/drm/xe/xe_migrate.c +15 -10
index fc918b4fba54..519f7c70abfb 100644
@@ -1022,10 +1022,11 @@ static struct dma_fence *__xe_migrate_copy(struct xe_migrate *m,
 		}
 
 		mutex_lock(&m->job_mutex);
-		xe_sched_job_arm(job);
 		dma_fence_put(fence);
-		fence = dma_fence_get(&job->drm.s_fence->finished);
+		fence = xe_sched_job_arm(job);
+		dma_fence_get(fence);
 		xe_sched_job_push(job);
+		xe_sched_job_put(job);
 
 		dma_fence_put(m->fence);
 		m->fence = dma_fence_get(fence);
@@ -1438,10 +1439,11 @@ struct dma_fence *xe_migrate_vram_copy_chunk(struct xe_bo *vram_bo, u64 vram_off
 						     DMA_RESV_USAGE_BOOKKEEP));
 
 		scoped_guard(mutex, &m->job_mutex) {
-			xe_sched_job_arm(job);
 			dma_fence_put(fence);
-			fence = dma_fence_get(&job->drm.s_fence->finished);
+			fence = xe_sched_job_arm(job);
+			dma_fence_get(fence);
 			xe_sched_job_push(job);
+			xe_sched_job_put(job);
 
 			dma_fence_put(m->fence);
 			m->fence = dma_fence_get(fence);
@@ -1676,10 +1678,11 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,
 		}
 
 		mutex_lock(&m->job_mutex);
-		xe_sched_job_arm(job);
 		dma_fence_put(fence);
-		fence = dma_fence_get(&job->drm.s_fence->finished);
+		fence = xe_sched_job_arm(job);
+		dma_fence_get(fence);
 		xe_sched_job_push(job);
+		xe_sched_job_put(job);
 
 		dma_fence_put(m->fence);
 		m->fence = dma_fence_get(fence);
@@ -1996,9 +1999,10 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,
 	if (is_migrate)
 		mutex_lock(&m->job_mutex);
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	if (is_migrate)
 		mutex_unlock(&m->job_mutex);
@@ -2282,9 +2286,10 @@ static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
 	}
 
 	mutex_lock(&m->job_mutex);
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	dma_fence_put(m->fence);
 	m->fence = dma_fence_get(fence);
drivers/gpu/drm/xe/xe_oa.c +3 -2
index c176a61febb2..8de8952a2ca7 100644
@@ -653,9 +653,10 @@ static struct dma_fence *xe_oa_submit_bb(struct xe_oa_stream *stream, enum xe_oa
 		}
 	}
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	xe_oa_unlock_vma(q);
 
drivers/gpu/drm/xe/xe_pxp_submit.c +6 -4
index e60526e30030..5af384acae82 100644
@@ -315,9 +315,10 @@ int xe_pxp_submit_session_termination(struct xe_pxp *pxp, u32 id)
 	if (IS_ERR(job))
 		return PTR_ERR(job);
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	timeout = dma_fence_wait_timeout(fence, false, HZ);
 
@@ -378,9 +379,10 @@ static int pxp_pkt_submit(struct xe_exec_queue *q, u64 batch_addr)
 	if (IS_ERR(job))
 		return PTR_ERR(job);
 
-	xe_sched_job_arm(job);
-	fence = dma_fence_get(&job->drm.s_fence->finished);
+	fence = xe_sched_job_arm(job);
+	dma_fence_get(fence);
 	xe_sched_job_push(job);
+	xe_sched_job_put(job);
 
 	timeout = dma_fence_wait_timeout(fence, false, HZ);
 	dma_fence_put(fence);
drivers/gpu/drm/xe/xe_sched_job.c +3 -2
index ae5b38b2a884..99f11bb4d2b9 100644
@@ -243,7 +243,7 @@ bool xe_sched_job_completed(struct xe_sched_job *job)
 				     xe_lrc_seqno(lrc));
 }
 
-void xe_sched_job_arm(struct xe_sched_job *job)
+struct dma_fence *xe_sched_job_arm(struct xe_sched_job *job)
 {
 	struct xe_exec_queue *q = job->q;
 	struct dma_fence *fence, *prev;
@@ -288,6 +288,8 @@ void xe_sched_job_arm(struct xe_sched_job *job)
 
 	job->fence = dma_fence_get(fence);	/* Pairs with put in scheduler */
 	drm_sched_job_arm(&job->drm);
+
+	return &job->drm.s_fence->finished;
 }
 
 void xe_sched_job_push(struct xe_sched_job *job)
@@ -295,7 +297,6 @@ void xe_sched_job_push(struct xe_sched_job *job)
 	xe_sched_job_get(job);
 	trace_xe_sched_job_exec(job);
 	drm_sched_entity_push_job(&job->drm);
-	xe_sched_job_put(job);
 }
 
 /**
drivers/gpu/drm/xe/xe_sched_job.h +1 -1
index 1c1cb44216c3..a39cc4ab980b 100644
@@ -55,7 +55,7 @@ static inline bool xe_sched_job_is_error(struct xe_sched_job *job)
 bool xe_sched_job_started(struct xe_sched_job *job);
 bool xe_sched_job_completed(struct xe_sched_job *job);
 
-void xe_sched_job_arm(struct xe_sched_job *job);
+struct dma_fence *xe_sched_job_arm(struct xe_sched_job *job);
 void xe_sched_job_push(struct xe_sched_job *job);
 
 void xe_sched_job_init_user_fence(struct xe_sched_job *job,
Signature · 1 line
2.34.1
author maintainer
#54 lore raw

RFC PATCH 6/12 drm/xe: Convert to DRM dep queue scheduler layer

In reply to Matthew Brost

Replace the drm_gpu_scheduler/drm_sched_entity pair used throughout Xe with the new drm_dep layer (struct drm_dep_queue / struct drm_dep_job).

The conversion spans three submission backends — GuC, execlist, and the generic dependency scheduler (xe_dep_scheduler) — as well as the job lifecycle, TDR callbacks, and device teardown sequencing.

xe_gpu_scheduler: struct drm_gpu_scheduler base replaced with struct drm_dep_queue. xe_sched_init() updated to drm_dep_queue_init() args. The xe_sched_entity alias now maps to drm_dep_queue (the N:1 entity-to-scheduler distinction disappears entirely since each queue is its own entity). drm_sched_for_each_pending_job replaced with drm_dep_queue_for_each_pending_job. drm_sched_tdr_queue_imm replaced with drm_dep_queue_trigger_timeout.

GuC backend: guc_exec_queue_free_job() removed; job lifetime is now managed by drm_dep_job refcounting rather than a free_job vfunc. guc_exec_queue_timedout_job() updated to return drm_dep_timedout_stat values; the already-signaled check is replaced with drm_dep_job_is_finished(); vf_recovery paths now return DRM_DEP_TIMEDOUT_STAT_REQUEUE_JOB. guc_dep_queue_release() added as the .release vfunc performing kfree_rcu on the embedded xe_guc_exec_queue.

xe_dep_scheduler: struct drm_gpu_scheduler + struct drm_sched_entity + struct rcu_head collapsed into a single struct drm_dep_queue (which carries its own rcu_head). drm_sched_entity_init/fini removed. xe_dep_scheduler_fini() simplified to drm_dep_queue_put(). xe_dep_scheduler_entity() renamed to xe_dep_scheduler_dep_q() to match the new naming.

GuC teardown sequencing: the previous wait_event_timeout-based drain is replaced by the drm_dep module's built-in unload protection. Each drm_dep_queue holds a drm_dev_get() reference on its owning struct drm_device; drm_dev_put() is called as the final step of queue teardown. This ensures the driver module cannot be unloaded while any queue is still alive without requiring a separate drain API or per-device hash table. guc_submit_fini() is simplified accordingly.

xe_sched_job: to_xe_sched_job() and drm_job accessors updated for struct drm_dep_job. Job dependency calls updated throughout (drm_dep_job_add_dependency, drm_dep_job_arm, drm_dep_job_push).

exec_queue: q->entity renamed to q->dep_q throughout.

Assisted-by GitHub Copilot:claude-sonnet-4.6

Me: WIP

---
 drivers/gpu/drm/xe/Kconfig                   |   2 +-
 drivers/gpu/drm/xe/xe_dep_job_types.h        |   8 +-
 drivers/gpu/drm/xe/xe_dep_scheduler.c        |  81 ++++----
 drivers/gpu/drm/xe/xe_dep_scheduler.h        |   7 +-
 drivers/gpu/drm/xe/xe_exec_queue_types.h     |   6 +-
 drivers/gpu/drm/xe/xe_execlist.c             |  43 ++---
 drivers/gpu/drm/xe/xe_execlist_types.h       |   4 +-
 drivers/gpu/drm/xe/xe_gpu_scheduler.c        |  38 ++--
 drivers/gpu/drm/xe/xe_gpu_scheduler.h        |  50 ++---
 drivers/gpu/drm/xe/xe_gpu_scheduler_types.h  |   9 +-
 drivers/gpu/drm/xe/xe_guc_exec_queue_types.h |   8 +-
 drivers/gpu/drm/xe/xe_guc_submit.c           | 184 ++++++++-----------
 drivers/gpu/drm/xe/xe_migrate.c              |   2 +-
 drivers/gpu/drm/xe/xe_pt.c                   |   2 +-
 drivers/gpu/drm/xe/xe_sched_job.c            |  52 +++---
 drivers/gpu/drm/xe/xe_sched_job.h            |   7 +-
 drivers/gpu/drm/xe/xe_sched_job_types.h      |   8 +-
 drivers/gpu/drm/xe/xe_sync.c                 |   2 +-
 drivers/gpu/drm/xe/xe_tlb_inval_job.c        |  86 ++++-----
 19 files changed, 255 insertions(+), 344 deletions(-)
drivers/gpu/drm/xe/Kconfig +1 -1
index 4d7dcaff2b91..9430877d6294 100644
@@ -41,7 +41,7 @@ config DRM_XE
 	select DRM_EXEC
 	select DRM_GPUSVM if !UML
 	select DRM_GPUVM
-	select DRM_SCHED
+	select DRM_DEP
 	select MMU_NOTIFIER
 	select WANT_DEV_COREDUMP
 	select AUXILIARY_BUS
drivers/gpu/drm/xe/xe_dep_job_types.h +3 -5
index c6a484f24c8c..891fe5cfcf89 100644
@@ -6,7 +6,7 @@
 #ifndef _XE_DEP_JOB_TYPES_H_
 #define _XE_DEP_JOB_TYPES_H_
 
-#include <drm/gpu_scheduler.h>
+#include <drm/drm_dep.h>
 
 struct xe_dep_job;
 
@@ -14,14 +14,12 @@ struct xe_dep_job;
 struct xe_dep_job_ops {
 	/** @run_job: Run generic Xe dependency job */
 	struct dma_fence *(*run_job)(struct xe_dep_job *job);
-	/** @free_job: Free generic Xe dependency job */
-	void (*free_job)(struct xe_dep_job *job);
 };
 
 /** struct xe_dep_job - Generic dependency Xe job */
 struct xe_dep_job {
-	/** @drm: base DRM scheduler job */
-	struct drm_sched_job drm;
+	/** @drm: base DRM dependency job */
+	struct drm_dep_job drm;
 	/** @ops: dependency job operations */
 	const struct xe_dep_job_ops *ops;
 };
drivers/gpu/drm/xe/xe_dep_scheduler.c +33 -48

137 lines not on this page · raw message

drivers/gpu/drm/xe/xe_dep_scheduler.h +4 -3
index 853961eec64b..c32b6f4f8c04 100644
@@ -5,8 +5,9 @@
 
 #include <linux/types.h>
 
-struct drm_sched_entity;
+struct drm_dep_queue;
 struct workqueue_struct;
+struct xe_dep_job;
 struct xe_dep_scheduler;
 struct xe_device;
 
@@ -17,5 +18,5 @@ xe_dep_scheduler_create(struct xe_device *xe,
 
 void xe_dep_scheduler_fini(struct xe_dep_scheduler *dep_scheduler);
 
-struct drm_sched_entity *
-xe_dep_scheduler_entity(struct xe_dep_scheduler *dep_scheduler);
+struct drm_dep_queue *
+xe_dep_scheduler_dep_q(struct xe_dep_scheduler *dep_scheduler);
drivers/gpu/drm/xe/xe_exec_queue_types.h +3 -3
index 8ce78e0b1d50..35c7625a2df5 100644
@@ -8,7 +8,7 @@
 
 #include <linux/kref.h>
 
-#include <drm/gpu_scheduler.h>
+#include <drm/drm_dep.h>
 
 #include "xe_gpu_scheduler_types.h"
 #include "xe_hw_engine_types.h"
@@ -245,8 +245,8 @@ struct xe_exec_queue {
 
 	/** @ring_ops: ring operations for this exec queue */
 	const struct xe_ring_ops *ring_ops;
-	/** @entity: DRM sched entity for this exec queue (1 to 1 relationship) */
-	struct drm_sched_entity *entity;
+	/** @dep_q: dep queue for this exec queue (1 to 1 relationship) */
+	struct drm_dep_queue *dep_q;
 
 #define XE_MAX_JOB_COUNT_PER_EXEC_QUEUE	1000
 	/** @job_cnt: number of drm jobs in this exec queue */
drivers/gpu/drm/xe/xe_execlist.c +17 -26

91 lines not on this page · raw message

drivers/gpu/drm/xe/xe_execlist_types.h +1 -3
index 92c4ba52db0c..c2c8218db350 100644
@@ -34,9 +34,7 @@ struct xe_execlist_port {
 struct xe_execlist_exec_queue {
 	struct xe_exec_queue *q;
 
-	struct drm_gpu_scheduler sched;
-
-	struct drm_sched_entity entity;
+	struct drm_dep_queue queue;
 
 	struct xe_execlist_port *port;
 
drivers/gpu/drm/xe/xe_gpu_scheduler.c +18 -20
index 9c8004d5dd91..a8e6384dffe8 100644
@@ -7,8 +7,7 @@
 
 static void xe_sched_process_msg_queue(struct xe_gpu_scheduler *sched)
 {
-	if (!drm_sched_is_stopped(&sched->base))
-		queue_work(sched->base.submit_wq, &sched->work_process_msg);
+	drm_dep_queue_work_enqueue(&sched->base, &sched->work_process_msg);
 }
 
 static void xe_sched_process_msg_queue_if_ready(struct xe_gpu_scheduler *sched)
@@ -43,7 +42,9 @@ static void xe_sched_process_msg_work(struct work_struct *w)
 		container_of(w, struct xe_gpu_scheduler, work_process_msg);
 	struct xe_sched_msg *msg;
 
-	if (drm_sched_is_stopped(&sched->base))
+	drm_dep_queue_sched_guard(&sched->base);
+
+	if (drm_dep_queue_is_stopped(&sched->base))
 		return;
 
 	msg = xe_sched_get_msg(sched);
@@ -55,25 +56,23 @@ static void xe_sched_process_msg_work(struct work_struct *w)
 }
 
 int xe_sched_init(struct xe_gpu_scheduler *sched,
-		  const struct drm_sched_backend_ops *ops,
+		  const struct drm_dep_queue_ops *ops,
 		  const struct xe_sched_backend_ops *xe_ops,
 		  struct workqueue_struct *submit_wq,
-		  uint32_t hw_submission, unsigned hang_limit,
-		  long timeout, struct workqueue_struct *timeout_wq,
-		  atomic_t *score, const char *name,
-		  struct device *dev)
+		  uint32_t hw_submission, long timeout,
+		  struct workqueue_struct *timeout_wq,
+		  enum drm_dep_queue_flags flags,
+		  const char *name, struct drm_device *drm)
 {
-	const struct drm_sched_init_args args = {
+	const struct drm_dep_queue_init_args args = {
 		.ops = ops,
 		.submit_wq = submit_wq,
-		.num_rqs = 1,
 		.credit_limit = hw_submission,
-		.hang_limit = hang_limit,
 		.timeout = timeout,
 		.timeout_wq = timeout_wq,
-		.score = score,
 		.name = name,
-		.dev = dev,
+		.drm = drm,
+		.flags = flags,
 	};
 
 	sched->ops = xe_ops;
@@ -81,30 +80,29 @@ int xe_sched_init(struct xe_gpu_scheduler *sched,
 	INIT_LIST_HEAD(&sched->msgs);
 	INIT_WORK(&sched->work_process_msg, xe_sched_process_msg_work);
 
-	return drm_sched_init(&sched->base, &args);
+	return drm_dep_queue_init(&sched->base, &args);
 }
 
 void xe_sched_fini(struct xe_gpu_scheduler *sched)
 {
-	xe_sched_submission_stop(sched);
-	drm_sched_fini(&sched->base);
+	drm_dep_queue_put(&sched->base);
 }
 
 void xe_sched_submission_start(struct xe_gpu_scheduler *sched)
 {
-	drm_sched_wqueue_start(&sched->base);
-	queue_work(sched->base.submit_wq, &sched->work_process_msg);
+	drm_dep_queue_start(&sched->base);
+	drm_dep_queue_work_enqueue(&sched->base, &sched->work_process_msg);
 }
 
 void xe_sched_submission_stop(struct xe_gpu_scheduler *sched)
 {
-	drm_sched_wqueue_stop(&sched->base);
+	drm_dep_queue_stop(&sched->base);
 	cancel_work_sync(&sched->work_process_msg);
 }
 
 void xe_sched_submission_resume_tdr(struct xe_gpu_scheduler *sched)
 {
-	drm_sched_resume_timeout(&sched->base, sched->base.timeout);
+	drm_dep_queue_resume_timeout(&sched->base);
 }
 
 void xe_sched_add_msg(struct xe_gpu_scheduler *sched,
drivers/gpu/drm/xe/xe_gpu_scheduler.h +18 -32

87 lines not on this page · raw message

drivers/gpu/drm/xe/xe_gpu_scheduler_types.h +4 -5
index 63d9bf92583c..ff89d36d3b2a 100644
@@ -6,7 +6,7 @@
 #ifndef _XE_GPU_SCHEDULER_TYPES_H_
 #define _XE_GPU_SCHEDULER_TYPES_H_
 
-#include <drm/gpu_scheduler.h>
+#include <drm/drm_dep.h>
 
 /**
  * struct xe_sched_msg - an in-band (relative to GPU scheduler run queue)
@@ -41,8 +41,8 @@ struct xe_sched_backend_ops {
  * struct xe_gpu_scheduler - Xe GPU scheduler
  */
 struct xe_gpu_scheduler {
-	/** @base: DRM GPU scheduler */
-	struct drm_gpu_scheduler		base;
+	/** @base: DRM dependency queue */
+	struct drm_dep_queue			base;
 	/** @ops: Xe scheduler ops */
 	const struct xe_sched_backend_ops	*ops;
 	/** @msgs: list of messages to be processed in @work_process_msg */
@@ -53,7 +53,6 @@ struct xe_gpu_scheduler {
 	struct work_struct		work_process_msg;
 };
 
-#define xe_sched_entity		drm_sched_entity
-#define xe_sched_policy		drm_sched_policy
+#define xe_sched_entity		drm_dep_queue
 
 #endif
drivers/gpu/drm/xe/xe_guc_exec_queue_types.h +2 -6
index fd0915ed8eb1..42ba4892ff71 100644
@@ -18,14 +18,10 @@ struct xe_exec_queue;
  * struct xe_guc_exec_queue - GuC specific state for an xe_exec_queue
  */
 struct xe_guc_exec_queue {
-	/** @q: Backpointer to parent xe_exec_queue */
-	struct xe_exec_queue *q;
-	/** @rcu: For safe freeing of exported dma fences */
-	struct rcu_head rcu;
 	/** @sched: GPU scheduler for this xe_exec_queue */
 	struct xe_gpu_scheduler sched;
-	/** @entity: Scheduler entity for this xe_exec_queue */
-	struct xe_sched_entity entity;
+	/** @q: Backpointer to parent xe_exec_queue */
+	struct xe_exec_queue *q;
 	/**
 	 * @static_msgs: Static messages for this xe_exec_queue, used when
 	 * a message needs to sent through the GPU scheduler but memory
drivers/gpu/drm/xe/xe_guc_submit.c +73 -111

387 lines not on this page · raw message

drivers/gpu/drm/xe/xe_migrate.c +1 -1
index 519f7c70abfb..565054ba0c34 100644
@@ -2279,7 +2279,7 @@ static struct dma_fence *xe_migrate_vram(struct xe_migrate *m,
 
 	if (deps && !dma_fence_is_signaled(deps)) {
 		dma_fence_get(deps);
-		err = drm_sched_job_add_dependency(&job->drm, deps);
+		err = drm_dep_job_add_dependency(&job->drm, deps);
 		if (err)
 			dma_fence_wait(deps, false);
 		err = 0;
drivers/gpu/drm/xe/xe_pt.c +1 -1
index 13b355fadd58..24374a3459c2 100644
@@ -1322,7 +1322,7 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job,
 				return -ETIME;
 
 			dma_fence_get(fence);
-			err = drm_sched_job_add_dependency(&job->drm, fence);
+			err = drm_dep_job_add_dependency(&job->drm, fence);
 			if (err)
 				return err;
 		}
drivers/gpu/drm/xe/xe_sched_job.c +29 -23

115 lines not on this page · raw message

drivers/gpu/drm/xe/xe_sched_job.h +3 -4
index a39cc4ab980b..bdd0305970b0 100644
@@ -20,7 +20,6 @@ void xe_sched_job_module_exit(void);
 
 struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,
 					 u64 *batch_addr);
-void xe_sched_job_destroy(struct kref *ref);
 
 /**
  * xe_sched_job_get - get reference to Xe schedule job
@@ -30,7 +29,7 @@ void xe_sched_job_destroy(struct kref *ref);
  */
 static inline struct xe_sched_job *xe_sched_job_get(struct xe_sched_job *job)
 {
-	kref_get(&job->refcount);
+	drm_dep_job_get(&job->drm);
 	return job;
 }
 
@@ -43,7 +42,7 @@ static inline struct xe_sched_job *xe_sched_job_get(struct xe_sched_job *job)
  */
 static inline void xe_sched_job_put(struct xe_sched_job *job)
 {
-	kref_put(&job->refcount, xe_sched_job_destroy);
+	drm_dep_job_put(&job->drm);
 }
 
 void xe_sched_job_set_error(struct xe_sched_job *job, int error);
@@ -62,7 +61,7 @@ void xe_sched_job_init_user_fence(struct xe_sched_job *job,
 				  struct xe_sync_entry *sync);
 
 static inline struct xe_sched_job *
-to_xe_sched_job(struct drm_sched_job *drm)
+to_xe_sched_job(struct drm_dep_job *drm)
 {
 	return container_of(drm, struct xe_sched_job, drm);
 }
drivers/gpu/drm/xe/xe_sched_job_types.h +3 -5
index 13c2970e81a8..6b6189f58fd2 100644
@@ -8,7 +8,7 @@
 
 #include <linux/kref.h>
 
-#include <drm/gpu_scheduler.h>
+#include <drm/drm_dep.h>
 
 struct xe_exec_queue;
 struct dma_fence;
@@ -35,12 +35,10 @@ struct xe_job_ptrs {
  * struct xe_sched_job - Xe schedule job (batch buffer tracking)
  */
 struct xe_sched_job {
-	/** @drm: base DRM scheduler job */
-	struct drm_sched_job drm;
+	/** @drm: base DRM dependency job */
+	struct drm_dep_job drm;
 	/** @q: Exec queue */
 	struct xe_exec_queue *q;
-	/** @refcount: ref count of this job */
-	struct kref refcount;
 	/**
 	 * @fence: dma fence to indicate completion. 1 way relationship - job
 	 * can safely reference fence, fence cannot safely reference job.
drivers/gpu/drm/xe/xe_sync.c +1 -1
index 24d6d9af20d6..cbe9c0adc9a2 100644
@@ -234,7 +234,7 @@ ALLOW_ERROR_INJECTION(xe_sync_entry_parse, ERRNO);
 int xe_sync_entry_add_deps(struct xe_sync_entry *sync, struct xe_sched_job *job)
 {
 	if (sync->fence)
-		return  drm_sched_job_add_dependency(&job->drm,
+		return  drm_dep_job_add_dependency(&job->drm,
 						     dma_fence_get(sync->fence));
 
 	return 0;
drivers/gpu/drm/xe/xe_tlb_inval_job.c +40 -46

171 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#55 lore raw

RFC PATCH 7/12 drm/xe: Make scheduler message lock IRQ-safe

In reply to Matthew Brost

Make message enqueuing safe in IRQ context by converting the scheduler message lock to an IRQ-safe guard.

---
 drivers/gpu/drm/xe/xe_gpu_scheduler.c | 28 +++++++++++++--------------
 drivers/gpu/drm/xe/xe_gpu_scheduler.h | 17 ++++++++--------
 drivers/gpu/drm/xe/xe_guc_submit.c    | 23 ++++++++++------------
 3 files changed, 32 insertions(+), 36 deletions(-)
drivers/gpu/drm/xe/xe_gpu_scheduler.c +14 -14
index a8e6384dffe8..14c1b8df439f 100644
@@ -14,11 +14,12 @@ static void xe_sched_process_msg_queue_if_ready(struct xe_gpu_scheduler *sched)
 {
 	struct xe_sched_msg *msg;
 
-	xe_sched_msg_lock(sched);
-	msg = list_first_entry_or_null(&sched->msgs, struct xe_sched_msg, link);
-	if (msg)
-		xe_sched_process_msg_queue(sched);
-	xe_sched_msg_unlock(sched);
+	xe_sched_msg_scoped_guard(sched) {
+		msg = list_first_entry_or_null(&sched->msgs,
+					       struct xe_sched_msg, link);
+		if (msg)
+			xe_sched_process_msg_queue(sched);
+	}
 }
 
 static struct xe_sched_msg *
@@ -26,12 +27,12 @@ xe_sched_get_msg(struct xe_gpu_scheduler *sched)
 {
 	struct xe_sched_msg *msg;
 
-	xe_sched_msg_lock(sched);
-	msg = list_first_entry_or_null(&sched->msgs,
-				       struct xe_sched_msg, link);
-	if (msg)
-		list_del_init(&msg->link);
-	xe_sched_msg_unlock(sched);
+	xe_sched_msg_scoped_guard(sched) {
+		msg = list_first_entry_or_null(&sched->msgs,
+					       struct xe_sched_msg, link);
+		if (msg)
+			list_del_init(&msg->link);
+	}
 
 	return msg;
 }
@@ -108,9 +109,8 @@ void xe_sched_submission_resume_tdr(struct xe_gpu_scheduler *sched)
 void xe_sched_add_msg(struct xe_gpu_scheduler *sched,
 		      struct xe_sched_msg *msg)
 {
-	xe_sched_msg_lock(sched);
-	xe_sched_add_msg_locked(sched, msg);
-	xe_sched_msg_unlock(sched);
+	xe_sched_msg_scoped_guard(sched)
+		xe_sched_add_msg_locked(sched, msg);
 }
 
 void xe_sched_add_msg_locked(struct xe_gpu_scheduler *sched,
drivers/gpu/drm/xe/xe_gpu_scheduler.h +8 -9
index 4086aafb0a9a..71c060398be6 100644
@@ -31,15 +31,14 @@ void xe_sched_add_msg_locked(struct xe_gpu_scheduler *sched,
 void xe_sched_add_msg_head(struct xe_gpu_scheduler *sched,
 			   struct xe_sched_msg *msg);
 
-static inline void xe_sched_msg_lock(struct xe_gpu_scheduler *sched)
-{
-	spin_lock(&sched->msg_lock);
-}
-
-static inline void xe_sched_msg_unlock(struct xe_gpu_scheduler *sched)
-{
-	spin_unlock(&sched->msg_lock);
-}
+/**
+ * xe_sched_msg_scoped_guard() - Scoped guard for scheduler message lock
+ * @__sched: xe_gpu_scheduler object
+ *
+ * IRQ-safe scoped guard for scheduler message lock
+ */
+#define xe_sched_msg_scoped_guard(__sched)	\
+	scoped_guard(spinlock_irqsave, &(__sched)->msg_lock)
 
 static inline void xe_sched_tdr_queue_imm(struct xe_gpu_scheduler *sched)
 {
drivers/gpu/drm/xe/xe_guc_submit.c +10 -13
index fc9704fad177..2f91902bd2cb 100644
@@ -2183,10 +2183,10 @@ static int guc_exec_queue_suspend(struct xe_exec_queue *q)
 	if (exec_queue_killed_or_banned_or_wedged(q))
 		return -EINVAL;
 
-	xe_sched_msg_lock(sched);
-	if (guc_exec_queue_try_add_msg(q, msg, SUSPEND))
-		q->guc->suspend_pending = true;
-	xe_sched_msg_unlock(sched);
+	xe_sched_msg_scoped_guard(sched) {
+		if (guc_exec_queue_try_add_msg(q, msg, SUSPEND))
+			q->guc->suspend_pending = true;
+	}
 
 	return 0;
 }
@@ -2242,9 +2242,8 @@ static void guc_exec_queue_resume(struct xe_exec_queue *q)
 
 	xe_gt_assert(guc_to_gt(guc), !q->guc->suspend_pending);
 
-	xe_sched_msg_lock(sched);
-	guc_exec_queue_try_add_msg(q, msg, RESUME);
-	xe_sched_msg_unlock(sched);
+	xe_sched_msg_scoped_guard(sched)
+		guc_exec_queue_try_add_msg(q, msg, RESUME);
 }
 
 static bool guc_exec_queue_reset_status(struct xe_exec_queue *q)
@@ -2666,9 +2665,8 @@ static void guc_exec_queue_replay_pending_state_change(struct xe_exec_queue *q)
 	if (q->guc->needs_suspend) {
 		msg = q->guc->static_msgs + STATIC_MSG_SUSPEND;
 
-		xe_sched_msg_lock(sched);
-		guc_exec_queue_try_add_msg_head(q, msg, SUSPEND);
-		xe_sched_msg_unlock(sched);
+		xe_sched_msg_scoped_guard(sched)
+			guc_exec_queue_try_add_msg_head(q, msg, SUSPEND);
 
 		q->guc->needs_suspend = false;
 	}
@@ -2681,9 +2679,8 @@ static void guc_exec_queue_replay_pending_state_change(struct xe_exec_queue *q)
 	if (q->guc->needs_resume) {
 		msg = q->guc->static_msgs + STATIC_MSG_RESUME;
 
-		xe_sched_msg_lock(sched);
-		guc_exec_queue_try_add_msg_head(q, msg, RESUME);
-		xe_sched_msg_unlock(sched);
+		xe_sched_msg_scoped_guard(sched)
+			guc_exec_queue_try_add_msg_head(q, msg, RESUME);
 
 		q->guc->needs_resume = false;
 	}
Signature · 1 line
2.34.1
author maintainer
#56 lore raw

RFC PATCH 8/12 drm/xe: Rework exec queue object on top of DRM dep

In reply to Matthew Brost

Move drm_dep_queue from the backend scheduler object into the xe_exec_queue object. This allows reference counting to shift from xe_exec_queue to drm_dep_queue and enables use of the drm_dep_queue fini vfunc to initiate asynchronous queue teardown with the GuC firmware.

Rework and shuffle code so that xe_exec_queue_destroy and xe_exec_queue_fini become IRQ-safe. Once this rework is complete, the GuC submission backend async-destroy worker can be dropped in favor of using dep_free_wq for asynchronous exec-queue destruction.

---
 drivers/gpu/drm/xe/xe_exec_queue.c           |  94 ++++++++-------
 drivers/gpu/drm/xe/xe_exec_queue.h           |   9 +-
 drivers/gpu/drm/xe/xe_exec_queue_types.h     |   6 +-
 drivers/gpu/drm/xe/xe_execlist.c             |  18 +--
 drivers/gpu/drm/xe/xe_execlist_types.h       |   2 -
 drivers/gpu/drm/xe/xe_gpu_scheduler.c        |  18 +--
 drivers/gpu/drm/xe/xe_gpu_scheduler.h        |   8 +-
 drivers/gpu/drm/xe/xe_gpu_scheduler_types.h  |   6 +-
 drivers/gpu/drm/xe/xe_guc_exec_queue_types.h |   4 +-
 drivers/gpu/drm/xe/xe_guc_submit.c           | 119 ++++++++++---------
 drivers/gpu/drm/xe/xe_sched_job.c            |   4 +-
 11 files changed, 143 insertions(+), 145 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +52 -42
index b287d0e0e60a..b34af4594512 100644
@@ -142,6 +142,12 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q)
 {
 	int i;
 
+	if (q->ufence_syncobj)
+		drm_syncobj_put(q->ufence_syncobj);
+
+	for (i = 0; i < q->width; ++i)
+		xe_lrc_put(q->lrc[i]);
+
 	for (i = 0; i < XE_EXEC_QUEUE_TLB_INVAL_COUNT; ++i)
 		if (q->tlb_inval[i].dep_scheduler)
 			xe_dep_scheduler_fini(q->tlb_inval[i].dep_scheduler);
@@ -157,11 +163,19 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q)
 		xe_vm_put(q->vm);
 	}
 
+	if (q->user_vm)
+		xe_vm_put(q->user_vm);
+
 	if (q->xef)
 		xe_file_put(q->xef);
 
 	kvfree(q->replay_state);
-	kfree(q);
+	if (drm_dep_queue_is_initialized(&q->dep_q)) {
+		xe_assert(gt_to_xe(q->gt), !drm_dep_queue_refcount(&q->dep_q));
+		kfree_rcu(q, dep_q.rcu);
+	} else {
+		kfree(q);
+	}
 }
 
 static int alloc_dep_schedulers(struct xe_device *xe, struct xe_exec_queue *q)
@@ -214,7 +228,6 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,
 	if (!q)
 		return ERR_PTR(-ENOMEM);
 
-	kref_init(&q->refcount);
 	q->flags = flags;
 	q->hwe = hwe;
 	q->gt = gt;
@@ -321,16 +334,6 @@ struct xe_lrc *xe_exec_queue_lrc(struct xe_exec_queue *q)
 	return q->lrc[0];
 }
 
-static void __xe_exec_queue_fini(struct xe_exec_queue *q)
-{
-	int i;
-
-	q->ops->fini(q);
-
-	for (i = 0; i < q->width; ++i)
-		xe_lrc_put(q->lrc[i]);
-}
-
 static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)
 {
 	int i, err;
@@ -357,8 +360,13 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)
 		flags |= XE_LRC_DISABLE_STATE_CACHE_PERF_FIX;
 
 	err = q->ops->init(q);
-	if (err)
+	if (err) {
+		if (drm_dep_queue_is_initialized(&q->dep_q))
+			xe_exec_queue_put(q);
+		else
+			__xe_exec_queue_free(q);
 		return err;
+	}
 
 	/*
 	 * This must occur after q->ops->init to avoid race conditions during VF
@@ -398,7 +406,7 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)
 	return 0;
 
 err_lrc:
-	__xe_exec_queue_fini(q);
+	xe_exec_queue_put(q);
 	return err;
 }
 
@@ -434,7 +442,7 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
 
 	err = __xe_exec_queue_init(q, flags);
 	if (err)
-		goto err_post_alloc;
+		return ERR_PTR(err);
 
 	/*
 	 * We can only add the queue to the PXP list after the init is complete,
@@ -446,15 +454,13 @@ struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *v
 	if (xe_exec_queue_uses_pxp(q)) {
 		err = xe_pxp_exec_queue_add(xe->pxp, q);
 		if (err)
-			goto err_post_init;
+			goto err_put;
 	}
 
 	return q;
 
-err_post_init:
-	__xe_exec_queue_fini(q);
-err_post_alloc:
-	__xe_exec_queue_free(q);
+err_put:
+	xe_exec_queue_put(q);
 	return ERR_PTR(err);
 }
 ALLOW_ERROR_INJECTION(xe_exec_queue_create, ERRNO);
@@ -565,26 +571,18 @@ ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
 
 /**
  * xe_exec_queue_destroy() - Destroy an exec queue
- * @ref: Reference count of the exec queue
+ * @q: The exec queue
  *
  * Called when the last reference to the exec queue is dropped.
- * Cleans up all resources associated with the exec queue.
- * This function should not be called directly; use xe_exec_queue_put() instead.
+ * Starts the cleanup process for an exec queue.
  */
-void xe_exec_queue_destroy(struct kref *ref)
+void xe_exec_queue_destroy(struct xe_exec_queue *q)
 {
-	struct xe_exec_queue *q = container_of(ref, struct xe_exec_queue, refcount);
 	struct xe_exec_queue *eq, *next;
 	int i;
 
 	xe_assert(gt_to_xe(q->gt), atomic_read(&q->job_cnt) == 0);
 
-	if (q->ufence_syncobj)
-		drm_syncobj_put(q->ufence_syncobj);
-
-	if (xe_exec_queue_uses_pxp(q))
-		xe_pxp_exec_queue_remove(gt_to_xe(q->gt)->pxp, q);
-
 	xe_exec_queue_last_fence_put_unlocked(q);
 	for_each_tlb_inval(i)
 		xe_exec_queue_tlb_inval_last_fence_put_unlocked(q, i);
@@ -595,24 +593,25 @@ void xe_exec_queue_destroy(struct kref *ref)
 			xe_exec_queue_put(eq);
 	}
 
-	if (q->user_vm) {
-		xe_vm_put(q->user_vm);
-		q->user_vm = NULL;
-	}
-
 	q->ops->destroy(q);
 }
 
 /**
- * xe_exec_queue_fini() - Finalize an exec queue
+ * xe_exec_queue_fini() - Free an exec queue
  * @q: The exec queue
  *
- * Finalizes the exec queue by updating run ticks, releasing LRC references,
- * and freeing the queue structure. This is called after the queue has been
- * destroyed and all references have been dropped.
+ * Free an exec queues resources.
+ *
+ * Context: Process context, not in path of reclaim.
  */
-void xe_exec_queue_fini(struct xe_exec_queue *q)
+void xe_exec_queue_free(struct xe_exec_queue *q)
 {
+	/*
+	 * We may take dma-resv on final VM put, which is reclaim-unsafe.
+	 * Teach lockdep that this function is also reclaim-unsafe.
+	 */
+	might_alloc(GFP_KERNEL);
+
 	/*
 	 * Before releasing our ref to lrc and xef, accumulate our run ticks
 	 * and wakeup any waiters.
@@ -621,10 +620,21 @@ void xe_exec_queue_fini(struct xe_exec_queue *q)
 	if (q->xef && atomic_dec_and_test(&q->xef->exec_queue.pending_removal))
 		wake_up_var(&q->xef->exec_queue.pending_removal);
 
-	__xe_exec_queue_fini(q);
 	__xe_exec_queue_free(q);
 }
 
+/**
+ * xe_exec_queue_fini() - Finalize an exec queue
+ * @q: The exec queue
+ *
+ * This is called after the queue has been destroyed and all references in the
+ * firmware / hardware have been dropped.
+ */
+void xe_exec_queue_fini(struct xe_exec_queue *q)
+{
+	q->ops->fini(q);
+}
+
 /**
  * xe_exec_queue_assign_name() - Assign a name to an exec queue
  * @q: The exec queue
drivers/gpu/drm/xe/xe_exec_queue.h +5 -4
index a82d99bd77bc..694beae7f11b 100644
@@ -31,14 +31,15 @@ struct xe_exec_queue *xe_exec_queue_create_bind(struct xe_device *xe,
 						struct xe_vm *user_vm,
 						u32 flags, u64 extensions);
 
+void xe_exec_queue_destroy(struct xe_exec_queue *q);
 void xe_exec_queue_fini(struct xe_exec_queue *q);
-void xe_exec_queue_destroy(struct kref *ref);
+void xe_exec_queue_free(struct xe_exec_queue *q);
 void xe_exec_queue_assign_name(struct xe_exec_queue *q, u32 instance);
 
 static inline struct xe_exec_queue *
 xe_exec_queue_get_unless_zero(struct xe_exec_queue *q)
 {
-	if (kref_get_unless_zero(&q->refcount))
+	if (drm_dep_queue_get_unless_zero(&q->dep_q))
 		return q;
 
 	return NULL;
@@ -48,13 +49,13 @@ struct xe_exec_queue *xe_exec_queue_lookup(struct xe_file *xef, u32 id);
 
 static inline struct xe_exec_queue *xe_exec_queue_get(struct xe_exec_queue *q)
 {
-	kref_get(&q->refcount);
+	drm_dep_queue_get(&q->dep_q);
 	return q;
 }
 
 static inline void xe_exec_queue_put(struct xe_exec_queue *q)
 {
-	kref_put(&q->refcount, xe_exec_queue_destroy);
+	drm_dep_queue_put(&q->dep_q);
 }
 
 static inline bool xe_exec_queue_is_parallel(struct xe_exec_queue *q)
drivers/gpu/drm/xe/xe_exec_queue_types.h +1 -5
index 35c7625a2df5..adb82c9064af 100644
@@ -6,8 +6,6 @@
 #ifndef _XE_EXEC_QUEUE_TYPES_H_
 #define _XE_EXEC_QUEUE_TYPES_H_
 
-#include <linux/kref.h>
-
 #include <drm/drm_dep.h>
 
 #include "xe_gpu_scheduler_types.h"
@@ -88,8 +86,6 @@ struct xe_exec_queue {
 	 * really be used for submissions.
 	 */
 	struct xe_hw_engine *hwe;
-	/** @refcount: ref count of this exec queue */
-	struct kref refcount;
 	/** @vm: VM (address space) for this exec queue */
 	struct xe_vm *vm;
 	/**
@@ -246,7 +242,7 @@ struct xe_exec_queue {
 	/** @ring_ops: ring operations for this exec queue */
 	const struct xe_ring_ops *ring_ops;
 	/** @dep_q: dep queue for this exec queue (1 to 1 relationship) */
-	struct drm_dep_queue *dep_q;
+	struct drm_dep_queue dep_q;
 
 #define XE_MAX_JOB_COUNT_PER_EXEC_QUEUE	1000
 	/** @job_cnt: number of drm jobs in this exec queue */
drivers/gpu/drm/xe/xe_execlist.c +5 -13
index fb948b2c617c..771039b0f4fa 100644
@@ -319,16 +319,12 @@ execlist_run_job(struct drm_dep_job *drm_job)
 	return job->fence;
 }
 
-static void execlist_dep_queue_release(struct drm_dep_queue *q)
+static void execlist_dep_queue_release(struct drm_dep_queue *dep_q)
 {
-	struct xe_execlist_exec_queue *exl =
-		container_of(q, typeof(*exl), queue);
+	struct xe_exec_queue *q = container_of(dep_q, typeof(*q), dep_q);
 
-	/*
-	 * RCU free: the dep queue's name may be referenced by exported dma
-	 * fences (timeline name). Defer freeing until after any RCU readers.
-	 */
-	kfree_rcu(exl, queue.rcu);
+	kfree(q->execlist);
+	xe_exec_queue_free(q);
 }
 
 static const struct drm_dep_queue_ops execlist_dep_queue_ops = {
@@ -359,7 +355,7 @@ static int execlist_exec_queue_init(struct xe_exec_queue *q)
 
 	exl->q = q;
 
-	err = drm_dep_queue_init(&exl->queue, &args);
+	err = drm_dep_queue_init(&q->dep_q, &args);
 	if (err)
 		goto err_free;
 
@@ -367,7 +363,6 @@ static int execlist_exec_queue_init(struct xe_exec_queue *q)
 	exl->has_run = false;
 	exl->active_priority = XE_EXEC_QUEUE_PRIORITY_UNSET;
 	q->execlist = exl;
-	q->dep_q = &exl->queue;
 
 	xe_exec_queue_assign_name(q, ffs(q->logical_mask) - 1);
 
@@ -380,9 +375,6 @@ static int execlist_exec_queue_init(struct xe_exec_queue *q)
 
 static void execlist_exec_queue_fini(struct xe_exec_queue *q)
 {
-	struct xe_execlist_exec_queue *exl = q->execlist;
-
-	drm_dep_queue_put(&exl->queue);
 }
 
 static void execlist_exec_queue_destroy_async(struct work_struct *w)
drivers/gpu/drm/xe/xe_execlist_types.h +0 -2
index c2c8218db350..9711f1813287 100644
@@ -34,8 +34,6 @@ struct xe_execlist_port {
 struct xe_execlist_exec_queue {
 	struct xe_exec_queue *q;
 
-	struct drm_dep_queue queue;
-
 	struct xe_execlist_port *port;
 
 	bool has_run;
drivers/gpu/drm/xe/xe_gpu_scheduler.c +9 -9
index 14c1b8df439f..cae1c6428409 100644
@@ -7,7 +7,7 @@
 
 static void xe_sched_process_msg_queue(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_work_enqueue(&sched->base, &sched->work_process_msg);
+	drm_dep_queue_work_enqueue(sched->dep_q, &sched->work_process_msg);
 }
 
 static void xe_sched_process_msg_queue_if_ready(struct xe_gpu_scheduler *sched)
@@ -43,9 +43,9 @@ static void xe_sched_process_msg_work(struct work_struct *w)
 		container_of(w, struct xe_gpu_scheduler, work_process_msg);
 	struct xe_sched_msg *msg;
 
-	drm_dep_queue_sched_guard(&sched->base);
+	drm_dep_queue_sched_guard(sched->dep_q);
 
-	if (drm_dep_queue_is_stopped(&sched->base))
+	if (drm_dep_queue_is_stopped(sched->dep_q))
 		return;
 
 	msg = xe_sched_get_msg(sched);
@@ -81,29 +81,29 @@ int xe_sched_init(struct xe_gpu_scheduler *sched,
 	INIT_LIST_HEAD(&sched->msgs);
 	INIT_WORK(&sched->work_process_msg, xe_sched_process_msg_work);
 
-	return drm_dep_queue_init(&sched->base, &args);
+	return drm_dep_queue_init(sched->dep_q, &args);
 }
 
 void xe_sched_fini(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_put(&sched->base);
+	drm_dep_queue_fini(sched->dep_q);
 }
 
 void xe_sched_submission_start(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_start(&sched->base);
-	drm_dep_queue_work_enqueue(&sched->base, &sched->work_process_msg);
+	drm_dep_queue_start(sched->dep_q);
+	drm_dep_queue_work_enqueue(sched->dep_q, &sched->work_process_msg);
 }
 
 void xe_sched_submission_stop(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_stop(&sched->base);
+	drm_dep_queue_stop(sched->dep_q);
 	cancel_work_sync(&sched->work_process_msg);
 }
 
 void xe_sched_submission_resume_tdr(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_resume_timeout(&sched->base);
+	drm_dep_queue_resume_timeout(sched->dep_q);
 }
 
 void xe_sched_add_msg(struct xe_gpu_scheduler *sched,
drivers/gpu/drm/xe/xe_gpu_scheduler.h +4 -4
index 71c060398be6..7b7bfb2d63e0 100644
@@ -42,7 +42,7 @@ void xe_sched_add_msg_head(struct xe_gpu_scheduler *sched,
 
 static inline void xe_sched_tdr_queue_imm(struct xe_gpu_scheduler *sched)
 {
-	drm_dep_queue_trigger_timeout(&sched->base);
+	drm_dep_queue_trigger_timeout(sched->dep_q);
 }
 
 static inline void xe_sched_resubmit_jobs(struct xe_gpu_scheduler *sched)
@@ -51,11 +51,11 @@ static inline void xe_sched_resubmit_jobs(struct xe_gpu_scheduler *sched)
 	struct xe_sched_job *job;
 	bool restore_replay = false;
 
-	drm_dep_queue_for_each_pending_job(drm_job, &sched->base) {
+	drm_dep_queue_for_each_pending_job(drm_job, sched->dep_q) {
 		job = to_xe_sched_job(drm_job);
 		restore_replay |= job->restore_replay;
 		if (restore_replay || !drm_dep_job_is_signaled(drm_job))
-			sched->base.ops->run_job(drm_job);
+			sched->dep_q->ops->run_job(drm_job);
 	}
 }
 
@@ -76,7 +76,7 @@ struct xe_sched_job *xe_sched_first_pending_job(struct xe_gpu_scheduler *sched)
 {
 	struct drm_dep_job *drm_job;
 
-	drm_dep_queue_for_each_pending_job(drm_job, &sched->base)
+	drm_dep_queue_for_each_pending_job(drm_job, sched->dep_q)
 		if (!drm_dep_job_is_signaled(drm_job))
 			return to_xe_sched_job(drm_job);
 
drivers/gpu/drm/xe/xe_gpu_scheduler_types.h +2 -4
index ff89d36d3b2a..b059cb75fca8 100644
@@ -41,8 +41,8 @@ struct xe_sched_backend_ops {
  * struct xe_gpu_scheduler - Xe GPU scheduler
  */
 struct xe_gpu_scheduler {
-	/** @base: DRM dependency queue */
-	struct drm_dep_queue			base;
+	/** @dep_q: DRM dependency queue */
+	struct drm_dep_queue			*dep_q;
 	/** @ops: Xe scheduler ops */
 	const struct xe_sched_backend_ops	*ops;
 	/** @msgs: list of messages to be processed in @work_process_msg */
@@ -53,6 +53,4 @@ struct xe_gpu_scheduler {
 	struct work_struct		work_process_msg;
 };
 
-#define xe_sched_entity		drm_dep_queue
-
 #endif
drivers/gpu/drm/xe/xe_guc_exec_queue_types.h +1 -3
index 42ba4892ff71..cb15e86823d2 100644
@@ -29,8 +29,6 @@ struct xe_guc_exec_queue {
 	 */
 #define MAX_STATIC_MSG_TYPE	3
 	struct xe_sched_msg static_msgs[MAX_STATIC_MSG_TYPE];
-	/** @destroy_async: do final destroy async from this worker */
-	struct work_struct destroy_async;
 	/** @resume_time: time of last resume */
 	u64 resume_time;
 	/** @state: GuC specific state for this xe_exec_queue */
@@ -40,7 +38,7 @@ struct xe_guc_exec_queue {
 	/** @wqi_tail: work queue item tail */
 	u32 wqi_tail;
 	/** @id: GuC id for this exec_queue */
-	u16 id;
+	u32 id;
 	/** @suspend_wait: wait queue used to wait on pending suspends */
 	wait_queue_head_t suspend_wait;
 	/** @suspend_pending: a suspend of the exec_queue is pending */
drivers/gpu/drm/xe/xe_guc_submit.c +63 -56

259 lines not on this page · raw message

drivers/gpu/drm/xe/xe_sched_job.c +1 -3
index 6b83618e82aa..d83f0f9ea4ab 100644
@@ -118,7 +118,7 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,
 	err = drm_dep_job_init(&job->drm,
 			       &(const struct drm_dep_job_init_args){
 					.ops = &xe_sched_job_dep_ops,
-					.q = q->dep_q,
+					.q = &q->dep_q,
 					.credits = 1,
 			       });
 	if (err)
@@ -126,7 +126,6 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,
 
 	job->q = q;
 	job->sample_timestamp = U64_MAX;
-	xe_exec_queue_get(job->q);
 	atomic_inc(&q->job_cnt);
 	xe_pm_runtime_get_noresume(job_to_xe(job));
 
@@ -188,7 +187,6 @@ static void xe_sched_job_release(struct drm_dep_job *dep_job)
 	dma_fence_put(job->fence);
 	job_free(job);
 	atomic_dec(&q->job_cnt);
-	xe_exec_queue_put(q);
 	xe_pm_runtime_put(xe);
 }
 
Signature · 1 line
2.34.1
author maintainer
#57 lore raw

RFC PATCH 9/12 drm/xe: Enable IRQ job put in DRM dep

In reply to Matthew Brost

Enable IRQ-context job puts in DRM dep now that all required support is in place. Set this flag unconditionally on all drm_dep_queue instances and update the kernel-doc for functions that can now be called from IRQ context.

---
 drivers/gpu/drm/xe/xe_exec_queue.c | 2 ++
 drivers/gpu/drm/xe/xe_guc_submit.c | 3 ++-
 drivers/gpu/drm/xe/xe_sched_job.c  | 2 ++
 3 files changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_exec_queue.c +2 -0
index b34af4594512..3aeba2fd7963 100644
@@ -575,6 +575,8 @@ ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
  *
  * Called when the last reference to the exec queue is dropped.
  * Starts the cleanup process for an exec queue.
+ *
+ * Context: Any context.
  */
 void xe_exec_queue_destroy(struct xe_exec_queue *q)
 {
drivers/gpu/drm/xe/xe_guc_submit.c +2 -1
index af54ac9d5607..064cf15166b9 100644
@@ -1935,7 +1935,8 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)
 	struct xe_guc *guc = exec_queue_to_guc(q);
 	struct workqueue_struct *submit_wq = NULL;
 	struct xe_guc_exec_queue *ge;
-	enum drm_dep_queue_flags flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED;
+	enum drm_dep_queue_flags flags = DRM_DEP_QUEUE_FLAGS_BYPASS_SUPPORTED |
+		DRM_DEP_QUEUE_FLAGS_JOB_PUT_IRQ_SAFE;
 	long timeout;
 	int err, i;
 
drivers/gpu/drm/xe/xe_sched_job.c +2 -0
index d83f0f9ea4ab..e3f7251b35b5 100644
@@ -175,6 +175,8 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,
  *
  * Called when ref == 0, drop a reference to job's xe_engine + fence, cleanup
  * and free memory for Xe schedule job.
+ *
+ * Context: Any context.
  */
 static void xe_sched_job_release(struct drm_dep_job *dep_job)
 {
Signature · 1 line
2.34.1
author maintainer
#58 lore raw

RFC PATCH 10/12 drm/xe: Use DRM dep queue kill semantics

In reply to Matthew Brost

Once the GuC context has its scheduling disabled by TDR or kill work item, the queue is taken off the hardware and can no longer touch memory without risking corruption. Invoke drm_dep_queue_kill, which bypasses any remaining job dependencies in the queue and calls run_job immediately for each remaining job. In run_job, if the queue returns drm_dep_queue_is_killed, immediately signal the hardware fence, as the queue can no longer access any memory associated with the fence being signaled.

---
 drivers/gpu/drm/xe/xe_guc_exec_queue_types.h |   2 +
 drivers/gpu/drm/xe/xe_guc_submit.c           | 139 +++++++++++++------
 2 files changed, 97 insertions(+), 44 deletions(-)
drivers/gpu/drm/xe/xe_guc_exec_queue_types.h +2 -0
index cb15e86823d2..72de6d0a754a 100644
@@ -29,6 +29,8 @@ struct xe_guc_exec_queue {
 	 */
 #define MAX_STATIC_MSG_TYPE	3
 	struct xe_sched_msg static_msgs[MAX_STATIC_MSG_TYPE];
+	/** @kill_work: Kill work item */
+	struct delayed_work kill_work;
 	/** @resume_time: time of last resume */
 	u64 resume_time;
 	/** @state: GuC specific state for this xe_exec_queue */
drivers/gpu/drm/xe/xe_guc_submit.c +95 -44

216 lines not on this page · raw message

Signature · 1 line
2.34.1