Skip to content

RFC Introduce DRM dep queue

Matthew Brost dri-devel intel-xe lkml rust-for-linux 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest12 patches 48 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/12workqueue: Add interface to teach lockdep to warn on reclaim violationsinclude/linux/workqueue.h kernel/workqueue.c+44 -05 replies
2/12drm/dep: Add DRM dependency queue layerdrivers/gpu/drm/Kconfig drivers/gpu/drm/Makefile drivers/gpu/drm/dep/Makefile +7 more+3404 -042 replies
3/12drm/xe: Use WQ_MEM_WARN_ON_RECLAIM on all workqueues in the reclaim pathdrivers/gpu/drm/xe/xe_device.c drivers/gpu/drm/xe/xe_ggtt.c drivers/gpu/drm/xe/xe_gt.c +2 more+10 -50 replies
4/12drm/xe: Issue GGTT invalidation under lock in ggtt_node_removedrivers/gpu/drm/xe/xe_ggtt.c+2 -71 reply
5/12drm/xe: Return fence from xe_sched_job_arm and adjust job referencesdrivers/gpu/drm/xe/tests/xe_migrate.c drivers/gpu/drm/xe/xe_exec.c drivers/gpu/drm/xe/xe_gsc.c +7 more+49 -350 replies
6/12drm/xe: Convert to DRM dep queue scheduler layerdrivers/gpu/drm/xe/Kconfig drivers/gpu/drm/xe/xe_dep_job_types.h drivers/gpu/drm/xe/xe_dep_scheduler.c +16 more+255 -3440 replies
7/12drm/xe: Make scheduler message lock IRQ-safedrivers/gpu/drm/xe/xe_gpu_scheduler.c drivers/gpu/drm/xe/xe_gpu_scheduler.h drivers/gpu/drm/xe/xe_guc_submit.c+32 -360 replies
8/12drm/xe: Rework exec queue object on top of DRM depdrivers/gpu/drm/xe/xe_exec_queue.c drivers/gpu/drm/xe/xe_exec_queue.h drivers/gpu/drm/xe/xe_exec_queue_types.h +8 more+143 -1450 replies
9/12drm/xe: Enable IRQ job put in DRM depdrivers/gpu/drm/xe/xe_exec_queue.c drivers/gpu/drm/xe/xe_guc_submit.c drivers/gpu/drm/xe/xe_sched_job.c+6 -10 replies
10/12drm/xe: Use DRM dep queue kill semanticsdrivers/gpu/drm/xe/xe_guc_exec_queue_types.h drivers/gpu/drm/xe/xe_guc_submit.c+97 -440 replies
11/12accel/amdxdna: Convert to drm_dep scheduler layerdrivers/accel/amdxdna/Kconfig drivers/accel/amdxdna/aie2_ctx.c drivers/accel/amdxdna/aie2_pci.h +3 more+88 -830 replies
12/12drm/panthor: Convert to drm_dep scheduler layerdrivers/gpu/drm/panthor/Kconfig drivers/gpu/drm/panthor/panthor_device.c drivers/gpu/drm/panthor/panthor_device.h +5 more+223 -2490 replies

Reviews trailers per patch

2 of 12 patches have a Reviewed-by or Acked-by.

PatchMatthew BrostTejun HeoDnyaneshwar Bhadane
1/12A-b
2/12
3/12
4/12Signed-by R-b
5/12
6/12
7/12
8/12
9/12
10/12
11/12
12/12

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v1 thread

Cover letter [RFC PATCH 00/12] Introduce DRM dep queue

Diverging requirements between GPU drivers using firmware scheduling and those using hardware scheduling have shown that drm_gpu_scheduler is no longer sufficient for firmware-scheduled GPU drivers. The technical debt, lack of memory-safety guarantees, absence of clear object-lifetime rules, and numerous driver-specific hacks have rendered drm_gpu_scheduler unmaintainable. It is time for a fresh design for firmware-scheduled GPU drivers—one that addresses all of the aforementioned shortcomings.

Key changes from DRM scheduler:
- Unification of drm_gpu_scheduler, drm_sched_entity into drm_dep_queue
- Reference counting
- Clear object-lifetime ownership
- Privatisation of drm dep fence
- Drop scheduled fence from drm dep fence
- Submit workqueue bypass path (optional)
- IRQ removal of jobs (i.e., bypass worker to drop ref to job) (optional)
- Asynchronous teardown
- Extensive lockdep annotations and asserts
- Extensive kernel doc

Xe has been fully converted to drm dep and extensively tested with and without verbose kernel debug Kconfig options. Everything seems to be working locally. All permutations of drm_dep_queue_flags tested.

Compile tested only conversions - Panthor and AMDXDNA.

Other candidates to convert are: Nouveau, PVR, and AMDGPU VM queue.

Forward looking - this design seems suitable for Rust bindings and use by Nova, Tyr given the clean object-lifetime rules and reference counting.

The Xe team is aligned on moving forward with this; hopefully the rest of the community embraces it.

In addition to DRM dep, workqueues have been updated with interfaces to enforce reclaim safety with DRM dep, and Xe is utilizing these interfaces. This part could be split out into a separate series.

I'm also fairly certain that a sane HW queue component could be built on top of DRM dep as well, replacing DRM sched for all drivers they choose.

Matt

Matthew Brost (12):
workqueue: Add interface to teach lockdep to warn on reclaim
violations
drm/dep: Add DRM dependency queue layer
drm/xe: Use WQ_MEM_WARN_ON_RECLAIM on all workqueues in the reclaim
path
drm/xe: Issue GGTT invalidation under lock in ggtt_node_remove drm/xe: Return fence from xe_sched_job_arm and adjust job references drm/xe: Convert to DRM dep queue scheduler layer
drm/xe: Make scheduler message lock IRQ-safe
drm/xe: Rework exec queue object on top of DRM dep drm/xe: Enable IRQ job put in DRM dep
drm/xe: Use DRM dep queue kill semantics
accel/amdxdna: Convert to drm_dep scheduler layer drm/panthor: Convert to drm_dep scheduler layer

 drivers/accel/amdxdna/Kconfig                |    2 +-
 drivers/accel/amdxdna/aie2_ctx.c             |  144 +-
 drivers/accel/amdxdna/aie2_pci.h             |    4 +-
 drivers/accel/amdxdna/amdxdna_ctx.c          |    5 +-
 drivers/accel/amdxdna/amdxdna_ctx.h          |    4 +-
 drivers/gpu/drm/Kconfig                      |    4 +
 drivers/gpu/drm/Makefile                     |    1 +
 drivers/gpu/drm/dep/Makefile                 |    5 +
 drivers/gpu/drm/dep/drm_dep_fence.c          |  406 +++++
 drivers/gpu/drm/dep/drm_dep_fence.h          |   25 +
 drivers/gpu/drm/dep/drm_dep_job.c            |  675 +++++++
 drivers/gpu/drm/dep/drm_dep_job.h            |   13 +
 drivers/gpu/drm/dep/drm_dep_queue.c          | 1647 ++++++++++++++++++
 drivers/gpu/drm/dep/drm_dep_queue.h          |   31 +
 drivers/gpu/drm/panthor/Kconfig              |    2 +-
 drivers/gpu/drm/panthor/panthor_device.c     |    5 +-
 drivers/gpu/drm/panthor/panthor_device.h     |    2 +-
 drivers/gpu/drm/panthor/panthor_drv.c        |   35 +-
 drivers/gpu/drm/panthor/panthor_mmu.c        |  160 +-
 drivers/gpu/drm/panthor/panthor_mmu.h        |   14 +-
 drivers/gpu/drm/panthor/panthor_sched.c      |  242 ++-
 drivers/gpu/drm/panthor/panthor_sched.h      |   12 +-
 drivers/gpu/drm/xe/Kconfig                   |    2 +-
 drivers/gpu/drm/xe/tests/xe_migrate.c        |   10 +-
 drivers/gpu/drm/xe/xe_dep_job_types.h        |    8 +-
 drivers/gpu/drm/xe/xe_dep_scheduler.c        |   81 +-
 drivers/gpu/drm/xe/xe_dep_scheduler.h        |    7 +-
 drivers/gpu/drm/xe/xe_device.c               |    3 +-
 drivers/gpu/drm/xe/xe_exec.c                 |   12 +-
 drivers/gpu/drm/xe/xe_exec_queue.c           |   96 +-
 drivers/gpu/drm/xe/xe_exec_queue.h           |    9 +-
 drivers/gpu/drm/xe/xe_exec_queue_types.h     |   10 +-
 drivers/gpu/drm/xe/xe_execlist.c             |   39 +-
 drivers/gpu/drm/xe/xe_execlist_types.h       |    4 -
 drivers/gpu/drm/xe/xe_ggtt.c                 |   12 +-
 drivers/gpu/drm/xe/xe_gpu_scheduler.c        |   66 +-
 drivers/gpu/drm/xe/xe_gpu_scheduler.h        |   67 +-
 drivers/gpu/drm/xe/xe_gpu_scheduler_types.h  |    9 +-
 drivers/gpu/drm/xe/xe_gsc.c                  |    5 +-
 drivers/gpu/drm/xe/xe_gsc_submit.c           |    5 +-
 drivers/gpu/drm/xe/xe_gt.c                   |    8 +-
 drivers/gpu/drm/xe/xe_guc_ct.c               |    3 +-
 drivers/gpu/drm/xe/xe_guc_exec_queue_types.h |   14 +-
 drivers/gpu/drm/xe/xe_guc_submit.c           |  360 ++--
 drivers/gpu/drm/xe/xe_migrate.c              |   27 +-
 drivers/gpu/drm/xe/xe_oa.c                   |    5 +-
 drivers/gpu/drm/xe/xe_pt.c                   |    2 +-
 drivers/gpu/drm/xe/xe_pxp_submit.c           |   10 +-
 drivers/gpu/drm/xe/xe_sched_job.c            |   59 +-
 drivers/gpu/drm/xe/xe_sched_job.h            |    9 +-
 drivers/gpu/drm/xe/xe_sched_job_types.h      |    8 +-
 drivers/gpu/drm/xe/xe_sync.c                 |    2 +-
 drivers/gpu/drm/xe/xe_tlb_inval.c            |    3 +-
 drivers/gpu/drm/xe/xe_tlb_inval_job.c        |   86 +-
 include/drm/drm_dep.h                        |  597 +++++++
 include/linux/workqueue.h                    |    3 +
 include/trace/events/amdxdna.h               |   12 +-
 kernel/workqueue.c                           |   41 +
 58 files changed, 4268 insertions(+), 864 deletions(-)
 create mode 100644 drivers/gpu/drm/dep/Makefile
 create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_fence.h
 create mode 100644 drivers/gpu/drm/dep/drm_dep_job.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_job.h
 create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.c
 create mode 100644 drivers/gpu/drm/dep/drm_dep_queue.h
 create mode 100644 include/drm/drm_dep.h
Signature · 1 line
2.34.1