Skip to content

RFC ULLS for kernel submission of migration jobs

Matthew Brost intel-xe 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest8 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8drm/xe: Add xe_hw_engine_write_ring_taildrivers/gpu/drm/xe/xe_hw_engine.c drivers/gpu/drm/xe/xe_hw_engine.h+11 -00 replies
2/8drm/xe: Add ULLS support to LRCdrivers/gpu/drm/xe/xe_lrc.c drivers/gpu/drm/xe/xe_lrc.h drivers/gpu/drm/xe/xe_lrc_types.h+54 -00 replies
3/8drm/xe: Add ULLS flags for jobsdrivers/gpu/drm/xe/xe_sched_job_types.h+3 -00 replies
4/8drm/xe: Add ULLS migration job support to migration layerdrivers/gpu/drm/xe/xe_migrate.c drivers/gpu/drm/xe/xe_migrate.h+132 -20 replies
5/8drm/xe: Add MI_SEMAPHORE_WAIT instruction defsdrivers/gpu/drm/xe/instructions/xe_mi_commands.h+6 -00 replies
6/8drm/xe: Add ULLS migration job support to ring opsdrivers/gpu/drm/xe/xe_ring_ops.c+32 -00 replies
7/8drm/xe: Add ULLS migration job support to GuC submissiondrivers/gpu/drm/xe/xe_guc_submit.c+20 -60 replies
8/8drm/xe: Enable ULLS migration jobs when opening LR VMdrivers/gpu/drm/xe/xe_vm.c+10 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 0/8] ULLS for kernel submission of migration jobs

Ultra low latency for kernel submission of migration jobs.

The basic idea is that faults (CPU or GPU) typically depend on migration jobs. Faults should be addressed as quickly as possible, but context switches via GuC on hardware are slow. To avoid context switches, perform ULLS in the kernel for migration jobs on discrete faulting devices with an LR VM open.

This is implemented by switching the migration layer to ULLS mode upon opening an LR VM. In ULLS mode, migration jobs have a preamble and
postamble: the preamble clears the current semaphore value, and the postamble waits for the next semaphore value. Each job submission sets the current semaphore in memory, bypassing the GuC. The net effect is that the migration execution queue never gets switched off the hardware while an LR VM is open.

There may be concerns regarding power management, as the ring program continuously runs on a copy engine, and a force wake reference to a copy engine is held with an LR VM open.

The implementation has been lightly tested but seems to be working.

This approach will likely be put on hold until SVM is operational with benchmarks, but it is being posted early for feedback and as a public checkpoint.

Matt

Matthew Brost (8):
drm/xe: Add xe_hw_engine_write_ring_tail
drm/xe: Add ULLS support to LRC
drm/xe: Add ULLS flags for jobs
drm/xe: Add ULLS migration job support to migration layer drm/xe: Add MI_SEMAPHORE_WAIT instruction defs drm/xe: Add ULLS migration job support to ring ops drm/xe: Add ULLS migration job support to GuC submission drm/xe: Enable ULLS migration jobs when opening LR VM

 .../gpu/drm/xe/instructions/xe_mi_commands.h  |   6 +
 drivers/gpu/drm/xe/xe_guc_submit.c            |  26 +++-
 drivers/gpu/drm/xe/xe_hw_engine.c             |  10 ++
 drivers/gpu/drm/xe/xe_hw_engine.h             |   1 +
 drivers/gpu/drm/xe/xe_lrc.c                   |  49 +++++++
 drivers/gpu/drm/xe/xe_lrc.h                   |   3 +
 drivers/gpu/drm/xe/xe_lrc_types.h             |   2 +
 drivers/gpu/drm/xe/xe_migrate.c               | 130 +++++++++++++++++-
 drivers/gpu/drm/xe/xe_migrate.h               |   4 +
 drivers/gpu/drm/xe/xe_ring_ops.c              |  32 +++++
 drivers/gpu/drm/xe/xe_sched_job_types.h       |   3 +
 drivers/gpu/drm/xe/xe_vm.c                    |  10 ++
 12 files changed, 268 insertions(+), 8 deletions(-)
Signature · 1 line
2.34.1