CPU binds and ULLS on migration queue
Subsystems (4)
Matthew Brost
maintainerv6 cover CPU binds and ULLS on migration queue
We now have data demonstrating the need for CPU binds and ULLS on the migration queue, based on results generated from [1].
On BMG, measurements show that when the GPU is continuously processing faults, copy jobs run approximately 30–40µs faster (depending on the test case) with ULLS compared to traditional GuC submission with SLPC enabled on the migration queue. Startup from a cold GPU shows an even larger speedup. Given the critical nature of fault performance, ULLS appears to be a worthwhile feature.
In addition to driver telemetry, UMD compute benchmarks consistently show multiple GB/s improvement in pagefault benchmarks with ULLS enabled.
ULLS will consume more power (not yet measured) due to a continuously running batch on the paging engine. However, compute UMDs already do this on engines exposed to users, so this seems like a worthwhile tradeoff. To mitigate power concerns, ULLS will exit after a period of time in which no faults have been processed.
CPU binds are required for ULLS to function, as the migration queue needs exclusive access to the paging hardware engine. Thus, CPU binds are included here.
Beyond being a requirement for ULLS, CPU binds should also reduce VM-bind latency, provide clearer multi-tile and TLB-invalidation layering, reduce pressure on GuC during fault storms as it is bypassed, and decouple kernel binds from unrelated copy/clear jobs—especially beneficial when faults are serviced in parallel. In a parallel-faulting test case, average bind time was reduced by approximately 15µs. In the worst case, 2MB copy time (~60–140µs) × (number of pagefault threads −
1) of latency would otherwise be added to a single fault. Reducing this latency increases overall throughput of the fault handler.
This series can be merged in phases:
Phase 1: CPU binds (patches 1–13)
Phase 2: CPU-bind components and multi-tile relayers (patches 14–17) Phase 3: ULLS on the migration execution queue (patches 18–25)
v2:
- Use delayed worker to exit ULLS mode in an effort to save on power
- Various other cleanups
v3:
- CPU bind component, multi-tile relayer
- Split CPU bind patches in many small patches
v4:
- Rebase, address feedback, add ULLS doc patch
v5:
- Address Sashiko feedback, fix checkpatch issues
v6:
- Reworked ULLS to drop KMD MMIO ring tail write, move this to ring instruction(s)
- Several other small cleanups based on Sashiko feedback
Matt
[1] https://patchwork.freedesktop.org/series/149811/
Matthew Brost (24):
drm/xe: Drop struct xe_migrate_pt_update argument from populate/clear
vfuns
drm/xe: Add xe_migrate_update_pgtables_cpu_execute helper drm/xe: Decouple exec queue idle check from LRC
drm/xe: Add job count to GuC exec queue snapshot
drm/xe: Update xe_bo_put_deferred arguments to include writeback flag drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
drm/xe: Update scheduler job layer to support PT jobs drm/xe: Add helpers to access PT ops
drm/xe: Add struct xe_pt_job_ops
drm/xe: Update GuC submission backend to run PT jobs drm/xe: Store level in struct xe_vm_pgtable_update
drm/xe: Don't use migrate exec queue for page fault binds drm/xe: Enable CPU binds for jobs
drm/xe: Remove unused arguments from xe_migrate_pt_update_ops drm/xe: Make bind queues operate cross-tile
drm/xe: Add CPU bind layer
drm/xe: Add device flag to enable PT mirroring across tiles drm/xe: Add ULLS support to LRC
drm/xe: Add ULLS migration job support to migration layer drm/xe: Add ULLS migration job support to ring ops
drm/xe: Add ULLS migration job support to GuC submission drm/xe: Enter ULLS for migration jobs upon page fault or SVM prefetch drm/xe: Add modparam to enable / disable ULLS on migrate queue drm/xe: Document ULLS for migration jobs
Documentation/gpu/xe/xe_migrate.rst | 3 + drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_bo.c | 8 +- drivers/gpu/drm/xe/xe_bo.h | 11 +- drivers/gpu/drm/xe/xe_bo_types.h | 2 - drivers/gpu/drm/xe/xe_cpu_bind.c | 298 +++++++++ drivers/gpu/drm/xe/xe_cpu_bind.h | 118 ++++ drivers/gpu/drm/xe/xe_debugfs.c | 1 + drivers/gpu/drm/xe/xe_defaults.h | 1 + drivers/gpu/drm/xe/xe_device.c | 6 + drivers/gpu/drm/xe/xe_device_types.h | 11 + drivers/gpu/drm/xe/xe_drm_client.c | 2 +- drivers/gpu/drm/xe/xe_exec_queue.c | 163 ++--- drivers/gpu/drm/xe/xe_exec_queue.h | 16 +- drivers/gpu/drm/xe/xe_exec_queue_types.h | 20 +- drivers/gpu/drm/xe/xe_guc_submit.c | 79 ++- drivers/gpu/drm/xe/xe_guc_submit_types.h | 2 + drivers/gpu/drm/xe/xe_lrc.c | 73 +++ drivers/gpu/drm/xe/xe_lrc.h | 4 + drivers/gpu/drm/xe/xe_lrc_types.h | 4 + drivers/gpu/drm/xe/xe_migrate.c | 759 ++++++++++------------ drivers/gpu/drm/xe/xe_migrate.h | 95 +-- drivers/gpu/drm/xe/xe_module.c | 4 + drivers/gpu/drm/xe/xe_module.h | 1 + drivers/gpu/drm/xe/xe_pagefault.c | 3 + drivers/gpu/drm/xe/xe_pci.c | 2 + drivers/gpu/drm/xe/xe_pci_types.h | 3 +- drivers/gpu/drm/xe/xe_pt.c | 785 ++++++++++++++--------- drivers/gpu/drm/xe/xe_pt.h | 12 +- drivers/gpu/drm/xe/xe_pt_types.h | 49 +- drivers/gpu/drm/xe/xe_ring_ops.c | 86 +++ drivers/gpu/drm/xe/xe_ring_ops_types.h | 24 + drivers/gpu/drm/xe/xe_sched_job.c | 103 ++- drivers/gpu/drm/xe/xe_sched_job.h | 56 ++ drivers/gpu/drm/xe/xe_sched_job_types.h | 49 +- drivers/gpu/drm/xe/xe_sync.c | 20 +- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 28 +- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 4 +- drivers/gpu/drm/xe/xe_trace.h | 2 +- drivers/gpu/drm/xe/xe_vm.c | 240 +++---- drivers/gpu/drm/xe/xe_vm.h | 3 + drivers/gpu/drm/xe/xe_vm_types.h | 12 +- 42 files changed, 1956 insertions(+), 1207 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.c create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.h
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 1/24 drm/xe: Drop struct xe_migrate_pt_update argument from populate/clear vfuns
Remove the xe_migrate_pt_update argument from the populate and clear vfuns. This structure will not be available in run_job, where CPU binds will be implemented. The populate path no longer needs it, and the clear path already uses the VM field instead.
--- drivers/gpu/drm/xe/xe_migrate.c | 9 +++++---- drivers/gpu/drm/xe/xe_migrate.h | 12 +++++------- drivers/gpu/drm/xe/xe_pt.c | 12 +++++------- 3 files changed, 15 insertions(+), 18 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +5 -4
@@ -1760,6 +1760,7 @@ static void write_pgtable(struct xe_tile *tile, struct xe_bb *bb, u64 ppgtt_ofs, struct xe_migrate_pt_update *pt_update) {const struct xe_migrate_pt_update_ops *ops = pt_update->ops; +struct xe_vm *vm = pt_update->vops->vm;u32 chunk;u32 ofs = update->ofs, size = update->qwords;
@@ -1791,10 +1792,10 @@ static void write_pgtable(struct xe_tile *tile, struct xe_bb *bb, u64 ppgtt_ofs, bb->cs[bb->len++] =lower_32_bits (addr); bb->cs[bb->len++] =upper_32_bits (addr);if (pt_op->bind) - ops->populate (pt_update, tile,NULL , bb->cs + bb->len, + ops->populate (tile,NULL , bb->cs + bb->len, ofs, chunk, update);else - ops->clear (pt_update, tile,NULL , bb->cs + bb->len, + ops->clear (vm, tile,NULL , bb->cs + bb->len, ofs, chunk, update); bb->len += chunk *2 ;
@@ -1851,12 +1852,12 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m, &pt_op->entries[j];if (pt_op->bind) - ops->populate (pt_update, m->tile, + ops->populate (m->tile, &update->pt_bo->vmap,NULL , update->ofs, update->qwords, update);else - ops->clear (pt_update, m->tile, + ops->clear (vm, m->tile, &update->pt_bo->vmap,NULL , update->ofs, update->qwords, update); }
drivers/gpu/drm/xe/xe_migrate.h +5 -7
@@ -40,7 +40,6 @@ enum xe_migrate_copy_dir {struct xe_migrate_pt_update_ops {/** * @populate: Populate a command buffer or page-table with ptes. -* @pt_update: Embeddable callback argument. * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. * @pos: If @map is NULL, map into the memory to be populated.
@@ -52,13 +51,12 @@ struct xe_migrate_pt_update_ops {* page-table system to populate command buffers or shared * page-tables with PTEs. */ -void (*populate)(struct xe_migrate_pt_update *pt_update, -struct xe_tile *tile,struct iosys_map *map, +void (*populate)(struct xe_tile *tile,struct iosys_map *map,void *pos,u32 ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update);/** * @clear: Clear a command buffer or page-table with ptes. -* @pt_update: Embeddable callback argument. +* @vm: VM being updated * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. * @pos: If @map is NULL, map into the memory to be populated.
@@ -70,9 +68,9 @@ struct xe_migrate_pt_update_ops {* page-table system to populate command buffers or shared * page-tables with PTEs. */ -void (*clear)(struct xe_migrate_pt_update *pt_update, -struct xe_tile *tile,struct iosys_map *map, -void *pos,u32 ofs,u32 num_qwords, +void (*clear)(struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,void *pos,u32 ofs, +u32 num_qwords,const struct xe_vm_pgtable_update *update);/**
drivers/gpu/drm/xe/xe_pt.c +5 -7
@@ -1092,9 +1092,8 @@ bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm, }static void -xe_vm_populate_pgtable (struct xe_migrate_pt_update *pt_update,struct xe_tile *tile, -struct iosys_map *map,void *data, -u32 qword_ofs,u32 num_qwords, +xe_vm_populate_pgtable (struct xe_tile *tile,struct iosys_map *map, +void *data,u32 qword_ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update) {struct xe_pt_entry *ptes = update->pt_entries;
@@ -2008,12 +2007,11 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, }static void -xe_migrate_clear_pgtable_callback (struct xe_migrate_pt_update *pt_update, -struct xe_tile *tile,struct iosys_map *map, -void *ptr,u32 qword_ofs,u32 num_qwords, +xe_migrate_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,void *ptr, +u32 qword_ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update) { -struct xe_vm *vm = pt_update->vops->vm;u64 empty =__xe_pt_empty_pte (tile, vm, update->pt->level);int i;
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 2/24 drm/xe: Add xe_migrate_update_pgtables_cpu_execute helper
In reply to Matthew BrostAdd the xe_migrate_update_pgtables_cpu_execute helper, which performs the CPU-side page-table update. This will support implementing CPU binds, as the submission backend can call this helper once a bind job’s dependencies are resolved. While here, add assertions to provide basic sanity checks on tht function arguments.
--- drivers/gpu/drm/xe/xe_migrate.c | 58 ++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 23 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +35 -23
@@ -1819,6 +1819,38 @@ struct migrate_test_params { container_of(_priv, struct migrate_test_params, base)#endif +static void +xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, +const struct xe_migrate_pt_update_ops *ops, +struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops) +{ +u32 j, i; + +for (j =0 ; j < num_ops; ++j, ++pt_op) { +for (i =0 ; i < pt_op->num_entries; i++) { +const struct xe_vm_pgtable_update *update = + &pt_op->entries[i]; + +xe_tile_assert (tile, update); +xe_tile_assert (tile, update->pt_bo); +xe_tile_assert (tile, !iosys_map_is_null (&update->pt_bo->vmap)); + +if (pt_op->bind) + ops->populate (tile, &update->pt_bo->vmap, +NULL , update->ofs, update->qwords, + update); +else + ops->clear (vm, tile, &update->pt_bo->vmap, +NULL , update->ofs, update->qwords, + update); + } + } + +trace_xe_vm_cpu_bind (vm); +xe_device_wmb (vm->xe); +} +static struct dma_fence *xe_migrate_update_pgtables_cpu (struct xe_migrate *m,struct xe_migrate_pt_update *pt_update)
@@ -1831,7 +1863,6 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,struct xe_vm_pgtable_update_ops *pt_update_ops = &pt_update->vops->pt_update_ops[pt_update->tile_id];int err; -u32 i, j;if (XE_TEST_ONLY (test && test->force_gpu))return ERR_PTR (-ETIME );
@@ -1843,28 +1874,9 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,return ERR_PTR (err); } -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -const struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; - -for (j =0 ; j < pt_op->num_entries; j++) { -const struct xe_vm_pgtable_update *update = - &pt_op->entries[j]; - -if (pt_op->bind) - ops->populate (m->tile, - &update->pt_bo->vmap,NULL , - update->ofs, update->qwords, - update); -else - ops->clear (vm, m->tile, - &update->pt_bo->vmap,NULL , - update->ofs, update->qwords, update); - } - } - -trace_xe_vm_cpu_bind (vm); -xe_device_wmb (vm->xe); +xe_migrate_update_pgtables_cpu_execute (vm, m->tile, ops, + pt_update_ops->ops, + pt_update_ops->num_ops);return dma_fence_get_stub (); }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 02/24] drm/xe: Add xe_migrate_update_pgtables_cpu_execute helper
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Structurally flawed assertions cause kernel oops and include tautological checks. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Structurally flawed assertions cause kernel oops and include tautological checks. --
commit 780cddb686fdf43f07c0012e23c36d21d368d194
Author: Matthew Brost <[email protected]>
drm/xe: Add xe_migrate_update_pgtables_cpu_execute helper
This commit extracts the CPU-side page-table update logic into a new helper function xe_migrate_update_pgtables_cpu_execute. This helper is intended to support implementing CPU binds when a bind job's dependencies are resolved. It also introduces basic sanity checks on the function arguments.
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0
[ ... ]
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +6
-0@@ -1819,6 +1819,38 @@ struct migrate_test_params { container_of(_priv, struct migrate_test_params, base)#endif +static void +xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, +const struct xe_migrate_pt_update_ops *ops, +struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops) +{
drivers/gpu/drm/xe/xe_migrate.c struct migrate_test_params { · line 1837
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +10
-0+u32 j, i; + +for (j =0 ; j < num_ops; ++j, ++pt_op) { +for (i =0 ; i < pt_op->num_entries; i++) { +const struct xe_vm_pgtable_update *update = + &pt_op->entries[i]; + +xe_tile_assert (tile, update); +xe_tile_assert (tile, update->pt_bo); +xe_tile_assert (tile, !iosys_map_is_null (&update->pt_bo->vmap));
[Severity: Medium]
Are these assertions structurally flawed?
First, update is derived from &pt_op->entries[i], meaning the address is evaluated inline and will never be NULL, making xe_tile_assert(tile, update) a tautology.
Second, if update->pt_bo is indeed NULL, thexe_tile_assert(tile, update->pt_bo) macro will warn but execution will continue. The subsequent assertion computes &update->pt_bo->vmap, yielding an invalid pointer (NULL + offset), and passes it to iosys_map_is_null(). This inline function dereferences the pointer, causing a kernel oops/panic. This defeats the purpose of the sanity checks, as an internal invariant violation will cause a hard crash.
Note that this code is moved to xe_cpu_bind_update_pgtables_execute in drivers/gpu/drm/xe/xe_cpu_bind.c by the later patch "drm/xe: Add CPU bind layer", but the structural flaw remains unfixed there.
Quoted 5 lines
drivers/gpu/drm/xe/xe_migrate.c +5
-0+ +if (pt_op->bind) + ops->populate (tile, &update->pt_bo->vmap, +NULL , update->ofs, update->qwords, + update);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Matthew Brost
maintainerPATCH v6 3/24 drm/xe: Decouple exec queue idle check from LRC
In reply to Matthew BrostWe already maintain a job count for each exec queue, so simplify the idle check to rely on the job count rather than the LRC state. This decouples exec queues from LRC-based backends and avoids unnecessarily coupling idle detection to backend-specific implementation details.
--- drivers/gpu/drm/xe/xe_exec_queue.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +1 -14
@@ -1580,20 +1580,7 @@ bool xe_exec_queue_is_lr(struct xe_exec_queue *q)*/ bool xe_exec_queue_is_idle (struct xe_exec_queue *q) { -if (xe_exec_queue_is_parallel (q)) { -int i; - -for (i =0 ; i < q->width; ++i) { -if (xe_lrc_seqno (q->lrc[i]) != - q->lrc[i]->fence_ctx.next_seqno -1 ) -return false; - } - -return true; - } - -return xe_lrc_seqno (q->lrc[0 ]) == - q->lrc[0 ]->fence_ctx.next_seqno -1 ; +return !atomic_read (&q->job_cnt); }/**
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 4/24 drm/xe: Add job count to GuC exec queue snapshot
In reply to Matthew BrostAdd the job count to the GuC exec queue snapshot, as this is useful debug information.
--- drivers/gpu/drm/xe/xe_guc_submit.c | 2 ++ drivers/gpu/drm/xe/xe_guc_submit_types.h | 2 ++ 2 files changed, 4 insertions(+)
drivers/gpu/drm/xe/xe_guc_submit.c +2 -0
@@ -3684,6 +3684,7 @@ xe_guc_exec_queue_snapshot_capture(struct xe_exec_queue *q) snapshot->logical_mask = q->logical_mask; snapshot->width = q->width; snapshot->refcount =kref_read (&q->refcount); + snapshot->jobcount =atomic_read (&q->job_cnt); snapshot->sched_timeout = sched->base.timeout; snapshot->sched_props.timeslice_us = q->sched_props.timeslice_us; snapshot->sched_props.preempt_timeout_us =
@@ -3756,6 +3757,7 @@ xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snapsdrm_printf (p,"\tLogical mask: 0x%x\n" , snapshot->logical_mask);drm_printf (p,"\tWidth: %d\n" , snapshot->width);drm_printf (p,"\tRef: %d\n" , snapshot->refcount); +drm_printf (p,"\tJob count: %d\n" , snapshot->jobcount);drm_printf (p,"\tTimeout: %ld (ms)\n" , snapshot->sched_timeout);drm_printf (p,"\tTimeslice: %u (us)\n" , snapshot->sched_props.timeslice_us);
drivers/gpu/drm/xe/xe_guc_submit_types.h +2 -0
@@ -77,6 +77,8 @@ struct xe_guc_submit_exec_queue_snapshot {u16 width;/** @refcount: ref count of this exec queue */ u32 refcount; +/** @jobcount: job count of this exec queue */ +u32 jobcount;/** * @sched_timeout: the time after which a job is removed from the * scheduler.
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 04/24] drm/xe: Add job count to GuC exec queue snapshot
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] Format specifier mismatch when printing `jobcount` --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] Format specifier mismatch when printing jobcount
--
commit bcb5e4e44e3808966c558f04a63143c713dbc759
Author: Matthew Brost <[email protected]>
drm/xe: Add job count to GuC exec queue snapshot
This commit adds the job count to the GuC exec queue snapshot to provide useful debugging information.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_guc_submit.c xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snaps · line 3760
Quoted 5 lines
drivers/gpu/drm/xe/xe_guc_submit.c +1
-0@@ -3756,6 +3757,7 @@ xe_guc_exec_queue_snapshot_print(struct xe_guc_submit_exec_queue_snapshot *snapsdrm_printf (p,"\tLogical mask: 0x%x\n" , snapshot->logical_mask);drm_printf (p,"\tWidth: %d\n" , snapshot->width);drm_printf (p,"\tRef: %d\n" , snapshot->refcount); +drm_printf (p,"\tJob count: %d\n" , snapshot->jobcount);
[Severity: Low]
Does this code use the wrong format specifier for jobcount?
In xe_guc_submit_types.h, jobcount is declared as an unsigned 32-bit integer (u32), but it is printed here in xe_guc_exec_queue_snapshot_print() using the signed %d format specifier instead of %u. While the maximum job count (1000) easily fits within a signed 32-bit integer, this formally violates the printf-style format string API constraints.
Quoted 3 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0drm_printf (p,"\tTimeout: %ld (ms)\n" , snapshot->sched_timeout);drm_printf (p,"\tTimeslice: %u (us)\n" , snapshot->sched_props.timeslice_us);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Matthew Brost
maintainerPATCH v6 5/24 drm/xe: Update xe_bo_put_deferred arguments to include writeback flag
In reply to Matthew BrostUpdate the xe_bo_put_deferred arguments to include a writeback flag, which indicates whether the BO was added to the deferred list. This is useful when the caller needs to take additional actions after the BO has been queued for deferred release.
--- drivers/gpu/drm/xe/xe_bo.h | 10 ++++++++-- drivers/gpu/drm/xe/xe_drm_client.c | 2 +- drivers/gpu/drm/xe/xe_pt.c | 2 +- 3 files changed, 10 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_bo.h +8 -2
@@ -495,6 +495,8 @@ void __xe_bo_release_dummy(struct kref *kref);* @bo: The bo to put. * @deferred: List to which to add the buffer object if we cannot put, or * NULL if the function is to put unconditionally. +* @added: BO was added to deferred list, written back to caller, can be NULL if +* writeback is not needed. Only set to true when added, never set to false. * * Since the final freeing of an object includes both sleeping and (!) * memory allocation in the dma_resv individualization, it's not ok
@@ -514,7 +516,8 @@ void __xe_bo_release_dummy(struct kref *kref);* false otherwise. */ static inline bool -xe_bo_put_deferred (struct xe_bo *bo,struct llist_head *deferred) +xe_bo_put_deferred (struct xe_bo *bo,struct llist_head *deferred, +bool *added) {if (!deferred) {xe_bo_put (bo);
@@ -524,6 +527,9 @@ xe_bo_put_deferred(struct xe_bo *bo, struct llist_head *deferred)if (!kref_put (&bo->ttm.base.refcount, __xe_bo_release_dummy))return false; +if (added) + *added = true; +return llist_add (&bo->freed, deferred); }
@@ -540,7 +546,7 @@ xe_bo_put_async(struct xe_bo *bo) {struct xe_bo_dev *bo_device = &xe_bo_device (bo)->bo_device; -if (xe_bo_put_deferred (bo, &bo_device->async_list)) +if (xe_bo_put_deferred (bo, &bo_device->async_list,NULL ))schedule_work (&bo_device->async_free); }
drivers/gpu/drm/xe/xe_drm_client.c +1 -1
@@ -256,7 +256,7 @@ static void show_meminfo(struct drm_printer *p, struct drm_file *file)xe_assert (xef->xe, !list_empty (&bo->client_link)); } -xe_bo_put_deferred (bo, &deferred); +xe_bo_put_deferred (bo, &deferred,NULL ); }spin_unlock (&client->bos_lock);
drivers/gpu/drm/xe/xe_pt.c +1 -1
@@ -213,7 +213,7 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred)XE_WARN_ON (!list_empty (&pt->bo->ttm.base.gpuva.list));xe_bo_unpin (pt->bo); -xe_bo_put_deferred (pt->bo, deferred); +xe_bo_put_deferred (pt->bo, deferred,NULL );if (pt->level >0 && pt->num_live) {struct xe_pt_dir *pt_dir =as_xe_pt_dir (pt);
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 6/24 drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
In reply to Matthew BrostAdd XE_BO_FLAG_PUT_VM_ASYNC, which indicates that an async BO put must also drop an additional reference to the BO’s VM. This is useful when a kernel BO, one that does not normally hold a VM reference, needs to be put asynchronously, ensuring the shared dma-resv object does not disappear before the BO.
--- drivers/gpu/drm/xe/xe_bo.c | 8 +++++++- drivers/gpu/drm/xe/xe_bo.h | 1 + 2 files changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_bo.c +7 -1
@@ -3998,8 +3998,14 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)return ; -llist_for_each_entry_safe (bo, next, freed, freed) +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; +bool async = bo->flags &XE_BO_FLAG_PUT_VM_ASYNC ; +drm_gem_object_free (&bo->ttm.base.refcount); +if (async) +xe_vm_put (vm); + } }static void xe_bo_dev_work_func (struct work_struct *work)
drivers/gpu/drm/xe/xe_bo.h +1 -0
@@ -54,6 +54,7 @@#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26)#define XE_BO_FLAG_NEEDS_1G BIT(27) +#define XE_BO_FLAG_PUT_VM_ASYNC BIT(28)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 06/24] drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Premature drop of VM reference in xe_bo_put_commit() allows the VM and its shared dma-resv to be…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Premature drop of VM reference in xe_bo_put_commit() allows the VM and its shared dma-resv to be freed while the BO is still alive in TTM, leading to a use-after-free on the resv. --
commit b2fad1ac33df0229950ed8b4d71c885827c4b7e3
Author: Matthew Brost <[email protected]>
drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
This commit adds a new flag, XE_BO_FLAG_PUT_VM_ASYNC, which indicates that an asynchronous BO put should also drop an additional reference to the BO's VM to prevent the shared dma-resv from disappearing before the BO.
Quoted 6 lines
drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -3998,8 +3998,14 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)
drivers/gpu/drm/xe/xe_bo.c void xe_bo_put_commit(struct llist_head *deferred) · line 4007
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +6
-1return ; -llist_for_each_entry_safe (bo, next, freed, freed) +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; +bool async = bo->flags &XE_BO_FLAG_PUT_VM_ASYNC ; +drm_gem_object_free (&bo->ttm.base.refcount); +if (async) +xe_vm_put (vm);
[Severity: High]
Does this sequence drop the VM reference prematurely and risk a use-after-free on the VM's dma_resv?
Looking at xe_bo_put_commit(), we call drm_gem_object_free(), which can trigger ttm_bo_release(). If ttm_bo_individualize_resv() fails due to memory pressure while the VM's shared dma_resv is concurrently locked by another thread, TTM handles this by queuing the BO for delayed deletion via ttm_bo_delayed_delete() and returning early.
If TTM returns early keeping the BO alive, and we immediately proceed to drop the VM reference here:
drm_gem_object_free(&bo->ttm.base.refcount);
if (async)
xe_vm_put(vm);could this free the VM and its embedded dma_resv if it was the last reference? Later, when ttm_bo_delayed_delete() executes, it would attempt to call dma_resv_lock(bo->base.resv) on the freed VM resv, causing a regression.
Is there a mechanism to ensure the VM stays alive as long as TTM delays the BO deletion?
Quoted 2 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+ } }
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Re: [PATCH v6 06/24] drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
In reply to Matthew BrostQuoted 20 lines · Matthew Brost
Add
XE_BO_FLAG_PUT_VM_ASYNC, which indicates that an async BO put must also drop an additional reference to the BO’s VM. This is useful when a kernel BO, one that does not normally hold a VM reference, needs to be put asynchronously, ensuring the shared dma-resv object does not disappear before the BO.Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 8 +++++++- drivers/gpu/drm/xe/xe_bo.h | 1 + 2 files changed, 8 insertions(+), 1 deletion(-)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -3998,8 +3998,14 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)
drivers/gpu/drm/xe/xe_bo.c void xe_bo_put_commit(struct llist_head *deferred) · line 4007
Quoted 10 lines
drivers/gpu/drm/xe/xe_bo.c +6
-1return ; -llist_for_each_entry_safe (bo, next, freed, freed) +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; +bool async = bo->flags &XE_BO_FLAG_PUT_VM_ASYNC ; +drm_gem_object_free (&bo->ttm.base.refcount); +if (async) +xe_vm_put (vm);
You had mentioned reworking this part:
https://patchwork.freedesktop.org/patch/750807/?series=149888&rev=6#comment_1383757
Sashiko seems consistent here.
Francois
Quoted 19 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+ } }static void xe_bo_dev_work_func (struct work_struct *work)drivers/gpu/drm/xe/xe_bo.h +1
-0@@ -54,6 +54,7 @@#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26)#define XE_BO_FLAG_NEEDS_1G BIT(27) +#define XE_BO_FLAG_PUT_VM_ASYNC BIT(28)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v6 06/24] drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
Quoted 23 lines · Francois Dugast
Quoted 22 lines · Matthew Brost
Add
XE_BO_FLAG_PUT_VM_ASYNC, which indicates that an async BO put must also drop an additional reference to the BO’s VM. This is useful when a kernel BO, one that does not normally hold a VM reference, needs to be put asynchronously, ensuring the shared dma-resv object does not disappear before the BO.Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 8 +++++++- drivers/gpu/drm/xe/xe_bo.h | 1 + 2 files changed, 8 insertions(+), 1 deletion(-)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -3998,8 +3998,14 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)return ;
Quoted 12 lines
drivers/gpu/drm/xe/xe_bo.c void xe_bo_put_commit(struct llist_head *deferred) · line 4007
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +6
-1-llist_for_each_entry_safe (bo, next, freed, freed) +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; +bool async = bo->flags &XE_BO_FLAG_PUT_VM_ASYNC ; +drm_gem_object_free (&bo->ttm.base.refcount); +if (async) +xe_vm_put (vm);You had mentioned reworking this part:
https://patchwork.freedesktop.org/patch/750807/?series=149888&rev=6#comment_1383757
I reworked the PT BO reference count reinitialization in the PT layer. The Sashiko feedback was not correct in its original context, but it was right that transitioning a BO reference count from zero to one on the driver side is unsafe.
This change follows patch [1] in xe_pt_destroy. Compare it to the prior revision [2]; note that the kref_init in [2] is gone in [1].
[1] https://patchwork.freedesktop.org/patch/751032/?series=149888&rev=8
[2] https://patchwork.freedesktop.org/patch/750792/?series=149888&rev=6
Wrt. the comment about ttm_bo_individualize_resv() failing, if that occurs in ttm_bo_release() (we get here from drm_gem_object_free()), the fences are waited on directly before returning. See the dma_resv_wait_timeout() for 30 seconds, after which the BO is destroyed inline if VM lock can be taken. I guess the VM lock can fall on a trylock, triggering the delayed worker...
We could look at PT BOs holding a VM reference until xe_ttm_bo_destroy(), similar to user BOs, but there is some convoluted reason why only user BOs have held a VM reference from early on in Xe that may or may not still be true. Let me see if I can make this work as after typing this out it seems like an extreme corner case of posisble memory safety bug.
Thanks,
Matt
Quoted 23 lines
Sashiko seems consistent here.
Francois
Quoted 19 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+ } }static void xe_bo_dev_work_func (struct work_struct *work)drivers/gpu/drm/xe/xe_bo.h +1
-0@@ -54,6 +54,7 @@#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26)#define XE_BO_FLAG_NEEDS_1G BIT(27) +#define XE_BO_FLAG_PUT_VM_ASYNC BIT(28)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v6 06/24] drm/xe: Add XE_BO_FLAG_PUT_VM_ASYNC
Quoted 55 lines · Matthew Brost
Quoted 23 lines · Francois Dugast
Quoted 22 lines · Matthew Brost
Add
XE_BO_FLAG_PUT_VM_ASYNC, which indicates that an async BO put must also drop an additional reference to the BO’s VM. This is useful when a kernel BO, one that does not normally hold a VM reference, needs to be put asynchronously, ensuring the shared dma-resv object does not disappear before the BO.Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_bo.c | 8 +++++++- drivers/gpu/drm/xe/xe_bo.h | 1 + 2 files changed, 8 insertions(+), 1 deletion(-)drivers/gpu/drm/xe/xe_bo.c +0
-0@@ -3998,8 +3998,14 @@ void xe_bo_put_commit(struct llist_head *deferred)if (!freed)return ;Quoted 12 lines
drivers/gpu/drm/xe/xe_bo.c void xe_bo_put_commit(struct llist_head *deferred) · line 4007
Quoted 8 lines
drivers/gpu/drm/xe/xe_bo.c +6
-1-llist_for_each_entry_safe (bo, next, freed, freed) +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_vm *vm = bo->vm; +bool async = bo->flags &XE_BO_FLAG_PUT_VM_ASYNC ; +drm_gem_object_free (&bo->ttm.base.refcount); +if (async) +xe_vm_put (vm);You had mentioned reworking this part:
https://patchwork.freedesktop.org/patch/750807/?series=149888&rev=6#comment_1383757I reworked the PT BO reference count reinitialization in the PT layer. The Sashiko feedback was not correct in its original context, but it was right that transitioning a BO reference count from zero to one on the driver side is unsafe.
This change follows patch [1] in
xe_pt_destroy. Compare it to the prior revision [2]; note that thekref_initin [2] is gone in [1].[1] https://patchwork.freedesktop.org/patch/751032/?series=149888&rev=8
[2] https://patchwork.freedesktop.org/patch/750792/?series=149888&rev=6Wrt. the comment about
ttm_bo_individualize_resv()failing, if that occurs inttm_bo_release()(we get here fromdrm_gem_object_free()), the fences are waited on directly before returning. See thedma_resv_wait_timeout()for 30 seconds, after which the BO is destroyed inline if VM lock can be taken. I guess the VM lock can fall on a trylock, triggering the delayed worker...
Quoted 5 lines
We could look at PT BOs holding a VM reference until
xe_ttm_bo_destroy(), similar to user BOs, but there is some convoluted reason why only user BOs have held a VM reference from early on in Xe that may or may not still be true. Let me see if I can make this work as after typing this out it seems like an extreme corner case of posisble memory safety bug.
It is also worth noting Christian's series here [3] is actually the proper way to ensure what Sashiko is flagging isn't possible - ref count the dma-resv. Hopefully we get Christian's series in soon as it nice a cleanup and makes dma-resv sharing betweens BOs inherently safe + ownership clear.
Matt
[3] https://patchwork.freedesktop.org/series/173312/
Quoted 27 lines
Thanks,
MattQuoted 23 lines
Sashiko seems consistent here.
Francois
Quoted 19 lines
drivers/gpu/drm/xe/xe_bo.c +1
-0+ } }static void xe_bo_dev_work_func (struct work_struct *work)drivers/gpu/drm/xe/xe_bo.h +1
-0@@ -54,6 +54,7 @@#define XE_BO_FLAG_FORCE_USER_VRAM BIT(25)#define XE_BO_FLAG_NO_COMPRESSION BIT(26)#define XE_BO_FLAG_NEEDS_1G BIT(27) +#define XE_BO_FLAG_PUT_VM_ASYNC BIT(28)/* this one is trigger internally only */ #define XE_BO_FLAG_INTERNAL_TEST BIT(30)Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 7/24 drm/xe: Update scheduler job layer to support PT jobs
In reply to Matthew BrostUpdate the scheduler job layer to support PT jobs. PT jobs are executed entirely on the CPU and do not require LRC fences or a batch address. Repurpose the LRC fence storage to hold PT‑job arguments and update the scheduler job layer to distinguish between PT jobs and jobs that require an LRC.
--- drivers/gpu/drm/xe/xe_sched_job.c | 95 ++++++++++++++++--------- drivers/gpu/drm/xe/xe_sched_job_types.h | 31 +++++++- drivers/gpu/drm/xe/xe_trace.h | 2 +- 3 files changed, 92 insertions(+), 36 deletions(-)
drivers/gpu/drm/xe/xe_sched_job.c +63 -32
@@ -26,19 +26,22 @@ static struct kmem_cache *xe_sched_job_parallel_slab;int __init xe_sched_job_module_init (void ) { +struct xe_sched_job *job; +size_t size; + + size =struct_size (job, ptrs,1 ); xe_sched_job_slab = -kmem_cache_create ("xe_sched_job" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ),0 , +kmem_cache_create ("xe_sched_job" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_slab)return -ENOMEM ; + size =max_t (size_t, +struct_size (job, ptrs, +XE_HW_ENGINE_MAX_INSTANCE ), +struct_size (job, pt_update,1 )); xe_sched_job_parallel_slab = -kmem_cache_create ("xe_sched_job_parallel" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ) * -XE_HW_ENGINE_MAX_INSTANCE ,0 , +kmem_cache_create ("xe_sched_job_parallel" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_parallel_slab) {kmem_cache_destroy (xe_sched_job_slab);
@@ -84,6 +87,9 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) {int i; +if (job->is_pt_job) +return ; +for (i =0 ; i < job->q->width; ++i) {struct xe_job_ptrs *ptrs = &job->ptrs[i];
@@ -93,10 +99,23 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) } } +/** +* xe_sched_job_create() - Create a scheduler job +* @q: exec queue to create the scheduler job for +* @batch_addr: array of batch addresses for the job; must match the width of +* @q, or NULL to indicate a PT job that does not require a batch address +* +* Create a scheduler job for submission. +* +* Context: Reclaim +* +* Return: a &xe_sched_job object on success, or an ERR_PTR on failure. +*/ struct xe_sched_job *xe_sched_job_create (struct xe_exec_queue *q,u64 *batch_addr) {bool is_migration =xe_sched_job_is_migration (q); +struct xe_device *xe =gt_to_xe (q->gt);struct xe_sched_job *job;int err;int i;
@@ -105,6 +124,9 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,/* only a kernel context can submit a vm-less job */ XE_WARN_ON (!q->vm && !(q->flags &EXEC_QUEUE_FLAG_KERNEL )); +xe_assert (xe, batch_addr || + q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); + job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration);if (!job)return ERR_PTR (-ENOMEM );
@@ -119,34 +141,39 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,if (err) goto err_free; -for (i =0 ; i < q->width; ++i) { -struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); -struct dma_fence_chain *chain; - -if (IS_ERR (fence)) { - err =PTR_ERR (fence); - goto err_sched_job; +if (!batch_addr) { + job->fence =dma_fence_get_stub (); + job->is_pt_job = true; + }else { +for (i =0 ; i < q->width; ++i) { +struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); +struct dma_fence_chain *chain; + +if (IS_ERR (fence)) { + err =PTR_ERR (fence); + goto err_sched_job; + } + job->ptrs[i].lrc_fence = fence; + +if (i +1 == q->width) +continue ; + + chain =dma_fence_chain_alloc (); +if (!chain) { + err = -ENOMEM ; + goto err_sched_job; + } + job->ptrs[i].chain_fence = chain; } - job->ptrs[i].lrc_fence = fence; -if (i +1 == q->width) -continue ; + width = q->width; +if (is_migration) + width =2 ; - chain =dma_fence_chain_alloc (); -if (!chain) { - err = -ENOMEM ; - goto err_sched_job; - } - job->ptrs[i].chain_fence = chain; +for (i =0 ; i < width; ++i) + job->ptrs[i].batch_addr = batch_addr[i]; } - width = q->width; -if (is_migration) - width =2 ; - -for (i =0 ; i < width; ++i) - job->ptrs[i].batch_addr = batch_addr[i]; -atomic_inc (&q->job_cnt);xe_pm_runtime_get_noresume (job_to_xe (job));trace_xe_sched_job_create (job);
@@ -246,7 +273,7 @@ bool xe_sched_job_completed(struct xe_sched_job *job)void xe_sched_job_arm (struct xe_sched_job *job) {struct xe_exec_queue *q = job->q; -struct dma_fence *fence, *prev; +struct dma_fence *fence = job->fence, *prev;struct xe_vm *vm = q->vm;u64 seqno =0 ;int i;
@@ -266,6 +293,9 @@ void xe_sched_job_arm(struct xe_sched_job *job) job->ring_ops_flush_tlb = true; } +if (job->is_pt_job) + goto arm; +/* Arm the pre-allocated fences */ for (i =0 ; i < q->width; prev = fence, ++i) {struct dma_fence_chain *chain;
@@ -286,6 +316,7 @@ void xe_sched_job_arm(struct xe_sched_job *job) fence = &chain->base; } +arm: job->fence =dma_fence_get (fence);/* Pairs with put in scheduler */ drm_sched_job_arm (&job->drm); }
@@ -329,7 +360,7 @@ xe_sched_job_snapshot_capture(struct xe_sched_job *job) snapshot->batch_addr_len = q->width;for (i =0 ; i < q->width; i++) - snapshot->batch_addr[i] = + snapshot->batch_addr[i] = job->is_pt_job ?0 :xe_device_uncanonicalize_addr (xe, job->ptrs[i].batch_addr);return snapshot;
drivers/gpu/drm/xe/xe_sched_job_types.h +28 -3
@@ -10,10 +10,29 @@#include <drm/gpu_scheduler.h> -struct xe_exec_queue ;struct dma_fence ;struct dma_fence_chain ; +struct xe_exec_queue ; +struct xe_migrate_pt_update_ops ; +struct xe_pt_job_ops ; +struct xe_tile ; +struct xe_vm ; + +/** +* struct xe_pt_update_args - PT update arguments +*/ +struct xe_pt_update_args { +/** @vm: VM which is being bound */ +struct xe_vm *vm; +/** @tile: Tile which page tables belong to */ +struct xe_tile *tile; +/** @ops: Migrate PT update ops */ +const struct xe_migrate_pt_update_ops *ops; +/** @pt_job_ops: PT job ops state */ +struct xe_pt_job_ops *pt_job_ops; +}; +/** * struct xe_job_ptrs - Per hw engine instance data */
@@ -71,8 +90,14 @@ struct xe_sched_job {bool restore_replay;/** @last_replay: last job being replayed */ bool last_replay; -/** @ptrs: per instance pointers. */ -struct xe_job_ptrs ptrs[]; +/** @is_pt_job: is a PT job */ +bool is_pt_job; +union { +/** @ptrs: per instance pointers. */ + DECLARE_FLEX_ARRAY(struct xe_job_ptrs, ptrs); +/** @pt_update: PT update arguments */ + DECLARE_FLEX_ARRAY(struct xe_pt_update_args, pt_update); + }; };struct xe_sched_job_snapshot {
drivers/gpu/drm/xe/xe_trace.h +1 -1
@@ -261,7 +261,7 @@ DECLARE_EVENT_CLASS(xe_sched_job, __entry->flags = job->q->flags; __entry->error = job->fence ? job->fence->error :0 ; __entry->fence = job->fence; - __entry->batch_addr = (u64 )job->ptrs[0 ].batch_addr; + __entry->batch_addr = job->is_pt_job ?0 : (u64 )job->ptrs[0 ].batch_addr; ), TP_printk("dev=%s, fence=%p, seqno=%u, lrc_seqno=%u, gt=%u, guc_id=%d, batch_addr=0x%012llx, guc_state=0x%x, flags=0x%x, error=%d",
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 07/24] drm/xe: Update scheduler job layer to support PT jobs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Critical] PT jobs consume the TLB flush seqno without executing a hardware TLB flush, potentially…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] PT jobs consume the TLB flush seqno without executing a hardware TLB flush, potentially causing subsequent GPU jobs to run with stale page tables. --
commit 3d489d315c4f278dad409a11a04984a2139095c2
Author: Matthew Brost <[email protected]>
drm/xe: Update scheduler job layer to support PT jobs
Update the scheduler job layer to support PT jobs which execute entirely on the CPU without requiring LRC fences or a batch address. Repurpose LRC fence storage for PT-job arguments.
Quoted 3 lines
drivers/gpu/drm/xe/xe_sched_job.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_sched_job.c void xe_sched_job_arm(struct xe_sched_job *job) · line 300
Quoted 9 lines
drivers/gpu/drm/xe/xe_sched_job.c +3
-0@@ -266,6 +293,9 @@ void xe_sched_job_arm(struct xe_sched_job *job) job->ring_ops_flush_tlb = true; } +if (job->is_pt_job) + goto arm; +/* Arm the pre-allocated fences */ for (i =0 ; i < q->width; prev = fence, ++i) {
[Severity: Critical]
Will this cause subsequent GPU jobs to run with stale page tables?
Looking at xe_sched_job_arm() in drivers/gpu/drm/xe/xe_sched_job.c, the code just above this addition checks if a TLB flush is needed and updates q->tlb_flush_seqno:
if (vm && !xe_sched_job_is_migration(q) && !xe_vm_in_lr_mode(vm) &&
(vm->batch_invalidate_tlb || vm->tlb_flush_seqno != q->tlb_flush_seqno)) {
xe_vm_assert_held(vm);
q->tlb_flush_seqno = vm->tlb_flush_seqno;
job->ring_ops_flush_tlb = true;
}If the current job is a PT job, the new check jumps to the arm label, skipping the hardware ring submission where the flush would normally execute. Because q->tlb_flush_seqno was already updated, when a subsequent GPU payload job is submitted to this same queue, it will see the seqnos match and skip flushing. This could allow the GPU to access memory using a stale TLB.
Should the job->is_pt_job check precede the TLB invalidation check to prevent the CPU-only PT job from stealing the flush notification?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Re: [PATCH v6 07/24] drm/xe: Update scheduler job layer to support PT jobs
In reply to Matthew BrostQuoted 166 lines · Matthew Brost
Update the scheduler job layer to support PT jobs. PT jobs are executed entirely on the CPU and do not require LRC fences or a batch address. Repurpose the LRC fence storage to hold PT‑job arguments and update the scheduler job layer to distinguish between PT jobs and jobs that require an LRC.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_sched_job.c | 95 ++++++++++++++++--------- drivers/gpu/drm/xe/xe_sched_job_types.h | 31 +++++++- drivers/gpu/drm/xe/xe_trace.h | 2 +- 3 files changed, 92 insertions(+), 36 deletions(-)drivers/gpu/drm/xe/xe_sched_job.c +58
-31@@ -26,19 +26,22 @@ static struct kmem_cache *xe_sched_job_parallel_slab;int __init xe_sched_job_module_init (void ) { +struct xe_sched_job *job; +size_t size; + + size =struct_size (job, ptrs,1 ); xe_sched_job_slab = -kmem_cache_create ("xe_sched_job" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ),0 , +kmem_cache_create ("xe_sched_job" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_slab)return -ENOMEM ; + size =max_t (size_t, +struct_size (job, ptrs, +XE_HW_ENGINE_MAX_INSTANCE ), +struct_size (job, pt_update,1 )); xe_sched_job_parallel_slab = -kmem_cache_create ("xe_sched_job_parallel" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ) * -XE_HW_ENGINE_MAX_INSTANCE ,0 , +kmem_cache_create ("xe_sched_job_parallel" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_parallel_slab) {kmem_cache_destroy (xe_sched_job_slab);@@ -84,6 +87,9 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) {int i; +if (job->is_pt_job) +return ; +for (i =0 ; i < job->q->width; ++i) {struct xe_job_ptrs *ptrs = &job->ptrs[i];@@ -93,10 +99,23 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) } } +/** +* xe_sched_job_create() - Create a scheduler job +* @q: exec queue to create the scheduler job for +* @batch_addr: array of batch addresses for the job; must match the width of +* @q, or NULL to indicate a PT job that does not require a batch address +* +* Create a scheduler job for submission. +* +* Context: Reclaim +* +* Return: a &xe_sched_job object on success, or an ERR_PTR on failure. +*/ struct xe_sched_job *xe_sched_job_create (struct xe_exec_queue *q,u64 *batch_addr) {bool is_migration =xe_sched_job_is_migration (q); +struct xe_device *xe =gt_to_xe (q->gt);struct xe_sched_job *job;int err;int i;@@ -105,6 +124,9 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,/* only a kernel context can submit a vm-less job */ XE_WARN_ON (!q->vm && !(q->flags &EXEC_QUEUE_FLAG_KERNEL )); +xe_assert (xe, batch_addr || + q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); + job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration);if (!job)return ERR_PTR (-ENOMEM );@@ -119,34 +141,39 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,if (err) goto err_free; -for (i =0 ; i < q->width; ++i) { -struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); -struct dma_fence_chain *chain; - -if (IS_ERR (fence)) { - err =PTR_ERR (fence); - goto err_sched_job; +if (!batch_addr) { + job->fence =dma_fence_get_stub (); + job->is_pt_job = true; + }else { +for (i =0 ; i < q->width; ++i) { +struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); +struct dma_fence_chain *chain; + +if (IS_ERR (fence)) { + err =PTR_ERR (fence); + goto err_sched_job; + } + job->ptrs[i].lrc_fence = fence; + +if (i +1 == q->width) +continue ; + + chain =dma_fence_chain_alloc (); +if (!chain) { + err = -ENOMEM ; + goto err_sched_job; + } + job->ptrs[i].chain_fence = chain; } - job->ptrs[i].lrc_fence = fence; -if (i +1 == q->width) -continue ; + width = q->width; +if (is_migration) + width =2 ; - chain =dma_fence_chain_alloc (); -if (!chain) { - err = -ENOMEM ; - goto err_sched_job; - } - job->ptrs[i].chain_fence = chain; +for (i =0 ; i < width; ++i) + job->ptrs[i].batch_addr = batch_addr[i]; } - width = q->width; -if (is_migration) - width =2 ; - -for (i =0 ; i < width; ++i) - job->ptrs[i].batch_addr = batch_addr[i]; -atomic_inc (&q->job_cnt);xe_pm_runtime_get_noresume (job_to_xe (job));trace_xe_sched_job_create (job);@@ -246,7 +273,7 @@ bool xe_sched_job_completed(struct xe_sched_job *job)void xe_sched_job_arm (struct xe_sched_job *job) {struct xe_exec_queue *q = job->q; -struct dma_fence *fence, *prev; +struct dma_fence *fence = job->fence, *prev;struct xe_vm *vm = q->vm;u64 seqno =0 ;int i;
drivers/gpu/drm/xe/xe_sched_job.c void xe_sched_job_arm(struct xe_sched_job *job) · line 298
Quoted 6 lines
drivers/gpu/drm/xe/xe_sched_job.c +3
-0@@ -266,6 +293,9 @@ void xe_sched_job_arm(struct xe_sched_job *job) job->ring_ops_flush_tlb = true; } +if (job->is_pt_job) + goto arm; +
I think Sashiko's warning is valid, we should move this ^ one block above to skip touching q->tlb_flush_seqno and ring_ops_flush_tlb for PT jobs.
Francois
Quoted 88 lines
drivers/gpu/drm/xe/xe_sched_job.c +2
-1/* Arm the pre-allocated fences */ for (i =0 ; i < q->width; prev = fence, ++i) {struct dma_fence_chain *chain;@@ -286,6 +316,7 @@ void xe_sched_job_arm(struct xe_sched_job *job) fence = &chain->base; } +arm: job->fence =dma_fence_get (fence);/* Pairs with put in scheduler */ drm_sched_job_arm (&job->drm); }@@ -329,7 +360,7 @@ xe_sched_job_snapshot_capture(struct xe_sched_job *job) snapshot->batch_addr_len = q->width;for (i =0 ; i < q->width; i++) - snapshot->batch_addr[i] = + snapshot->batch_addr[i] = job->is_pt_job ?0 :xe_device_uncanonicalize_addr (xe, job->ptrs[i].batch_addr);return snapshot;drivers/gpu/drm/xe/xe_sched_job_types.h +28
-3@@ -10,10 +10,29 @@#include <drm/gpu_scheduler.h> -struct xe_exec_queue ;struct dma_fence ;struct dma_fence_chain ; +struct xe_exec_queue ; +struct xe_migrate_pt_update_ops ; +struct xe_pt_job_ops ; +struct xe_tile ; +struct xe_vm ; + +/** +* struct xe_pt_update_args - PT update arguments +*/ +struct xe_pt_update_args { +/** @vm: VM which is being bound */ +struct xe_vm *vm; +/** @tile: Tile which page tables belong to */ +struct xe_tile *tile; +/** @ops: Migrate PT update ops */ +const struct xe_migrate_pt_update_ops *ops; +/** @pt_job_ops: PT job ops state */ +struct xe_pt_job_ops *pt_job_ops; +}; +/** * struct xe_job_ptrs - Per hw engine instance data */ @@ -71,8 +90,14 @@ struct xe_sched_job {bool restore_replay;/** @last_replay: last job being replayed */ bool last_replay; -/** @ptrs: per instance pointers. */ -struct xe_job_ptrs ptrs[]; +/** @is_pt_job: is a PT job */ +bool is_pt_job; +union { +/** @ptrs: per instance pointers. */ + DECLARE_FLEX_ARRAY(struct xe_job_ptrs, ptrs); +/** @pt_update: PT update arguments */ + DECLARE_FLEX_ARRAY(struct xe_pt_update_args, pt_update); + }; };struct xe_sched_job_snapshot {drivers/gpu/drm/xe/xe_trace.h +1
-1@@ -261,7 +261,7 @@ DECLARE_EVENT_CLASS(xe_sched_job, __entry->flags = job->q->flags; __entry->error = job->fence ? job->fence->error :0 ; __entry->fence = job->fence; - __entry->batch_addr = (u64 )job->ptrs[0 ].batch_addr; + __entry->batch_addr = job->is_pt_job ?0 : (u64 )job->ptrs[0 ].batch_addr; ), TP_printk("dev=%s, fence=%p, seqno=%u, lrc_seqno=%u, gt=%u, guc_id=%d, batch_addr=0x%012llx, guc_state=0x%x, flags=0x%x, error=%d",Signature · 1 line
2.34.1
Matthew Brost
maintainerRe: [PATCH v6 07/24] drm/xe: Update scheduler job layer to support PT jobs
Quoted 170 lines · Francois Dugast
Quoted 169 lines · Matthew Brost
Update the scheduler job layer to support PT jobs. PT jobs are executed entirely on the CPU and do not require LRC fences or a batch address. Repurpose the LRC fence storage to hold PT‑job arguments and update the scheduler job layer to distinguish between PT jobs and jobs that require an LRC.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_sched_job.c | 95 ++++++++++++++++--------- drivers/gpu/drm/xe/xe_sched_job_types.h | 31 +++++++- drivers/gpu/drm/xe/xe_trace.h | 2 +- 3 files changed, 92 insertions(+), 36 deletions(-)drivers/gpu/drm/xe/xe_sched_job.c +58
-31@@ -26,19 +26,22 @@ static struct kmem_cache *xe_sched_job_parallel_slab;int __init xe_sched_job_module_init (void ) { +struct xe_sched_job *job; +size_t size; + + size =struct_size (job, ptrs,1 ); xe_sched_job_slab = -kmem_cache_create ("xe_sched_job" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ),0 , +kmem_cache_create ("xe_sched_job" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_slab)return -ENOMEM ; + size =max_t (size_t, +struct_size (job, ptrs, +XE_HW_ENGINE_MAX_INSTANCE ), +struct_size (job, pt_update,1 )); xe_sched_job_parallel_slab = -kmem_cache_create ("xe_sched_job_parallel" , -sizeof (struct xe_sched_job ) + -sizeof (struct xe_job_ptrs ) * -XE_HW_ENGINE_MAX_INSTANCE ,0 , +kmem_cache_create ("xe_sched_job_parallel" , size,0 ,SLAB_HWCACHE_ALIGN ,NULL );if (!xe_sched_job_parallel_slab) {kmem_cache_destroy (xe_sched_job_slab);@@ -84,6 +87,9 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) {int i; +if (job->is_pt_job) +return ; +for (i =0 ; i < job->q->width; ++i) {struct xe_job_ptrs *ptrs = &job->ptrs[i];@@ -93,10 +99,23 @@ static void xe_sched_job_free_fences(struct xe_sched_job *job) } } +/** +* xe_sched_job_create() - Create a scheduler job +* @q: exec queue to create the scheduler job for +* @batch_addr: array of batch addresses for the job; must match the width of +* @q, or NULL to indicate a PT job that does not require a batch address +* +* Create a scheduler job for submission. +* +* Context: Reclaim +* +* Return: a &xe_sched_job object on success, or an ERR_PTR on failure. +*/ struct xe_sched_job *xe_sched_job_create (struct xe_exec_queue *q,u64 *batch_addr) {bool is_migration =xe_sched_job_is_migration (q); +struct xe_device *xe =gt_to_xe (q->gt);struct xe_sched_job *job;int err;int i;@@ -105,6 +124,9 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,/* only a kernel context can submit a vm-less job */ XE_WARN_ON (!q->vm && !(q->flags &EXEC_QUEUE_FLAG_KERNEL )); +xe_assert (xe, batch_addr || + q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); + job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration);if (!job)return ERR_PTR (-ENOMEM );@@ -119,34 +141,39 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,if (err) goto err_free; -for (i =0 ; i < q->width; ++i) { -struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); -struct dma_fence_chain *chain; - -if (IS_ERR (fence)) { - err =PTR_ERR (fence); - goto err_sched_job; +if (!batch_addr) { + job->fence =dma_fence_get_stub (); + job->is_pt_job = true; + }else { +for (i =0 ; i < q->width; ++i) { +struct dma_fence *fence =xe_lrc_alloc_seqno_fence (); +struct dma_fence_chain *chain; + +if (IS_ERR (fence)) { + err =PTR_ERR (fence); + goto err_sched_job; + } + job->ptrs[i].lrc_fence = fence; + +if (i +1 == q->width) +continue ; + + chain =dma_fence_chain_alloc (); +if (!chain) { + err = -ENOMEM ; + goto err_sched_job; + } + job->ptrs[i].chain_fence = chain; } - job->ptrs[i].lrc_fence = fence; -if (i +1 == q->width) -continue ; + width = q->width; +if (is_migration) + width =2 ; - chain =dma_fence_chain_alloc (); -if (!chain) { - err = -ENOMEM ; - goto err_sched_job; - } - job->ptrs[i].chain_fence = chain; +for (i =0 ; i < width; ++i) + job->ptrs[i].batch_addr = batch_addr[i]; } - width = q->width; -if (is_migration) - width =2 ; - -for (i =0 ; i < width; ++i) - job->ptrs[i].batch_addr = batch_addr[i]; -atomic_inc (&q->job_cnt);xe_pm_runtime_get_noresume (job_to_xe (job));trace_xe_sched_job_create (job);@@ -246,7 +273,7 @@ bool xe_sched_job_completed(struct xe_sched_job *job)void xe_sched_job_arm (struct xe_sched_job *job) {struct xe_exec_queue *q = job->q; -struct dma_fence *fence, *prev; +struct dma_fence *fence = job->fence, *prev;struct xe_vm *vm = q->vm;u64 seqno =0 ;int i;@@ -266,6 +293,9 @@ void xe_sched_job_arm(struct xe_sched_job *job) job->ring_ops_flush_tlb = true; }
Quoted 8 lines
drivers/gpu/drm/xe/xe_sched_job.c void xe_sched_job_arm(struct xe_sched_job *job) · line 298
Quoted 3 lines
drivers/gpu/drm/xe/xe_sched_job.c +3
-0+if (job->is_pt_job) + goto arm; +I think Sashiko's warning is valid, we should move this ^ one block above to skip touching q->tlb_flush_seqno and ring_ops_flush_tlb for PT jobs.
Those are completely unused for PT jobs / queues which run PT jobs so the ordering is harmless but for clarity it would be better. Will adjust.
Matt
Quoted 90 lines
Francois
Quoted 88 lines
drivers/gpu/drm/xe/xe_sched_job.c +2
-1/* Arm the pre-allocated fences */ for (i =0 ; i < q->width; prev = fence, ++i) {struct dma_fence_chain *chain;@@ -286,6 +316,7 @@ void xe_sched_job_arm(struct xe_sched_job *job) fence = &chain->base; } +arm: job->fence =dma_fence_get (fence);/* Pairs with put in scheduler */ drm_sched_job_arm (&job->drm); }@@ -329,7 +360,7 @@ xe_sched_job_snapshot_capture(struct xe_sched_job *job) snapshot->batch_addr_len = q->width;for (i =0 ; i < q->width; i++) - snapshot->batch_addr[i] = + snapshot->batch_addr[i] = job->is_pt_job ?0 :xe_device_uncanonicalize_addr (xe, job->ptrs[i].batch_addr);return snapshot;drivers/gpu/drm/xe/xe_sched_job_types.h +28
-3@@ -10,10 +10,29 @@#include <drm/gpu_scheduler.h> -struct xe_exec_queue ;struct dma_fence ;struct dma_fence_chain ; +struct xe_exec_queue ; +struct xe_migrate_pt_update_ops ; +struct xe_pt_job_ops ; +struct xe_tile ; +struct xe_vm ; + +/** +* struct xe_pt_update_args - PT update arguments +*/ +struct xe_pt_update_args { +/** @vm: VM which is being bound */ +struct xe_vm *vm; +/** @tile: Tile which page tables belong to */ +struct xe_tile *tile; +/** @ops: Migrate PT update ops */ +const struct xe_migrate_pt_update_ops *ops; +/** @pt_job_ops: PT job ops state */ +struct xe_pt_job_ops *pt_job_ops; +}; +/** * struct xe_job_ptrs - Per hw engine instance data */ @@ -71,8 +90,14 @@ struct xe_sched_job {bool restore_replay;/** @last_replay: last job being replayed */ bool last_replay; -/** @ptrs: per instance pointers. */ -struct xe_job_ptrs ptrs[]; +/** @is_pt_job: is a PT job */ +bool is_pt_job; +union { +/** @ptrs: per instance pointers. */ + DECLARE_FLEX_ARRAY(struct xe_job_ptrs, ptrs); +/** @pt_update: PT update arguments */ + DECLARE_FLEX_ARRAY(struct xe_pt_update_args, pt_update); + }; };struct xe_sched_job_snapshot {drivers/gpu/drm/xe/xe_trace.h +1
-1@@ -261,7 +261,7 @@ DECLARE_EVENT_CLASS(xe_sched_job, __entry->flags = job->q->flags; __entry->error = job->fence ? job->fence->error :0 ; __entry->fence = job->fence; - __entry->batch_addr = (u64 )job->ptrs[0 ].batch_addr; + __entry->batch_addr = job->is_pt_job ?0 : (u64 )job->ptrs[0 ].batch_addr; ), TP_printk("dev=%s, fence=%p, seqno=%u, lrc_seqno=%u, gt=%u, guc_id=%d, batch_addr=0x%012llx, guc_state=0x%x, flags=0x%x, error=%d",Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 8/24 drm/xe: Add helpers to access PT ops
In reply to Matthew BrostAdd helpers to access PT ops, making it easier to shuffle the location of the ops structures without requiring widespread code changes.
--- drivers/gpu/drm/xe/xe_pt.c | 65 ++++++++++++++++++++++++++------------ 1 file changed, 45 insertions(+), 20 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +45 -20
@@ -2078,13 +2078,37 @@ xe_pt_commit_prepare_unbind(struct xe_vma *vma, } } +static struct xe_vm_pgtable_update_op * +to_pt_op (struct xe_vm_pgtable_update_ops *pt_update_ops,u32 op_idx) +{ +return &pt_update_ops->ops[op_idx]; +} + +static u32 +get_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) +{ +return pt_update_ops->current_op; +} + +static struct xe_vm_pgtable_update_op * +to_current_pt_op (struct xe_vm_pgtable_update_ops *pt_update_ops) +{ +return to_pt_op (pt_update_ops,get_current_op (pt_update_ops)); +} + +static void +incr_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) +{ + ++pt_update_ops->current_op; +} +static void xe_pt_update_ops_rfence_interval (struct xe_vm_pgtable_update_ops *pt_update_ops,u64 start,u64 end) {u64 last; -u32 current_op = pt_update_ops->current_op; -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; +struct xe_vm_pgtable_update_op *pt_op = +to_current_pt_op (pt_update_ops);int i, level =0 ;for (i =0 ; i < pt_op->num_entries; i++) {
@@ -2119,8 +2143,8 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_vma *vma, bool invalidate_on_bind) { - u32 current_op = pt_update_ops->current_op; -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; + struct xe_vm_pgtable_update_op *pt_op = +to_current_pt_op (pt_update_ops);int err;xe_tile_assert (tile, !xe_vma_is_cpu_addr_mirror (vma));
@@ -2149,7 +2173,7 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,xe_pt_update_ops_rfence_interval (pt_update_ops,xe_vma_start (vma),xe_vma_end (vma)); - ++pt_update_ops->current_op; +incr_current_op (pt_update_ops); pt_update_ops->needs_svm_lock |=xe_vma_is_userptr (vma);/*
@@ -2192,8 +2216,8 @@ static int bind_range_prepare(struct xe_vm *vm, struct xe_tile *tile,struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_vma *vma, struct xe_svm_range *range) { - u32 current_op = pt_update_ops->current_op; -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; + struct xe_vm_pgtable_update_op *pt_op = +to_current_pt_op (pt_update_ops);int err;xe_tile_assert (tile,xe_vma_is_cpu_addr_mirror (vma));
@@ -2217,7 +2241,7 @@ static int bind_range_prepare(struct xe_vm *vm, struct xe_tile *tile,xe_pt_update_ops_rfence_interval (pt_update_ops,xe_svm_range_start (range),xe_svm_range_end (range)); - ++pt_update_ops->current_op; +incr_current_op (pt_update_ops); pt_update_ops->needs_svm_lock = true; pt_op->vma = vma;
@@ -2235,8 +2259,8 @@ static int unbind_op_prepare(struct xe_tile *tile, struct xe_vma *vma) {struct xe_device *xe =tile_to_xe (tile); -u32 current_op = pt_update_ops->current_op; -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; +struct xe_vm_pgtable_update_op *pt_op = +to_current_pt_op (pt_update_ops);int err;if (!((vma->tile_present | vma->tile_staged) &BIT (tile->id)))
@@ -2275,7 +2299,7 @@ static int unbind_op_prepare(struct xe_tile *tile, pt_op->num_entries, false);xe_pt_update_ops_rfence_interval (pt_update_ops,xe_vma_start (vma),xe_vma_end (vma)); - ++pt_update_ops->current_op; +incr_current_op (pt_update_ops); pt_update_ops->needs_svm_lock |=xe_vma_is_userptr (vma); pt_update_ops->needs_invalidation = true;
@@ -2315,8 +2339,8 @@ static int unbind_range_prepare(struct xe_vm *vm,struct xe_vm_pgtable_update_ops *pt_update_ops, struct xe_svm_range *range) { - u32 current_op = pt_update_ops->current_op; -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; + struct xe_vm_pgtable_update_op *pt_op = +to_current_pt_op (pt_update_ops);if (!(range->tile_present &BIT (tile->id)))return 0 ;
@@ -2337,7 +2361,7 @@ static int unbind_range_prepare(struct xe_vm *vm, pt_op->num_entries, false);xe_pt_update_ops_rfence_interval (pt_update_ops,xe_svm_range_start (range),xe_svm_range_end (range)); - ++pt_update_ops->current_op; +incr_current_op (pt_update_ops); pt_update_ops->needs_svm_lock = true; pt_update_ops->needs_invalidation |=xe_vm_has_scratch (vm) || xe_vm_has_valid_gpu_mapping(tile, range->tile_present,
@@ -2494,7 +2518,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)return err; } -xe_tile_assert (tile, pt_update_ops->current_op <= +xe_tile_assert (tile,get_current_op (pt_update_ops) <= pt_update_ops->num_ops);#ifdef TEST_VM_OPS_ERROR
@@ -2727,7 +2751,7 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)lockdep_assert_held (&vm->lock);xe_vm_assert_held (vm); -if (!pt_update_ops->current_op) { +if (!get_current_op (pt_update_ops)) {xe_tile_assert (tile,xe_vm_in_fault_mode (vm));return dma_fence_get_stub ();
@@ -2795,8 +2819,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) }/* Point of no return - VM killed if failure after this */ -for (i =0 ; i < pt_update_ops->current_op; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; +for (i =0 ; i <get_current_op (pt_update_ops); ++i) { +struct xe_vm_pgtable_update_op *pt_op = +to_pt_op (pt_update_ops, i);xe_pt_commit (pt_op->vma, pt_op->entries, pt_op->num_entries, &pt_update_ops->deferred);
@@ -2920,9 +2945,9 @@ void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops)for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; +to_pt_op (pt_update_ops, i); -if (!pt_op->vma || i >= pt_update_ops->current_op) +if (!pt_op->vma || i >=get_current_op (pt_update_ops))continue ;if (pt_op->bind)
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v6 9/24 drm/xe: Add struct xe_pt_job_ops
In reply to Matthew BrostAdd struct xe_pt_job_ops, a dynamically refcounted object that contains the information required to issue a CPU bind via a job after the initial bind IOCTL returns.
--- drivers/gpu/drm/xe/xe_migrate.c | 10 +-- drivers/gpu/drm/xe/xe_pt.c | 136 +++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_pt.h | 4 + drivers/gpu/drm/xe/xe_pt_types.h | 27 ++++-- drivers/gpu/drm/xe/xe_vm.c | 10 +-- 5 files changed, 152 insertions(+), 35 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +5 -5
@@ -1875,7 +1875,7 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m, }xe_migrate_update_pgtables_cpu_execute (vm, m->tile, ops, - pt_update_ops->ops, + pt_update_ops->pt_job_ops->ops, pt_update_ops->num_ops);return dma_fence_get_stub ();
@@ -1902,7 +1902,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,bool usm = is_migrate && xe->info.has_usm;for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; +struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries; num_updates += pt_op->num_entries;
@@ -1971,7 +1971,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,for (; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (; j < pt_op->num_entries; ++j, ++current_update, ++idx) {
@@ -2008,7 +2008,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m, (page_ofs /sizeof (u64)) *XE_PAGE_SIZE ;for (i =0 ; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (j =0 ; j < pt_op->num_entries; ++j) {
@@ -2026,7 +2026,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,for (i =0 ; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (j =0 ; j < pt_op->num_entries; ++j)
drivers/gpu/drm/xe/xe_pt.c +118 -18
188 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.h +4 -0
@@ -49,4 +49,8 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma);bool xe_pt_zap_ptes_range (struct xe_tile *tile,struct xe_vm *vm,struct xe_svm_range *range); +struct xe_pt_job_ops *xe_pt_job_ops_alloc (u32 num_ops); +struct xe_pt_job_ops *xe_pt_job_ops_get (struct xe_pt_job_ops *pt_job_ops); +void xe_pt_job_ops_put (struct xe_pt_job_ops *pt_job_ops); +#endif
drivers/gpu/drm/xe/xe_pt_types.h +21 -6
@@ -91,12 +91,29 @@ struct xe_vm_pgtable_update_op {bool rebind; }; +/** +* struct xe_pt_job_ops - Page-table update operations (dynamically allocated) +* +* This is the portion of &struct xe_vma_ops and +* &struct xe_vm_pgtable_update_ops that is dynamically allocated, as it +* must remain valid until the associated bind job completes. A reference +* count controls its lifetime. +*/ +struct xe_pt_job_ops { +/** @current_op: current page-table update operation */ +u32 current_op; +/** @refcount: reference count */ +struct kref refcount; +/** @deferred: list of deferred PT entries to destroy */ +struct llist_head deferred; +/** @ops: page-table update operations */ +struct xe_vm_pgtable_update_op *ops; +}; +/** struct xe_vm_pgtable_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops { -/** @ops: operations */ -struct xe_vm_pgtable_update_op *ops; -/** @deferred: deferred list to destroy PT entries */ -struct llist_head deferred; +/** @pt_job_ops: PT update operations dynamic allocation*/ +struct xe_pt_job_ops *pt_job_ops;/** @q: exec queue for PT operations */ struct xe_exec_queue *q;/** @prl: embedded page reclaim list */
@@ -107,8 +124,6 @@ struct xe_vm_pgtable_update_ops {u64 last;/** @num_ops: number of operations */ u32 num_ops; -/** @current_op: current operations */ -u32 current_op;/** @needs_svm_lock: Needs SVM lock */ bool needs_svm_lock;/** @needs_invalidation: Needs invalidation */
drivers/gpu/drm/xe/xe_vm.c +4 -6
@@ -681,11 +681,9 @@ static int xe_vma_ops_alloc(struct xe_vma_ops *vops, bool array_of_binds)if (!vops->pt_update_ops[i].num_ops)continue ; - vops->pt_update_ops[i].ops = -kmalloc_objs (*vops->pt_update_ops[i].ops, - vops->pt_update_ops[i].num_ops, -GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN); -if (!vops->pt_update_ops[i].ops) + vops->pt_update_ops[i].pt_job_ops = +xe_pt_job_ops_alloc (vops->pt_update_ops[i].num_ops); +if (!vops->pt_update_ops[i].pt_job_ops)return array_of_binds ? -ENOBUFS : -ENOMEM ; }
@@ -732,7 +730,7 @@ static void xe_vma_ops_fini(struct xe_vma_ops *vops)xe_vma_svm_prefetch_ops_fini (vops);for (i =0 ; i <XE_MAX_TILES_PER_DEVICE ; ++i) -kfree (vops->pt_update_ops[i].ops); +xe_pt_job_ops_put (vops->pt_update_ops[i].pt_job_ops); }static void xe_vma_ops_incr_pt_update_ops (struct xe_vma_ops *vops,u8 tile_mask,int inc_val)
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 09/24] drm/xe: Add struct xe_pt_job_ops
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing BO refcount re-initialization causes refcount underflow and memory leak during async free.…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing BO refcount re-initialization causes refcount underflow and memory leak during async free. --
commit 6e95dc31189c14b0582d03e7feb8e249921d75d3
Author: Matthew Brost <[email protected]>
Subject: drm/xe: Add struct xe_pt_job_ops
This commit introduces struct xe_pt_job_ops, a dynamically refcounted object containing the information required to issue a CPU bind via a job after the initial bind IOCTL returns.
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0
[ ... ]
Quoted 8 lines
drivers/gpu/drm/xe/xe_pt.c +3
-1@@ -213,7 +214,19 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred)XE_WARN_ON (!list_empty (&pt->bo->ttm.base.gpuva.list));xe_bo_unpin (pt->bo); -xe_bo_put_deferred (pt->bo, deferred,NULL ); +xe_bo_put_deferred (pt->bo, deferred, &added); +if (added) { +xe_assert (pt->bo->vm->xe, !kref_read (&pt->bo->ttm.base.refcount));
drivers/gpu/drm/xe/xe_pt.c void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred) · line 229
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +10
-0+ +/* +* We need the VM present until the BO is destroyed as it shares +* a dma-resv and BO destroy is async. Reinit BO refcount so +* xe_bo_put_async can be used when the PT job ops refcount goes +* to zero. +*/ +xe_vm_get (pt->bo->vm); + pt->bo->flags |=XE_BO_FLAG_PUT_VM_ASYNC ; + }
[Severity: High]
Does this code forget to actually re-initialize the BO refcount? The comment explicitly mentions reinitializing the BO refcount so xe_bo_put_async can be used, but it looks like only the VM is retrieved and the flag is set. Without a call to reinitialize the kref (like kref_init for pt->bo->ttm.base.refcount), the refcount remains at zero.
[ ... ]
Quoted 14 lines
drivers/gpu/drm/xe/xe_pt.c +14
-0+static void xe_pt_job_ops_destroy (struct kref *ref) +{ +struct xe_pt_job_ops *pt_job_ops = + container_of(ref, struct xe_pt_job_ops, refcount); +struct llist_node *freed; +struct xe_bo *bo, *next; + +xe_pt_update_ops_free (pt_job_ops->ops, + pt_job_ops->current_op); + + freed =llist_del_all (&pt_job_ops->deferred); +if (freed) { +llist_for_each_entry_safe (bo, next, freed, freed) { + struct xe_bo_dev *bo_device =
drivers/gpu/drm/xe/xe_pt.c void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops) · line 3045
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +10
-0+ &xe_bo_device (bo)->bo_device; +/* +* If called from run_job, we are in the dma-fencing +* path and cannot take dma-resv locks so use an async +* put. +*/ +if (llist_add (&bo->freed, &bo_device->async_list)) +schedule_work (&bo_device->async_free); + } + }
[Severity: High]
Because the BO refcount was left at zero during xe_pt_destroy, will the later xe_bo_put in the async_free worker trigger a refcount underflow here? A kref_put on a zero refcount returns false and skips the release function (such as drm_gem_object_free). Can this cause the page table BO memory to be permanently leaked?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Re: [PATCH v6 09/24] drm/xe: Add struct xe_pt_job_ops
In reply to Matthew BrostQuoted 81 lines · Matthew Brost
Add struct
xe_pt_job_ops, a dynamically refcounted object that contains the information required to issue a CPU bind via a job after the initial bind IOCTL returns.Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_migrate.c | 10 +-- drivers/gpu/drm/xe/xe_pt.c | 136 +++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_pt.h | 4 + drivers/gpu/drm/xe/xe_pt_types.h | 27 ++++-- drivers/gpu/drm/xe/xe_vm.c | 10 +-- 5 files changed, 152 insertions(+), 35 deletions(-)drivers/gpu/drm/xe/xe_migrate.c +5
-5@@ -1875,7 +1875,7 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m, }xe_migrate_update_pgtables_cpu_execute (vm, m->tile, ops, - pt_update_ops->ops, + pt_update_ops->pt_job_ops->ops, pt_update_ops->num_ops);return dma_fence_get_stub ();@@ -1902,7 +1902,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,bool usm = is_migrate && xe->info.has_usm;for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; +struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries; num_updates += pt_op->num_entries;@@ -1971,7 +1971,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,for (; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (; j < pt_op->num_entries; ++j, ++current_update, ++idx) {@@ -2008,7 +2008,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m, (page_ofs /sizeof (u64)) *XE_PAGE_SIZE ;for (i =0 ; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (j =0 ; j < pt_op->num_entries; ++j) {@@ -2026,7 +2026,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,for (i =0 ; i < pt_update_ops->num_ops; ++i) {struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->ops[i]; + &pt_update_ops->pt_job_ops->ops[i];struct xe_vm_pgtable_update *updates = pt_op->entries;for (j =0 ; j < pt_op->num_entries; ++j)drivers/gpu/drm/xe/xe_pt.c +1
-1@@ -206,6 +206,7 @@ unsigned int xe_pt_shift(unsigned int level)*/ void xe_pt_destroy (struct xe_pt *pt,u32 flags,struct llist_head *deferred) { +bool added = false;int i;if (!pt)@@ -213,7 +214,19 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred)XE_WARN_ON (!list_empty (&pt->bo->ttm.base.gpuva.list));xe_bo_unpin (pt->bo); -xe_bo_put_deferred (pt->bo, deferred,NULL );
drivers/gpu/drm/xe/xe_pt.c void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred) · line 226
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +10
-0+xe_bo_put_deferred (pt->bo, deferred, &added); +if (added) { +xe_assert (pt->bo->vm->xe, !kref_read (&pt->bo->ttm.base.refcount)); + +/* +* We need the VM present until the BO is destroyed as it shares +* a dma-resv and BO destroy is async. Reinit BO refcount so +* xe_bo_put_async can be used when the PT job ops refcount goes +* to zero. +*/
Seems BO refcount logic is from previous version and might be the reason prompting Sashiko comments.
Quoted 111 lines
drivers/gpu/drm/xe/xe_pt.c +45
-17+xe_vm_get (pt->bo->vm); + pt->bo->flags |=XE_BO_FLAG_PUT_VM_ASYNC ; + }if (pt->level >0 && pt->num_live) {struct xe_pt_dir *pt_dir =as_xe_pt_dir (pt);@@ -2081,13 +2094,13 @@ xe_pt_commit_prepare_unbind(struct xe_vma *vma,static struct xe_vm_pgtable_update_op *to_pt_op (struct xe_vm_pgtable_update_ops *pt_update_ops,u32 op_idx) { -return &pt_update_ops->ops[op_idx]; +return &pt_update_ops->pt_job_ops->ops[op_idx]; }static u32 get_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) { -return pt_update_ops->current_op; +return pt_update_ops->pt_job_ops->current_op; } static struct xe_vm_pgtable_update_op *@@ -2099,7 +2112,7 @@ to_current_pt_op(struct xe_vm_pgtable_update_ops *pt_update_ops)static void incr_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) { - ++pt_update_ops->current_op; + ++pt_update_ops->pt_job_ops->current_op; } static void@@ -2473,8 +2486,7 @@ static int op_prepare(struct xe_vm *vm,static void xe_pt_update_ops_init (struct xe_vm_pgtable_update_ops *pt_update_ops) { -init_llist_head (&pt_update_ops->deferred); - pt_update_ops->current_op =0 ; + pt_update_ops->pt_job_ops->current_op =0 ; pt_update_ops->start = ~0x0ull ; pt_update_ops->last =0x0ull ; pt_update_ops->needs_svm_lock = false;@@ -2824,7 +2836,8 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)to_pt_op (pt_update_ops, i);xe_pt_commit (pt_op->vma, pt_op->entries, - pt_op->num_entries, &pt_update_ops->deferred); + pt_op->num_entries, + &pt_update_ops->pt_job_ops->deferred); pt_op->vma =NULL ;/* skip in xe_pt_update_ops_abort */ }@@ -2912,19 +2925,8 @@ void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops) {struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[tile->id]; -int i;xe_page_reclaim_entries_put (pt_update_ops->prl.entries); - -lockdep_assert_held (&vops->vm->lock); -xe_vm_assert_held (vops->vm); - -for (i =0 ; i < pt_update_ops->current_op; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; - -xe_pt_free_bind (pt_op->entries, pt_op->num_entries); - } -xe_bo_put_commit (&vops->pt_update_ops[tile->id].deferred); }/** @@ -2961,3 +2963,101 @@ void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops)xe_pt_update_ops_fini (tile, vops); } + +/** +* xe_pt_job_ops_alloc() - Allocate PT job ops +* @num_ops: Number of VM PT update ops +* +* Allocate PT job ops and internal array of VM PT update ops. +* +* Return: Pointer to PT job ops or NULL +*/ +struct xe_pt_job_ops *xe_pt_job_ops_alloc (u32 num_ops) +{ +struct xe_pt_job_ops *pt_job_ops; + + pt_job_ops =kmalloc_obj (*pt_job_ops); +if (!pt_job_ops) +return NULL ; + + pt_job_ops->ops =kvmalloc_array (num_ops,sizeof (*pt_job_ops->ops), +GFP_KERNEL ); +if (!pt_job_ops->ops) { +kfree (pt_job_ops); +return NULL ; + } + + pt_job_ops->current_op =0 ; +kref_init (&pt_job_ops->refcount); +init_llist_head (&pt_job_ops->deferred); + +return pt_job_ops; +} + +/** +* xe_pt_job_ops_get() - Get PT job ops +* @pt_job_ops: PT job ops to get +* +* Take a reference to PT job ops
drivers/gpu/drm/xe/xe_pt.c void xe_pt_update_ops_abort(struct xe_tile *tile, struct xe_vma_ops *vops) · line 3011
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +10
-0+* +* Return: Pointer to PT job ops or NULL +*/ +struct xe_pt_job_ops *xe_pt_job_ops_get (struct xe_pt_job_ops *pt_job_ops) +{ +if (pt_job_ops) +kref_get (&pt_job_ops->refcount); + +return pt_job_ops; +}
Although unused in this patch, but seems correct place to introduce it.
Quoted 132 lines
drivers/gpu/drm/xe/xe_pt.c +52
-0+ +static void xe_pt_update_ops_free (struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops) +{ +u32 i; + +for (i =0 ; i < num_ops; ++i, ++pt_op) +xe_pt_free_bind (pt_op->entries, pt_op->num_entries); +} + +static void xe_pt_job_ops_destroy (struct kref *ref) +{ +struct xe_pt_job_ops *pt_job_ops = + container_of(ref, struct xe_pt_job_ops, refcount); +struct llist_node *freed; +struct xe_bo *bo, *next; + +xe_pt_update_ops_free (pt_job_ops->ops, + pt_job_ops->current_op); + + freed =llist_del_all (&pt_job_ops->deferred); +if (freed) { +llist_for_each_entry_safe (bo, next, freed, freed) { +struct xe_bo_dev *bo_device = + &xe_bo_device (bo)->bo_device; +/* +* If called from run_job, we are in the dma-fencing +* path and cannot take dma-resv locks so use an async +* put. +*/ +if (llist_add (&bo->freed, &bo_device->async_list)) +schedule_work (&bo_device->async_free); + } + } + +kvfree (pt_job_ops->ops); +kfree (pt_job_ops); +} + +/** +* xe_pt_job_ops_put() - Put PT job ops +* @pt_job_ops: PT job ops to put +* +* Drop a reference to PT job ops +*/ +void xe_pt_job_ops_put (struct xe_pt_job_ops *pt_job_ops) +{ +if (!pt_job_ops) +return ; + +kref_put (&pt_job_ops->refcount, xe_pt_job_ops_destroy); +}drivers/gpu/drm/xe/xe_pt.h +4
-0@@ -49,4 +49,8 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma);bool xe_pt_zap_ptes_range (struct xe_tile *tile,struct xe_vm *vm,struct xe_svm_range *range); +struct xe_pt_job_ops *xe_pt_job_ops_alloc (u32 num_ops); +struct xe_pt_job_ops *xe_pt_job_ops_get (struct xe_pt_job_ops *pt_job_ops); +void xe_pt_job_ops_put (struct xe_pt_job_ops *pt_job_ops); +#endif drivers/gpu/drm/xe/xe_pt_types.h +21
-6@@ -91,12 +91,29 @@ struct xe_vm_pgtable_update_op {bool rebind; }; +/** +* struct xe_pt_job_ops - Page-table update operations (dynamically allocated) +* +* This is the portion of &struct xe_vma_ops and +* &struct xe_vm_pgtable_update_ops that is dynamically allocated, as it +* must remain valid until the associated bind job completes. A reference +* count controls its lifetime. +*/ +struct xe_pt_job_ops { +/** @current_op: current page-table update operation */ +u32 current_op; +/** @refcount: reference count */ +struct kref refcount; +/** @deferred: list of deferred PT entries to destroy */ +struct llist_head deferred; +/** @ops: page-table update operations */ +struct xe_vm_pgtable_update_op *ops; +}; +/** struct xe_vm_pgtable_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops { -/** @ops: operations */ -struct xe_vm_pgtable_update_op *ops; -/** @deferred: deferred list to destroy PT entries */ -struct llist_head deferred; +/** @pt_job_ops: PT update operations dynamic allocation*/ +struct xe_pt_job_ops *pt_job_ops;/** @q: exec queue for PT operations */ struct xe_exec_queue *q;/** @prl: embedded page reclaim list */ @@ -107,8 +124,6 @@ struct xe_vm_pgtable_update_ops {u64 last;/** @num_ops: number of operations */ u32 num_ops; -/** @current_op: current operations */ -u32 current_op;/** @needs_svm_lock: Needs SVM lock */ bool needs_svm_lock;/** @needs_invalidation: Needs invalidation */ drivers/gpu/drm/xe/xe_vm.c +3
-5@@ -681,11 +681,9 @@ static int xe_vma_ops_alloc(struct xe_vma_ops *vops, bool array_of_binds)if (!vops->pt_update_ops[i].num_ops)continue ; - vops->pt_update_ops[i].ops = -kmalloc_objs (*vops->pt_update_ops[i].ops, - vops->pt_update_ops[i].num_ops, -GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN); -if (!vops->pt_update_ops[i].ops) + vops->pt_update_ops[i].pt_job_ops = +xe_pt_job_ops_alloc (vops->pt_update_ops[i].num_ops); +if (!vops->pt_update_ops[i].pt_job_ops)return array_of_binds ? -ENOBUFS : -ENOMEM ; }
drivers/gpu/drm/xe/xe_vm.c static void xe_vma_ops_fini(struct xe_vma_ops *vops) · line 733
Quoted 5 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -732,7 +730,7 @@ static void xe_vma_ops_fini(struct xe_vma_ops *vops)xe_vma_svm_prefetch_ops_fini (vops);for (i =0 ; i <XE_MAX_TILES_PER_DEVICE ; ++i) -kfree (vops->pt_update_ops[i].ops); +xe_pt_job_ops_put (vops->pt_update_ops[i].pt_job_ops);
With stale comment removal. Patch LGTM.
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0}static void xe_vma_ops_incr_pt_update_ops (struct xe_vma_ops *vops,u8 tile_mask,int inc_val)
Matthew Brost
maintainerPATCH v6 10/24 drm/xe: Update GuC submission backend to run PT jobs
In reply to Matthew BrostPT jobs bypass GPU execution for the final step of a bind job, using the CPU to program the required page tables. Teach the GuC submission backend how to execute these jobs.
PT job submission is implemented in the GuC backend for simplicity. A follow-up patch could introduce a dedicated backend for PT jobs.
--- drivers/gpu/drm/xe/xe_guc_submit.c | 37 ++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_migrate.c | 13 ++++++++++- drivers/gpu/drm/xe/xe_migrate.h | 7 ++++++ 3 files changed, 51 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_guc_submit.c +32 -5
@@ -38,9 +38,11 @@#include "xe_lrc.h" #include "xe_macros.h" #include "xe_map.h" +#include "xe_migrate.h" #include "xe_mocs.h" #include "xe_module.h" #include "xe_pm.h" +#include "xe_pt.h" #include "xe_ring_ops_types.h" #include "xe_sched_job.h" #include "xe_sleep.h"
@@ -1237,6 +1239,20 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job) } } +static bool is_pt_job (struct xe_sched_job *job) +{ +return job->is_pt_job; +} + +static void run_pt_job (struct xe_sched_job *job) +{ +xe_migrate_update_pgtables_cpu_execute (job->pt_update[0 ].vm, + job->pt_update[0 ].tile, + job->pt_update[0 ].ops, + job->pt_update[0 ].pt_job_ops->ops, + job->pt_update[0 ].pt_job_ops->current_op); +} +static struct dma_fence *guc_exec_queue_run_job (struct drm_sched_job *drm_job) {
@@ -1262,14 +1278,25 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)register_exec_queue (primary,GUC_CONTEXT_NORMAL ); } -if (!exec_queue_registered (q)) -register_exec_queue (q,GUC_CONTEXT_NORMAL ); -if (!job->restore_replay) - q->ring_ops->emit_job (job); -submit_exec_queue (q, job); +if (is_pt_job (job)) { +xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); +run_pt_job (job); + }else { +if (!exec_queue_registered (q)) +register_exec_queue (q,GUC_CONTEXT_NORMAL ); +if (!job->restore_replay) + q->ring_ops->emit_job (job); +submit_exec_queue (q, job); + } job->restore_replay = false; } +if (is_pt_job (job)) { +xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +dma_fence_put (job->fence);/* Drop ref from xe_sched_job_arm */ +return NULL ; + } + run_job_out:return job->fence;
drivers/gpu/drm/xe/xe_migrate.c +12 -1
@@ -1819,7 +1819,18 @@ struct migrate_test_params { container_of(_priv, struct migrate_test_params, base)#endif -static void +/** +* xe_migrate_update_pgtables_cpu_execute() - Update a VM's PTEs via the CPU +* @vm: The VM being updated +* @tile: The tile being updated +* @ops: The migrate PT update ops +* @pt_ops: The VM PT update ops +* @num_ops: The number of The VM PT update ops +* +* Execute the VM PT update ops array which results in a VM's PTEs being updated +* via the CPU. +*/ +void xe_migrate_update_pgtables_cpu_execute(struct xe_vm *vm, struct xe_tile *tile, const struct xe_migrate_pt_update_ops *ops, struct xe_vm_pgtable_update_op *pt_op,
drivers/gpu/drm/xe/xe_migrate.h +7 -0
@@ -24,6 +24,7 @@ struct xe_pt;struct xe_tile ;struct xe_vm ;struct xe_vm_pgtable_update ; +struct xe_vm_pgtable_update_op ;struct xe_vma ;enum xe_sriov_vf_ccs_rw_ctxs ;
@@ -163,6 +164,12 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,struct xe_vm *xe_migrate_get_vm (struct xe_migrate *m); +void +xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, +const struct xe_migrate_pt_update_ops *ops, +struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops); +struct dma_fence *xe_migrate_update_pgtables (struct xe_migrate *m,struct xe_migrate_pt_update *pt_update);
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 10/24] drm/xe: Update GuC submission backend to run PT jobs
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] PT jobs swallow dependency and cancellation errors, silently reporting false success and…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] PT jobs swallow dependency and cancellation errors, silently reporting false success and potentially leaving page tables in an inconsistent state.
- [High] Memory leak of xe_pt_job_ops and a dma_fence reference when a PT job is canceled or destroyed without being executed.
--
commit c6e167292efa7934a75cd29809e1057eae0d18c2
Author: Matthew Brost <[email protected]>
drm/xe: Update GuC submission backend to run PT jobs
PT jobs bypass GPU execution for the final step of a bind job, using the CPU to program the required page tables. Teach the GuC submission backend how to execute these jobs.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0
[ ... ]
Quoted 17 lines
drivers/gpu/drm/xe/xe_guc_submit.c +8
-5@@ -1262,14 +1278,25 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)register_exec_queue (primary,GUC_CONTEXT_NORMAL ); } -if (!exec_queue_registered (q)) -register_exec_queue (q,GUC_CONTEXT_NORMAL ); -if (!job->restore_replay) - q->ring_ops->emit_job (job); -submit_exec_queue (q, job); +if (is_pt_job (job)) { +xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); +run_pt_job (job); + }else { +if (!exec_queue_registered (q)) +register_exec_queue (q,GUC_CONTEXT_NORMAL ); +if (!job->restore_replay) + q->ring_ops->emit_job (job);
drivers/gpu/drm/xe/xe_guc_submit.c guc_exec_queue_run_job(struct drm_sched_job *drm_job) · line 1298
Quoted 10 lines
drivers/gpu/drm/xe/xe_guc_submit.c +7
-0+submit_exec_queue (q, job); + } job->restore_replay = false; } +if (is_pt_job (job)) { +xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +dma_fence_put (job->fence);/* Drop ref from xe_sched_job_arm */ +return NULL ; + }
[Severity: High]
Does this code leak the xe_pt_job_ops array and dma_fence objects when a PT job is canceled before execution?
Because the cleanup operations (xe_pt_job_ops_put() and the extra dma_fence_put()) are tied to job execution here in guc_exec_queue_run_job(), they appear to be missed if the job is destroyed before running.
Looking at xe_sched_job_destroy() in drivers/gpu/drm/xe/xe_sched_job.c:
void xe_sched_job_destroy(struct kref *ref)
{
...
xe_sched_job_free_fences(job);
dma_fence_put(job->fence);
drm_sched_job_cleanup(&job->drm);
job_free(job);
...
}If a PT job is armed, it takes a reference to pt_job_ops and an extra reference to job->fence. If the queue is killed and the job aborted before it executes, xe_sched_job_destroy() misses the xe_pt_job_ops_put() call and only drops one dma_fence_put() reference.
Can an unprivileged user repeatedly submit PT jobs via bind ioctls and immediately destroy the queue before execution to trigger this?
[Severity: Critical]
Does returning NULL unconditionally here swallow dependency and cancellation errors?
If a queue is banned, guc_exec_queue_run_job() skips run_pt_job() but still returns NULL here, which the DRM scheduler treats as a synchronous success.
Additionally, PT jobs are initialized with a stub fence:
struct xe_sched_job *xe_sched_job_create(...)
{
...
if (!batch_addr) {
job->fence = dma_fence_get_stub();
job->is_pt_job = true;
}
...
}Since a stub fence is permanently signaled with error = 0, xe_sched_job_set_error() fails to set any error state, and xe_sched_job_is_error() evaluates to false.
Can this result in page table updates silently failing while reporting success to userspace? For unbind operations, this could leave memory mapped in the GPU after it is freed, potentially allowing read or write access to reallocated memory from another queue in the same VM.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0run_job_out:return job->fence;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Re: [PATCH v6 10/24] drm/xe: Update GuC submission backend to run PT jobs
Quoted 69 lines · On 05-09-2026 03:09, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] PT jobs swallow dependency and cancellation errors, silently reporting false success and potentially leaving page tables in an inconsistent state.
- [High] Memory leak ofxe_pt_job_opsand adma_fencereference when a PT job is canceled or destroyed without being executed.
--commit
c6e167292efa7934a75cd29809e1057eae0d18c2
Author: Matthew Brost <[email protected]>drm/xe: Update GuC submission backend to run PT jobs
PT jobs bypass GPU execution for the final step of a bind job, using the CPU to program the required page tables. Teach the GuC submission backend how to execute these jobs.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0[ ... ]
Quoted 17 lines
drivers/gpu/drm/xe/xe_guc_submit.c +8
-5@@ -1262,14 +1278,25 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)register_exec_queue (primary,GUC_CONTEXT_NORMAL ); } -if (!exec_queue_registered (q)) -register_exec_queue (q,GUC_CONTEXT_NORMAL ); -if (!job->restore_replay) - q->ring_ops->emit_job (job); -submit_exec_queue (q, job); +if (is_pt_job (job)) { +xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); +run_pt_job (job); + }else { +if (!exec_queue_registered (q)) +register_exec_queue (q,GUC_CONTEXT_NORMAL ); +if (!job->restore_replay) + q->ring_ops->emit_job (job);drivers/gpu/drm/xe/xe_guc_submit.c guc_exec_queue_run_job(struct drm_sched_job *drm_job) · line 1298
Quoted 10 lines
drivers/gpu/drm/xe/xe_guc_submit.c +7
-0+submit_exec_queue (q, job); + } job->restore_replay = false; } +if (is_pt_job (job)) { +xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +dma_fence_put (job->fence);/* Drop ref from xe_sched_job_arm */ +return NULL ; + }[Severity: High]
Does this code leak thexe_pt_job_opsarray anddma_fenceobjects when a PT job is canceled before execution?Because the cleanup operations (
xe_pt_job_ops_put()and the extradma_fence_put()) are tied to job execution here inguc_exec_queue_run_job(), they appear to be missed if the job is destroyed before running.Looking at
xe_sched_job_destroy()indrivers/gpu/drm/xe/xe_sched_job.c:void
xe_sched_job_destroy(struct kref *ref){ ... xe_sched_job_free_fences(job); dma_fence_put(job->fence); drm_sched_job_cleanup(&job->drm); job_free(job); ... }
Quoted 7 lines
If a PT job is armed, it takes a reference to
pt_job_opsand an extra reference tojob->fence. If the queue is killed and the job aborted before it executes,xe_sched_job_destroy()misses thexe_pt_job_ops_put()call and only drops onedma_fence_put()reference.Can an unprivileged user repeatedly submit PT jobs via bind ioctls and immediately destroy the queue before execution to trigger this?
Seems really a valid comment. better to handle xe_pt_job_ops_put and dma_fence_put in guc_exec_queue_free_job ?
Quoted 4 lines
[Severity: Critical]
Does returning NULL unconditionally here swallow dependency and cancellation errors?
How about we return -ECANCELLED for banned jobs here in guc_exec_queue_run_job?
Quoted 30 lines
If a queue is banned,
guc_exec_queue_run_job()skipsrun_pt_job()but still returns NULL here, which the DRM scheduler treats as a synchronous success.Additionally, PT jobs are initialized with a stub fence:
struct
xe_sched_job*xe_sched_job_create(...){ ... if (!batch_addr) { job->fence = dma_fence_get_stub(); job->is_pt_job = true; } ... }Since a stub fence is permanently signaled with error = 0,
xe_sched_job_set_error()fails to set any error state, andxe_sched_job_is_error()evaluates to false.Can this result in page table updates silently failing while reporting success to userspace? For unbind operations, this could leave memory mapped in the GPU after it is freed, potentially allowing read or write access to reallocated memory from another queue in the same VM.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0run_job_out:return job->fence;
Matthew Brost
maintainerRe: [PATCH v6 10/24] drm/xe: Update GuC submission backend to run PT jobs
Quoted 75 lines · Himal Prasad
Quoted 72 lines · On 05-09-2026 03:09, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] PT jobs swallow dependency and cancellation errors, silently reporting false success and potentially leaving page tables in an inconsistent state.
- [High] Memory leak ofxe_pt_job_opsand adma_fencereference when a PT job is canceled or destroyed without being executed.
--commit
c6e167292efa7934a75cd29809e1057eae0d18c2
Author: Matthew Brost <[email protected]>drm/xe: Update GuC submission backend to run PT jobs
PT jobs bypass GPU execution for the final step of a bind job, using the CPU to program the required page tables. Teach the GuC submission backend how to execute these jobs.
Quoted 4 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0[ ... ]
Quoted 15 lines
drivers/gpu/drm/xe/xe_guc_submit.c +7
-5@@ -1262,14 +1278,25 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)register_exec_queue (primary,GUC_CONTEXT_NORMAL ); } -if (!exec_queue_registered (q)) -register_exec_queue (q,GUC_CONTEXT_NORMAL ); -if (!job->restore_replay) - q->ring_ops->emit_job (job); -submit_exec_queue (q, job); +if (is_pt_job (job)) { +xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); +run_pt_job (job); + }else { +if (!exec_queue_registered (q)) +register_exec_queue (q,GUC_CONTEXT_NORMAL ); +if (!job->restore_replay)drivers/gpu/drm/xe/xe_guc_submit.c guc_exec_queue_run_job(struct drm_sched_job *drm_job) · line 1296
Quoted 10 lines
drivers/gpu/drm/xe/xe_guc_submit.c +8
-0+ q->ring_ops->emit_job (job); +submit_exec_queue (q, job); + } job->restore_replay = false; } +if (is_pt_job (job)) { +xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +dma_fence_put (job->fence);/* Drop ref from xe_sched_job_arm */ +return NULL ; + }[Severity: High]
Does this code leak thexe_pt_job_opsarray anddma_fenceobjects when a PT job is canceled before execution?Because the cleanup operations (
xe_pt_job_ops_put()and the extradma_fence_put()) are tied to job execution here inguc_exec_queue_run_job(), they appear to be missed if the job is destroyed before running.Looking at
xe_sched_job_destroy()indrivers/gpu/drm/xe/xe_sched_job.c:void
xe_sched_job_destroy(struct kref *ref){ ... xe_sched_job_free_fences(job); dma_fence_put(job->fence); drm_sched_job_cleanup(&job->drm); job_free(job); ... }If a PT job is armed, it takes a reference to
pt_job_opsand an extra reference tojob->fence. If the queue is killed and the job aborted before it executes,xe_sched_job_destroy()misses thexe_pt_job_ops_put()call and only drops onedma_fence_put()reference.
Quoted 6 lines
Quoted 2 lines
Can an unprivileged user repeatedly submit PT jobs via bind ioctls and immediately destroy the queue before execution to trigger this?
Seems really a valid comment. better to handle
xe_pt_job_ops_putanddma_fence_putinguc_exec_queue_free_job?
I can move this but the comment is invalid. Once arm is called, run_job is always called - i.e., even if a queue is closed we still get a run_job(). This is existing bookkeeping that certainly works.
Matt
Quoted 38 lines
Quoted 4 lines
[Severity: Critical]
Does returning NULL unconditionally here swallow dependency and cancellation errors?How about we return
-ECANCELLEDfor banned jobs here inguc_exec_queue_run_job?Quoted 28 lines
If a queue is banned,
guc_exec_queue_run_job()skipsrun_pt_job()but still returns NULL here, which the DRM scheduler treats as a synchronous success.Additionally, PT jobs are initialized with a stub fence:
struct
xe_sched_job*xe_sched_job_create(...){ ... if (!batch_addr) { job->fence = dma_fence_get_stub(); job->is_pt_job = true; } ... }Since a stub fence is permanently signaled with error = 0,
xe_sched_job_set_error()fails to set any error state, andxe_sched_job_is_error()evaluates to false.Can this result in page table updates silently failing while reporting success to userspace? For unbind operations, this could leave memory mapped in the GPU after it is freed, potentially allowing read or write access to reallocated memory from another queue in the same VM.
Quoted 2 lines
drivers/gpu/drm/xe/xe_guc_submit.c +0
-0run_job_out:return job->fence;
Matthew Brost
maintainerPATCH v6 11/24 drm/xe: Store level in struct xe_vm_pgtable_update
In reply to Matthew BrostThe level was previously extracted from struct xe_pt inside xe_vm_pgtable_update during CPU binds, which always occurred during the bind IOCTL. With CPU binds now supported in bind jobs, struct xe_pt may no longer be valid in memory at that point. To address this, store the level directly in struct xe_vm_pgtable_update.
--- drivers/gpu/drm/xe/xe_pt.c | 3 ++- drivers/gpu/drm/xe/xe_pt_types.h | 8 +++++++- 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +2 -1
@@ -388,6 +388,7 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, entry->flags =0 ; entry->qwords =0 ; entry->pt_bo->update_index =-1 ; + entry->level = parent->level;if (alloc_entries) { entry->pt_entries =kmalloc_objs (*entry->pt_entries,XE_PDES );
@@ -2025,7 +2026,7 @@ xe_migrate_clear_pgtable_callback(struct xe_vm *vm, struct xe_tile *tile, u32 qword_ofs, u32 num_qwords, const struct xe_vm_pgtable_update *update) { - u64 empty = __xe_pt_empty_pte(tile, vm, update->pt->level); + u64 empty = __xe_pt_empty_pte(tile, vm, update->level);int i;if (map && map->is_iomem)
drivers/gpu/drm/xe/xe_pt_types.h +7 -1
@@ -65,12 +65,18 @@ struct xe_vm_pgtable_update {/** @qwords: number of PTE's to write */ u32 qwords; -/** @pt: opaque pointer useful for the caller of xe_migrate_update_pgtables */ +/** +* @pt: opaque pointer useful for PT building in the bind IOCTL. Only +* safe to touch during the bind IOCTL (i.e., not in bind jobs). +*/ struct xe_pt *pt;/** @pt_entries: Newly added pagetable entries */ struct xe_pt_entry *pt_entries; +/** @level: level of update */ +unsigned int level; +/** @flags: Target flags */ u32 flags; };
Signature · 1 line
2.34.1
Re: [PATCH v6 11/24] drm/xe: Store level in struct xe_vm_pgtable_update
Quoted 47 lines · Matthew Brost
The level was previously extracted from struct
xe_ptinsidexe_vm_pgtable_updateduring CPU binds, which always occurred during the bind IOCTL. With CPU binds now supported in bind jobs, structxe_ptmay no longer be valid in memory at that point. To address this, store the level directly in structxe_vm_pgtable_update.Signed-off-by Matthew Brost <[email protected]>Reviewed-by Stuart Summers <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_pt.c | 3 ++- drivers/gpu/drm/xe/xe_pt_types.h | 8 +++++++- 2 files changed, 9 insertions(+), 2 deletions(-)drivers/gpu/drm/xe/xe_pt.c +2
-1@@ -388,6 +388,7 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, entry->flags =0 ; entry->qwords =0 ; entry->pt_bo->update_index =-1 ; + entry->level = parent->level;if (alloc_entries) { entry->pt_entries =kmalloc_objs (*entry->pt_entries,XE_PDES );@@ -2025,7 +2026,7 @@ xe_migrate_clear_pgtable_callback(struct xe_vm *vm, struct xe_tile *tile, u32 qword_ofs, u32 num_qwords, const struct xe_vm_pgtable_update *update) { - u64 empty = __xe_pt_empty_pte(tile, vm, update->pt->level); + u64 empty = __xe_pt_empty_pte(tile, vm, update->level);int i;if (map && map->is_iomem)drivers/gpu/drm/xe/xe_pt_types.h +2
-1@@ -65,12 +65,18 @@ struct xe_vm_pgtable_update {/** @qwords: number of PTE's to write */ u32 qwords; -/** @pt: opaque pointer useful for the caller of xe_migrate_update_pgtables */ +/** +* @pt: opaque pointer useful for PT building in the bind IOCTL. Only
drivers/gpu/drm/xe/xe_pt_types.h struct xe_vm_pgtable_update { · line 79
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt_types.h +5
-0+* safe to touch during the bind IOCTL (i.e., not in bind jobs). +*/ struct xe_pt *pt;/** @pt_entries: Newly added pagetable entries */ struct xe_pt_entry *pt_entries; +/** @level: level of update */ +unsigned int level; +
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt_types.h +0
-0/** @flags: Target flags */ u32 flags; };
Matthew Brost
maintainerPATCH v6 12/24 drm/xe: Don't use migrate exec queue for page fault binds
In reply to Matthew BrostNow that the CPU is always used for binds even in jobs, CPU bind jobs can pass GPU jobs in the same exec queue resulting dma-fences signaling out-of-order. Use a dedicated exec queue for binds issued from page faults to avoid ordering issues and avoid blocking kernel binds on unrelated copies / clears.
--- drivers/gpu/drm/xe/xe_migrate.c | 47 ++++++++++++++++++++++++++++++--- drivers/gpu/drm/xe/xe_migrate.h | 1 + drivers/gpu/drm/xe/xe_vm.c | 17 +++++++----- 3 files changed, 55 insertions(+), 10 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +43 -4
@@ -51,6 +51,8 @@struct xe_migrate {/** @q: Default exec queue used for migration */ struct xe_exec_queue *q; +/** @bind_q: Default exec queue used for binds */ +struct xe_exec_queue *bind_q;/** @tile: Backpointer to the tile this struct xe_migrate belongs to. */ struct xe_tile *tile;/** @job_mutex: Timeline mutex for @eng. */
@@ -115,6 +117,7 @@ static void xe_migrate_fini(void *arg)mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q); +xe_exec_queue_put (m->bind_q); } static inline u16 xe_migrate_pat_index(struct xe_device *xe,
@@ -504,6 +507,15 @@ int xe_migrate_init(struct xe_migrate *m) goto err_out; } + m->bind_q =xe_exec_queue_create (xe, vm, logical_mask,1 , hwe0, +EXEC_QUEUE_FLAG_KERNEL | +EXEC_QUEUE_FLAG_HIGH_PRIORITY | +EXEC_QUEUE_FLAG_MIGRATE ,0 ); +if (IS_ERR (m->bind_q)) { + err =PTR_ERR (m->bind_q); + goto err_out; + } +/* * XXX: Currently only reserving 1 (likely slow) BCS instance on * PVC, may want to revisit if performance is needed.
@@ -514,6 +526,15 @@ int xe_migrate_init(struct xe_migrate *m)EXEC_QUEUE_FLAG_MIGRATE | EXEC_QUEUE_FLAG_LOW_LATENCY, 0); }else { + m->bind_q =xe_exec_queue_create_class (xe, primary_gt, vm, +XE_ENGINE_CLASS_COPY , +EXEC_QUEUE_FLAG_KERNEL | +EXEC_QUEUE_FLAG_MIGRATE ,0 ); +if (IS_ERR (m->bind_q)) { + err =PTR_ERR (m->bind_q); + goto err_out; + } + m->q = xe_exec_queue_create_class(xe, primary_gt, vm, XE_ENGINE_CLASS_COPY, EXEC_QUEUE_FLAG_KERNEL |
@@ -549,6 +570,8 @@ int xe_migrate_init(struct xe_migrate *m)return err; err_out: +if (!IS_ERR_OR_NULL (m->bind_q)) +xe_exec_queue_put (m->bind_q);xe_vm_close_and_put (vm);return err;
@@ -1505,6 +1528,17 @@ static u32 blt_mem_set_cmd_len(struct xe_device *xe)return 7 ; } +/** +* xe_get_migrate_bind_queue() - Get the bind queue from migrate context. +* @migrate: Migrate context. +* +* Return: Pointer to bind queue on success, error on failure +*/ +struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate) +{ +return migrate->bind_q; +} +static void emit_clear_link_copy (struct xe_gt *gt,struct xe_bb *bb,u64 src_ofs,u32 size,u32 pitch) {
@@ -1892,6 +1926,11 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,return dma_fence_get_stub (); } +static bool is_migrate_queue (struct xe_migrate *m,struct xe_exec_queue *q) +{ +return m->bind_q == q; +} +static struct dma_fence * __xe_migrate_update_pgtables(struct xe_migrate *m, struct xe_migrate_pt_update *pt_update,
@@ -1909,7 +1948,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,u32 num_updates =0 , current_update =0 ;u64 addr;int err =0 ; -bool is_migrate = pt_update_ops->q == m->q; +bool is_migrate =is_migrate_queue (m, pt_update_ops->q);bool usm = is_migrate && xe->info.has_usm;for (i =0 ; i < pt_update_ops->num_ops; ++i) {
@@ -2631,7 +2670,7 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,*/ void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q) { -bool is_migrate = q == m->q; +bool is_migrate =is_migrate_queue (m, q);if (is_migrate)mutex_lock (&m->job_mutex);
@@ -2649,7 +2688,7 @@ void xe_migrate_job_lock(struct xe_migrate *m, struct xe_exec_queue *q)*/ void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q) { -bool is_migrate = q == m->q; +bool is_migrate =is_migrate_queue (m, q);if (is_migrate)mutex_unlock (&m->job_mutex);
@@ -2666,7 +2705,7 @@ void xe_migrate_job_lock_assert(struct xe_exec_queue *q) {struct xe_migrate *m =gt_to_tile (q->gt)->migrate; -xe_gt_assert (q->gt, q == m->q); +xe_gt_assert (q->gt, q == m->bind_q);lockdep_assert_held (&m->job_mutex); }#endif
drivers/gpu/drm/xe/xe_migrate.h +1 -0
@@ -146,6 +146,7 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,struct xe_lrc *xe_migrate_lrc (struct xe_migrate *migrate);struct xe_exec_queue *xe_migrate_exec_queue (struct xe_migrate *migrate); +struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate);struct dma_fence *xe_migrate_vram_copy_chunk (struct xe_bo *vram_bo,u64 vram_offset,struct xe_bo *sysmem_bo,u64 sysmem_offset,u64 size,enum xe_migrate_copy_dir dir);
drivers/gpu/drm/xe/xe_vm.c +11 -6
@@ -795,7 +795,9 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)struct xe_vma *vma, *next;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -int err, i; +struct xe_tile *tile; +u8 id; +int err;lockdep_assert_held (&vm->lock); if ((xe_vm_in_lr_mode(vm) && !rebind_worker) ||
@@ -803,8 +805,11 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)return 0 ;xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for (i =0 ; i <XE_MAX_TILES_PER_DEVICE ; ++i) - vops.pt_update_ops[i].wait_vm_bookkeep = true; +for_each_tile (tile, vm->xe, id) { + vops.pt_update_ops[id].wait_vm_bookkeep = true; + vops.pt_update_ops[id].q = +xe_migrate_bind_queue (tile->migrate); + }xe_vm_assert_held (vm);list_for_each_entry (vma, &vm->rebind_list, combined_links.rebind) {
@@ -862,7 +867,7 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_mafor_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); } err =xe_vm_ops_add_rebind (&vops, vma, tile_mask);
@@ -954,7 +959,7 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm,for_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); } err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask);
@@ -1038,7 +1043,7 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm,for_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); } err =xe_vm_ops_add_range_unbind (&vops, range);
Signature · 1 line
2.34.1
Re: [PATCH v6 12/24] drm/xe: Don't use migrate exec queue for page fault binds
Quoted 203 lines · Matthew Brost
Now that the CPU is always used for binds even in jobs, CPU bind jobs can pass GPU jobs in the same exec queue resulting dma-fences signaling out-of-order. Use a dedicated exec queue for binds issued from page faults to avoid ordering issues and avoid blocking kernel binds on unrelated copies / clears.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_migrate.c | 47 ++++++++++++++++++++++++++++++--- drivers/gpu/drm/xe/xe_migrate.h | 1 + drivers/gpu/drm/xe/xe_vm.c | 17 +++++++----- 3 files changed, 55 insertions(+), 10 deletions(-)drivers/gpu/drm/xe/xe_migrate.c +43
-4@@ -51,6 +51,8 @@struct xe_migrate {/** @q: Default exec queue used for migration */ struct xe_exec_queue *q; +/** @bind_q: Default exec queue used for binds */ +struct xe_exec_queue *bind_q;/** @tile: Backpointer to the tile this struct xe_migrate belongs to. */ struct xe_tile *tile;/** @job_mutex: Timeline mutex for @eng. */ @@ -115,6 +117,7 @@ static void xe_migrate_fini(void *arg)mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q); +xe_exec_queue_put (m->bind_q); } static inline u16 xe_migrate_pat_index(struct xe_device *xe,@@ -504,6 +507,15 @@ int xe_migrate_init(struct xe_migrate *m) goto err_out; } + m->bind_q =xe_exec_queue_create (xe, vm, logical_mask,1 , hwe0, +EXEC_QUEUE_FLAG_KERNEL | +EXEC_QUEUE_FLAG_HIGH_PRIORITY | +EXEC_QUEUE_FLAG_MIGRATE ,0 ); +if (IS_ERR (m->bind_q)) { + err =PTR_ERR (m->bind_q); + goto err_out; + } +/* * XXX: Currently only reserving 1 (likely slow) BCS instance on * PVC, may want to revisit if performance is needed. @@ -514,6 +526,15 @@ int xe_migrate_init(struct xe_migrate *m)EXEC_QUEUE_FLAG_MIGRATE | EXEC_QUEUE_FLAG_LOW_LATENCY, 0); }else { + m->bind_q =xe_exec_queue_create_class (xe, primary_gt, vm, +XE_ENGINE_CLASS_COPY , +EXEC_QUEUE_FLAG_KERNEL | +EXEC_QUEUE_FLAG_MIGRATE ,0 ); +if (IS_ERR (m->bind_q)) { + err =PTR_ERR (m->bind_q); + goto err_out; + } + m->q = xe_exec_queue_create_class(xe, primary_gt, vm, XE_ENGINE_CLASS_COPY, EXEC_QUEUE_FLAG_KERNEL |@@ -549,6 +570,8 @@ int xe_migrate_init(struct xe_migrate *m)return err; err_out: +if (!IS_ERR_OR_NULL (m->bind_q)) +xe_exec_queue_put (m->bind_q);xe_vm_close_and_put (vm);return err;@@ -1505,6 +1528,17 @@ static u32 blt_mem_set_cmd_len(struct xe_device *xe)return 7 ; } +/** +* xe_get_migrate_bind_queue() - Get the bind queue from migrate context. +* @migrate: Migrate context. +* +* Return: Pointer to bind queue on success, error on failure +*/ +struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate) +{ +return migrate->bind_q; +} +static void emit_clear_link_copy (struct xe_gt *gt,struct xe_bb *bb,u64 src_ofs,u32 size,u32 pitch) {@@ -1892,6 +1926,11 @@ xe_migrate_update_pgtables_cpu(struct xe_migrate *m,return dma_fence_get_stub (); } +static bool is_migrate_queue (struct xe_migrate *m,struct xe_exec_queue *q) +{ +return m->bind_q == q; +} +static struct dma_fence * __xe_migrate_update_pgtables(struct xe_migrate *m, struct xe_migrate_pt_update *pt_update,@@ -1909,7 +1948,7 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,u32 num_updates =0 , current_update =0 ;u64 addr;int err =0 ; -bool is_migrate = pt_update_ops->q == m->q; +bool is_migrate =is_migrate_queue (m, pt_update_ops->q);bool usm = is_migrate && xe->info.has_usm;for (i =0 ; i < pt_update_ops->num_ops; ++i) {@@ -2631,7 +2670,7 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,*/ void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q) { -bool is_migrate = q == m->q; +bool is_migrate =is_migrate_queue (m, q);if (is_migrate)mutex_lock (&m->job_mutex);@@ -2649,7 +2688,7 @@ void xe_migrate_job_lock(struct xe_migrate *m, struct xe_exec_queue *q)*/ void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q) { -bool is_migrate = q == m->q; +bool is_migrate =is_migrate_queue (m, q);if (is_migrate)mutex_unlock (&m->job_mutex);@@ -2666,7 +2705,7 @@ void xe_migrate_job_lock_assert(struct xe_exec_queue *q) {struct xe_migrate *m =gt_to_tile (q->gt)->migrate; -xe_gt_assert (q->gt, q == m->q); +xe_gt_assert (q->gt, q == m->bind_q);lockdep_assert_held (&m->job_mutex); }#endif drivers/gpu/drm/xe/xe_migrate.h +1
-0@@ -146,6 +146,7 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,struct xe_lrc *xe_migrate_lrc (struct xe_migrate *migrate);struct xe_exec_queue *xe_migrate_exec_queue (struct xe_migrate *migrate); +struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate);struct dma_fence *xe_migrate_vram_copy_chunk (struct xe_bo *vram_bo,u64 vram_offset,struct xe_bo *sysmem_bo,u64 sysmem_offset,u64 size,enum xe_migrate_copy_dir dir);drivers/gpu/drm/xe/xe_vm.c +10
-5@@ -795,7 +795,9 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)struct xe_vma *vma, *next;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -int err, i; +struct xe_tile *tile; +u8 id; +int err;lockdep_assert_held (&vm->lock); if ((xe_vm_in_lr_mode(vm) && !rebind_worker) ||@@ -803,8 +805,11 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)return 0 ;xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for (i =0 ; i <XE_MAX_TILES_PER_DEVICE ; ++i) - vops.pt_update_ops[i].wait_vm_bookkeep = true; +for_each_tile (tile, vm->xe, id) { + vops.pt_update_ops[id].wait_vm_bookkeep = true; + vops.pt_update_ops[id].q = +xe_migrate_bind_queue (tile->migrate); + }xe_vm_assert_held (vm);list_for_each_entry (vma, &vm->rebind_list, combined_links.rebind) {@@ -862,7 +867,7 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_mafor_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); } err =xe_vm_ops_add_rebind (&vops, vma, tile_mask);@@ -954,7 +959,7 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm,for_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); } err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask);
drivers/gpu/drm/xe/xe_vm.c struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, · line 1047
Quoted 6 lines
drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -1038,7 +1043,7 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm,for_each_tile (tile, vm->xe, id) { vops.pt_update_ops[id].wait_vm_bookkeep = true; vops.pt_update_ops[tile->id].q = -xe_migrate_exec_queue (tile->migrate); +xe_migrate_bind_queue (tile->migrate); }
LGTM
Quoted 2 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0err =xe_vm_ops_add_range_unbind (&vops, range);
Matthew Brost
maintainerPATCH v6 13/24 drm/xe: Enable CPU binds for jobs
In reply to Matthew BrostNo reason to use the GPU for binds.
Benefits of CPU-based binds:
- Lower latency once dependencies are resolved, as there is no interaction with the GuC or a hardware context switch both of which are relatively slow.
- Large arrays of binds do not risk running out of migration PTEs, avoiding -ENOBUFS being returned to userspace.
- Kernel binds are decoupled from the migration exec queue (which issues copies and clears), so they cannot get stuck behind unrelated jobs—this can be a problem with parallel GPU faults.
- Paves the for path decouping binds from tiles and individual engines
- Enables ULLS on the migration exec queue, as this queue has exclusive access to the paging copy engine.
Update migration layer to formulate a PT job which will issue CPU bind in the submission backend.
All code related to GPU-based binding has been removed.
--- drivers/gpu/drm/xe/xe_bo_types.h | 2 - drivers/gpu/drm/xe/xe_migrate.c | 249 +++---------------------------- drivers/gpu/drm/xe/xe_pt.c | 1 - 3 files changed, 17 insertions(+), 235 deletions(-)
drivers/gpu/drm/xe/xe_bo_types.h +0 -2
@@ -90,8 +90,6 @@ struct xe_bo {/** @freed: List node for delayed put. */ struct llist_node freed; -/** @update_index: Update index if PT BO */ -int update_index;/** @created: Whether the bo has passed initial creation */ bool created;
drivers/gpu/drm/xe/xe_migrate.c +17 -232
340 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.c +0 -1
@@ -387,7 +387,6 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, entry->pt = parent; entry->flags =0 ; entry->qwords =0 ; - entry->pt_bo->update_index =-1 ; entry->level = parent->level;if (alloc_entries) {
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 13/24] drm/xe: Enable CPU binds for jobs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Memory leak of `xe_pt_job_ops` and `job->fence` when PT jobs are aborted by the DRM scheduler. --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Memory leak of xe_pt_job_ops and job->fence when PT jobs are aborted by the DRM scheduler. --
commit 5533668320d4e1611a5ff8ddda82df8a65a1b27b
Author: Matthew Brost <[email protected]>
drm/xe: Enable CPU binds for jobs
This patch removes GPU-based binding and updates the migration layer to formulate a PT job which will issue CPU bind in the submission backend. This lowers latency and avoids running out of migration PTEs.
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_migrate.c __xe_migrate_update_pgtables(struct xe_migrate *m, · line 1900
Quoted 13 lines
drivers/gpu/drm/xe/xe_migrate.c +6
-0@@ -2104,6 +1889,12 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,if (is_migrate)mutex_lock (&m->job_mutex); + job->pt_update[0 ].vm = pt_update->vops->vm; + job->pt_update[0 ].tile = tile; + job->pt_update[0 ].ops = ops; + job->pt_update[0 ].pt_job_ops = +xe_pt_job_ops_get (pt_update_ops->pt_job_ops); +xe_sched_job_arm (job); fence =dma_fence_get (&job->drm.s_fence->finished);xe_sched_job_push (job);
[Severity: Medium]
Are these references correctly released if the DRM scheduler aborts the job?
When a PT job is created and armed, pt_job_ops gets a reference here, and the job's fence receives an extra reference. Normally, guc_exec_queue_run_job() drops both of these references:
drivers/gpu/drm/xe/xe_guc_submit.c:guc_exec_queue_run_job() {
if (is_pt_job(job)) {
xe_pt_job_ops_put(job->pt_update[0].pt_job_ops);
dma_fence_put(job->fence); /* Drop ref from xe_sched_job_arm */
return NULL;
}...
}
However, if the job is aborted (e.g., queue closed, wedging), run_job() is bypassed and the scheduler calls xe_sched_job_destroy() instead. Looking at xe_sched_job_destroy(), it only puts job->fence once and lacks a call to xe_pt_job_ops_put() entirely:
drivers/gpu/drm/xe/xe_sched_job.c:xe_sched_job_destroy() {
...
xe_sched_job_free_fences(job); dma_fence_put(job->fence); drm_sched_job_cleanup(&job->drm); job_free(job);
...
}
Does this permanently leak the pt_job_ops structure and the second fence reference when PT jobs are aborted during error paths?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Re: [PATCH v6 13/24] drm/xe: Enable CPU binds for jobs
In reply to Matthew BrostQuoted 402 lines · Matthew Brost
No reason to use the GPU for binds.
Benefits of CPU-based binds:
- Lower latency once dependencies are resolved, as there is no interaction with the GuC or a hardware context switch both of which are relatively slow.
- Large arrays of binds do not risk running out of migration PTEs, avoiding-ENOBUFSbeing returned to userspace.
- Kernel binds are decoupled from the migration exec queue (which issues copies and clears), so they cannot get stuck behind unrelated jobs—this can be a problem with parallel GPU faults.
- Paves the for path decouping binds from tiles and individual engines
- Enables ULLS on the migration exec queue, as this queue has exclusive access to the paging copy engine.Update migration layer to formulate a PT job which will issue CPU bind in the submission backend.
All code related to GPU-based binding has been removed.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_bo_types.h | 2 - drivers/gpu/drm/xe/xe_migrate.c | 249 +++---------------------------- drivers/gpu/drm/xe/xe_pt.c | 1 - 3 files changed, 17 insertions(+), 235 deletions(-)drivers/gpu/drm/xe/xe_bo_types.h +0
-2@@ -90,8 +90,6 @@ struct xe_bo {/** @freed: List node for delayed put. */ struct llist_node freed; -/** @update_index: Update index if PT BO */ -int update_index;/** @created: Whether the bo has passed initial creation */ bool created;drivers/gpu/drm/xe/xe_migrate.c +17
-232@@ -77,18 +77,12 @@ struct xe_migrate {* Protected by @job_mutex. */ struct dma_fence *fence; -/** -* @vm_update_sa: For integrated, used to suballocate page-tables -* out of the pt_bo. -*/ -struct drm_suballoc_manager vm_update_sa;/** @min_chunk_size: For dgfx, Minimum chunk size */ u64 min_chunk_size; };#define MAX_PREEMPTDISABLE_TRANSFER SZ_8M/* Around 1ms. */ #define MAX_CCS_LIMITED_TRANSFER SZ_4M/* XE_PAGE_SIZE * (FIELD_MAX(XE2_CCS_SIZE_MASK) + 1) */ -#define NUM_KERNEL_PDE 15#define NUM_PT_SLOTS 48#define LEVEL0_PAGE_TABLE_ENCODE_SIZE SZ_2M#define MAX_NUM_PTE 512@@ -113,7 +107,6 @@ static void xe_migrate_fini(void *arg)dma_fence_put (m->fence);xe_bo_put (m->pt_bo); -drm_suballoc_manager_fini (&m->vm_update_sa);mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q);@@ -234,8 +227,6 @@ static int xe_migrate_pt_bo_alloc(struct xe_tile *tile, struct xe_migrate *m,BUILD_BUG_ON (NUM_PT_SLOTS >SZ_2M /XE_PAGE_SIZE );/* Must be a multiple of 64K to support all platforms */ BUILD_BUG_ON (NUM_PT_SLOTS *XE_PAGE_SIZE %SZ_64K ); -/* And one slot reserved for the 4KiB page table updates */ -BUILD_BUG_ON (!(NUM_KERNEL_PDE &1 ));/* Need to be sure everything fits in the first PT, or create more */ xe_tile_assert (tile, m->batch_base_ofs +xe_bo_size (batch) <SZ_2M );@@ -391,17 +382,9 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, } } -if (ofs) - *ofs = map_ofs; -} - -static void xe_migrate_suballoc_manager_init (struct xe_migrate *m,u32 map_ofs) -{/* * Example layout created above, with root level = 3: * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's -* [PT8]: Kernel PT for VM_BIND, 4 KiB PTE's -* [PT9...PT40]: Userspace PT's for VM_BIND, 4 KiB PTE's * [PT41 = PDE 0] [PT44...PT47 = 4K and 2M vram identity maps] * * This makes the lowest part of the VM point to the pagetables. @@ -409,19 +392,13 @@ static void xe_migrate_suballoc_manager_init(struct xe_migrate *m, u32 map_ofs)* and flushes, other parts of the VM can be used either for copying and * clearing. * -* For performance, the kernel reserves PDE's, so about 20 are left -* for async VM updates. -* * To make it easier to work, each scratch PT is put in slot (1 + PT #) * everywhere, this allows lockless updates to scratch pages by using * the different addresses in VM. */ -#define NUM_VMUSA_UNIT_PER_PAGE 32 -#define VM_SA_UPDATE_UNIT_SIZE (XE_PAGE_SIZE / NUM_VMUSA_UNIT_PER_PAGE) -#define NUM_VMUSA_WRITES_PER_UNIT (VM_SA_UPDATE_UNIT_SIZE / sizeof(u64)) -drm_suballoc_manager_init (&m->vm_update_sa, - (size_t )(map_ofs /XE_PAGE_SIZE -NUM_KERNEL_PDE ) * -NUM_VMUSA_UNIT_PER_PAGE ,0 ); + +if (ofs) + *ofs = map_ofs; }static bool xe_migrate_needs_ccs_emit (struct xe_device *xe)@@ -466,7 +443,6 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mreturn err;xe_migrate_prepare_vm (tile, m, vm, &map_ofs); -xe_migrate_suballoc_manager_init (m, map_ofs);drm_exec_retry_on_contention (&exec);xe_validation_retry_on_oom (&ctx, &err); }@@ -1169,6 +1145,9 @@ struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate)return migrate->q->lrc[0 ]; } +/* XXX: With CPU binds this can be removed in a follow up */ +#define NUM_KERNEL_PDE 15 +static u64 migrate_vm_ppgtt_addr_tlb_inval (void ) {/* @@ -1788,56 +1767,6 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,return fence; } -static void write_pgtable (struct xe_tile *tile,struct xe_bb *bb,u64 ppgtt_ofs, -const struct xe_vm_pgtable_update_op *pt_op, -const struct xe_vm_pgtable_update *update, -struct xe_migrate_pt_update *pt_update) -{ -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_vm *vm = pt_update->vops->vm; -u32 chunk; -u32 ofs = update->ofs, size = update->qwords; - -/* -* If we have 512 entries (max), we would populate it ourselves, -* and update the PDE above it to the new pointer. -* The only time this can only happen if we have to update the top -* PDE. This requires a BO that is almost vm->size big. -* -* This shouldn't be possible in practice.. might change when 16K -* pages are used. Hence the assert. -*/ -xe_tile_assert (tile, update->qwords <MAX_NUM_PTE ); -if (!ppgtt_ofs) - ppgtt_ofs =xe_migrate_vram_ofs (tile_to_xe (tile), -xe_bo_addr (update->pt_bo,0 , -XE_PAGE_SIZE ), false); - -do { -u64 addr = ppgtt_ofs + ofs *8 ; - - chunk =min (size,MAX_PTE_PER_SDI ); - -/* Ensure populatefn can do memset64 by aligning bb->cs */ -if (!(bb->len &1 )) - bb->cs[bb->len++] =MI_NOOP ; - - bb->cs[bb->len++] =MI_STORE_DATA_IMM |MI_SDI_NUM_QW (chunk); - bb->cs[bb->len++] =lower_32_bits (addr); - bb->cs[bb->len++] =upper_32_bits (addr); -if (pt_op->bind) - ops->populate (tile,NULL , bb->cs + bb->len, - ofs, chunk, update); -else - ops->clear (vm, tile,NULL , bb->cs + bb->len, - ofs, chunk, update); - - bb->len += chunk *2 ; - ofs += chunk; - size -= chunk; - }while (size); -} -struct xe_vm *xe_migrate_get_vm (struct xe_migrate *m) {return xe_vm_get (m->q->vm);@@ -1938,162 +1867,18 @@ __xe_migrate_update_pgtables(struct xe_migrate *m, {const struct xe_migrate_pt_update_ops *ops = pt_update->ops;struct xe_tile *tile = m->tile; -struct xe_gt *gt = tile->primary_gt; -struct xe_device *xe =tile_to_xe (tile);struct xe_sched_job *job;struct dma_fence *fence; -struct drm_suballoc *sa_bo =NULL ; -struct xe_bb *bb; -u32 i, j, batch_size =0 , ppgtt_ofs, update_idx, page_ofs =0 ; -u32 num_updates =0 , current_update =0 ; -u64 addr; -int err =0 ;bool is_migrate =is_migrate_queue (m, pt_update_ops->q); -bool usm = is_migrate && xe->info.has_usm; - -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - - num_updates += pt_op->num_entries; -for (j =0 ; j < pt_op->num_entries; ++j) { -u32 num_cmds =DIV_ROUND_UP (updates[j].qwords, -MAX_PTE_PER_SDI ); - -/* align noop + MI_STORE_DATA_IMM cmd prefix */ - batch_size +=4 * num_cmds + updates[j].qwords *2 ; - } - } - -/* fixed + PTE entries */ -if (IS_DGFX (xe)) - batch_size +=2 ; -else - batch_size +=6 * (num_updates /MAX_PTE_PER_SDI +1 ) + - num_updates *2 ; - - bb =xe_bb_new (gt, batch_size, usm); -if (IS_ERR (bb)) -return ERR_CAST (bb); - -/* For sysmem PTE's, need to map them in our hole.. */ -if (!IS_DGFX (xe)) { -u16 pat_index =xe_cache_pat_idx (xe,XE_CACHE_WB ); -u32 ptes, ofs; - - ppgtt_ofs =NUM_KERNEL_PDE -1 ; -if (!is_migrate) { -u32 num_units =DIV_ROUND_UP (num_updates, -NUM_VMUSA_WRITES_PER_UNIT ); - -if (num_units > m->vm_update_sa.size) { - err = -ENOBUFS ; - goto err_bb; - } - sa_bo =drm_suballoc_new (&m->vm_update_sa, num_units, -GFP_KERNEL , true,0 ); -if (IS_ERR (sa_bo)) { - err =PTR_ERR (sa_bo); - goto err_bb; - } - - ppgtt_ofs =NUM_KERNEL_PDE + - (drm_suballoc_soffset (sa_bo) / -NUM_VMUSA_UNIT_PER_PAGE ); - page_ofs = (drm_suballoc_soffset (sa_bo) % -NUM_VMUSA_UNIT_PER_PAGE ) * -VM_SA_UPDATE_UNIT_SIZE ; - } - -/* Map our PT's to gtt */ - i =0 ; - j =0 ; - ptes = num_updates; - ofs = ppgtt_ofs *XE_PAGE_SIZE + page_ofs; -while (ptes) { -u32 chunk =min (MAX_PTE_PER_SDI , ptes); -u32 idx =0 ; - - bb->cs[bb->len++] =MI_STORE_DATA_IMM | -MI_SDI_NUM_QW (chunk); - bb->cs[bb->len++] = ofs; - bb->cs[bb->len++] =0 ;/* upper_32_bits */ - -for (; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (; j < pt_op->num_entries; ++j, ++current_update, ++idx) { -struct xe_vm *vm = pt_update->vops->vm; -struct xe_bo *pt_bo = updates[j].pt_bo; - -if (idx == chunk) - goto next_cmd; - -xe_tile_assert (tile,xe_bo_size (pt_bo) ==SZ_4K ); - -/* Map a PT at most once */ -if (pt_bo->update_index <0 ) - pt_bo->update_index = current_update; - - addr = vm->pt_ops->pte_encode_bo (pt_bo,0 , - pat_index,0 ); - bb->cs[bb->len++] =lower_32_bits (addr); - bb->cs[bb->len++] =upper_32_bits (addr); - } - - j =0 ; - } - -next_cmd: - ptes -= chunk; - ofs += chunk *sizeof (u64); - } - - bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; - update_idx = bb->len; - - addr =xe_migrate_vm_addr (ppgtt_ofs,0 ) + - (page_ofs /sizeof (u64)) *XE_PAGE_SIZE ; -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (j =0 ; j < pt_op->num_entries; ++j) { -struct xe_bo *pt_bo = updates[j].pt_bo; - -write_pgtable (tile, bb, addr + - pt_bo->update_index *XE_PAGE_SIZE , - pt_op, &updates[j], pt_update); - } - } - }else { -/* phys pages, no preamble required */ - bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; - update_idx = bb->len; - -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (j =0 ; j < pt_op->num_entries; ++j) -write_pgtable (tile, bb,0 , pt_op, &updates[j], - pt_update); - } - } +int err; - job =xe_bb_create_migration_job (pt_update_ops->q, bb, -xe_migrate_batch_base (m, usm), - update_idx); + job =xe_sched_job_create (pt_update_ops->q,NULL );if (IS_ERR (job)) { err =PTR_ERR (job); - goto err_sa; + goto err_out; } -xe_sched_job_add_migrate_flush (job,MI_INVALIDATE_TLB ); +xe_tile_assert (tile, job->is_pt_job);if (ops->pre_commit) { pt_update->job = job;@@ -2104,6 +1889,12 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,if (is_migrate)mutex_lock (&m->job_mutex); + job->pt_update[0 ].vm = pt_update->vops->vm; + job->pt_update[0 ].tile = tile; + job->pt_update[0 ].ops = ops; + job->pt_update[0 ].pt_job_ops = +xe_pt_job_ops_get (pt_update_ops->pt_job_ops); +xe_sched_job_arm (job); fence =dma_fence_get (&job->drm.s_fence->finished);xe_sched_job_push (job);@@ -2111,17 +1902,11 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,if (is_migrate)mutex_unlock (&m->job_mutex); -xe_bb_free (bb, fence); -drm_suballoc_free (sa_bo, fence); -return fence; err_job:xe_sched_job_put (job); -err_sa: -drm_suballoc_free (sa_bo,NULL ); -err_bb: -xe_bb_free (bb,NULL ); +err_out:return ERR_PTR (err); }drivers/gpu/drm/xe/xe_pt.c +0
-0
drivers/gpu/drm/xe/xe_pt.c xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, · line 390
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1@@ -387,7 +387,6 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, entry->pt = parent; entry->flags =0 ; entry->qwords =0 ; - entry->pt_bo->update_index =-1 ;
Patch LGTM
Sashiko's comment regarding "Memory leak of xe_pt_job_ops and `job->fence" needs to be handled in
https://patchwork.freedesktop.org/patch/751036/?series=149888&rev=8
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0entry->level = parent->level;if (alloc_entries) {
Matthew Brost
maintainerRe: [PATCH v6 13/24] drm/xe: Enable CPU binds for jobs
Quoted 394 lines · Himal Prasad
Quoted 385 lines · Matthew Brost
No reason to use the GPU for binds.
Benefits of CPU-based binds:
- Lower latency once dependencies are resolved, as there is no interaction with the GuC or a hardware context switch both of which are relatively slow.
- Large arrays of binds do not risk running out of migration PTEs, avoiding-ENOBUFSbeing returned to userspace.
- Kernel binds are decoupled from the migration exec queue (which issues copies and clears), so they cannot get stuck behind unrelated jobs—this can be a problem with parallel GPU faults.
- Paves the for path decouping binds from tiles and individual engines
- Enables ULLS on the migration exec queue, as this queue has exclusive access to the paging copy engine.Update migration layer to formulate a PT job which will issue CPU bind in the submission backend.
All code related to GPU-based binding has been removed.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_bo_types.h | 2 - drivers/gpu/drm/xe/xe_migrate.c | 249 +++---------------------------- drivers/gpu/drm/xe/xe_pt.c | 1 - 3 files changed, 17 insertions(+), 235 deletions(-)drivers/gpu/drm/xe/xe_bo_types.h +0
-2@@ -90,8 +90,6 @@ struct xe_bo {/** @freed: List node for delayed put. */ struct llist_node freed; -/** @update_index: Update index if PT BO */ -int update_index;/** @created: Whether the bo has passed initial creation */ bool created;drivers/gpu/drm/xe/xe_migrate.c +17
-232@@ -77,18 +77,12 @@ struct xe_migrate {* Protected by @job_mutex. */ struct dma_fence *fence; -/** -* @vm_update_sa: For integrated, used to suballocate page-tables -* out of the pt_bo. -*/ -struct drm_suballoc_manager vm_update_sa;/** @min_chunk_size: For dgfx, Minimum chunk size */ u64 min_chunk_size; };#define MAX_PREEMPTDISABLE_TRANSFER SZ_8M/* Around 1ms. */ #define MAX_CCS_LIMITED_TRANSFER SZ_4M/* XE_PAGE_SIZE * (FIELD_MAX(XE2_CCS_SIZE_MASK) + 1) */ -#define NUM_KERNEL_PDE 15#define NUM_PT_SLOTS 48#define LEVEL0_PAGE_TABLE_ENCODE_SIZE SZ_2M#define MAX_NUM_PTE 512@@ -113,7 +107,6 @@ static void xe_migrate_fini(void *arg)dma_fence_put (m->fence);xe_bo_put (m->pt_bo); -drm_suballoc_manager_fini (&m->vm_update_sa);mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q);@@ -234,8 +227,6 @@ static int xe_migrate_pt_bo_alloc(struct xe_tile *tile, struct xe_migrate *m,BUILD_BUG_ON (NUM_PT_SLOTS >SZ_2M /XE_PAGE_SIZE );/* Must be a multiple of 64K to support all platforms */ BUILD_BUG_ON (NUM_PT_SLOTS *XE_PAGE_SIZE %SZ_64K ); -/* And one slot reserved for the 4KiB page table updates */ -BUILD_BUG_ON (!(NUM_KERNEL_PDE &1 ));/* Need to be sure everything fits in the first PT, or create more */ xe_tile_assert (tile, m->batch_base_ofs +xe_bo_size (batch) <SZ_2M );@@ -391,17 +382,9 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, } } -if (ofs) - *ofs = map_ofs; -} - -static void xe_migrate_suballoc_manager_init (struct xe_migrate *m,u32 map_ofs) -{/* * Example layout created above, with root level = 3: * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's -* [PT8]: Kernel PT for VM_BIND, 4 KiB PTE's -* [PT9...PT40]: Userspace PT's for VM_BIND, 4 KiB PTE's * [PT41 = PDE 0] [PT44...PT47 = 4K and 2M vram identity maps] * * This makes the lowest part of the VM point to the pagetables. @@ -409,19 +392,13 @@ static void xe_migrate_suballoc_manager_init(struct xe_migrate *m, u32 map_ofs)* and flushes, other parts of the VM can be used either for copying and * clearing. * -* For performance, the kernel reserves PDE's, so about 20 are left -* for async VM updates. -* * To make it easier to work, each scratch PT is put in slot (1 + PT #) * everywhere, this allows lockless updates to scratch pages by using * the different addresses in VM. */ -#define NUM_VMUSA_UNIT_PER_PAGE 32 -#define VM_SA_UPDATE_UNIT_SIZE (XE_PAGE_SIZE / NUM_VMUSA_UNIT_PER_PAGE) -#define NUM_VMUSA_WRITES_PER_UNIT (VM_SA_UPDATE_UNIT_SIZE / sizeof(u64)) -drm_suballoc_manager_init (&m->vm_update_sa, - (size_t )(map_ofs /XE_PAGE_SIZE -NUM_KERNEL_PDE ) * -NUM_VMUSA_UNIT_PER_PAGE ,0 ); + +if (ofs) + *ofs = map_ofs; }static bool xe_migrate_needs_ccs_emit (struct xe_device *xe)@@ -466,7 +443,6 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mreturn err;xe_migrate_prepare_vm (tile, m, vm, &map_ofs); -xe_migrate_suballoc_manager_init (m, map_ofs);drm_exec_retry_on_contention (&exec);xe_validation_retry_on_oom (&ctx, &err); }@@ -1169,6 +1145,9 @@ struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate)return migrate->q->lrc[0 ]; } +/* XXX: With CPU binds this can be removed in a follow up */ +#define NUM_KERNEL_PDE 15 +static u64 migrate_vm_ppgtt_addr_tlb_inval (void ) {/* @@ -1788,56 +1767,6 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,return fence; } -static void write_pgtable (struct xe_tile *tile,struct xe_bb *bb,u64 ppgtt_ofs, -const struct xe_vm_pgtable_update_op *pt_op, -const struct xe_vm_pgtable_update *update, -struct xe_migrate_pt_update *pt_update) -{ -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_vm *vm = pt_update->vops->vm; -u32 chunk; -u32 ofs = update->ofs, size = update->qwords; - -/* -* If we have 512 entries (max), we would populate it ourselves, -* and update the PDE above it to the new pointer. -* The only time this can only happen if we have to update the top -* PDE. This requires a BO that is almost vm->size big. -* -* This shouldn't be possible in practice.. might change when 16K -* pages are used. Hence the assert. -*/ -xe_tile_assert (tile, update->qwords <MAX_NUM_PTE ); -if (!ppgtt_ofs) - ppgtt_ofs =xe_migrate_vram_ofs (tile_to_xe (tile), -xe_bo_addr (update->pt_bo,0 , -XE_PAGE_SIZE ), false); - -do { -u64 addr = ppgtt_ofs + ofs *8 ; - - chunk =min (size,MAX_PTE_PER_SDI ); - -/* Ensure populatefn can do memset64 by aligning bb->cs */ -if (!(bb->len &1 )) - bb->cs[bb->len++] =MI_NOOP ; - - bb->cs[bb->len++] =MI_STORE_DATA_IMM |MI_SDI_NUM_QW (chunk); - bb->cs[bb->len++] =lower_32_bits (addr); - bb->cs[bb->len++] =upper_32_bits (addr); -if (pt_op->bind) - ops->populate (tile,NULL , bb->cs + bb->len, - ofs, chunk, update); -else - ops->clear (vm, tile,NULL , bb->cs + bb->len, - ofs, chunk, update); - - bb->len += chunk *2 ; - ofs += chunk; - size -= chunk; - }while (size); -} -struct xe_vm *xe_migrate_get_vm (struct xe_migrate *m) {return xe_vm_get (m->q->vm);@@ -1938,162 +1867,18 @@ __xe_migrate_update_pgtables(struct xe_migrate *m, {const struct xe_migrate_pt_update_ops *ops = pt_update->ops;struct xe_tile *tile = m->tile; -struct xe_gt *gt = tile->primary_gt; -struct xe_device *xe =tile_to_xe (tile);struct xe_sched_job *job;struct dma_fence *fence; -struct drm_suballoc *sa_bo =NULL ; -struct xe_bb *bb; -u32 i, j, batch_size =0 , ppgtt_ofs, update_idx, page_ofs =0 ; -u32 num_updates =0 , current_update =0 ; -u64 addr; -int err =0 ;bool is_migrate =is_migrate_queue (m, pt_update_ops->q); -bool usm = is_migrate && xe->info.has_usm; - -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - - num_updates += pt_op->num_entries; -for (j =0 ; j < pt_op->num_entries; ++j) { -u32 num_cmds =DIV_ROUND_UP (updates[j].qwords, -MAX_PTE_PER_SDI ); - -/* align noop + MI_STORE_DATA_IMM cmd prefix */ - batch_size +=4 * num_cmds + updates[j].qwords *2 ; - } - } - -/* fixed + PTE entries */ -if (IS_DGFX (xe)) - batch_size +=2 ; -else - batch_size +=6 * (num_updates /MAX_PTE_PER_SDI +1 ) + - num_updates *2 ; - - bb =xe_bb_new (gt, batch_size, usm); -if (IS_ERR (bb)) -return ERR_CAST (bb); - -/* For sysmem PTE's, need to map them in our hole.. */ -if (!IS_DGFX (xe)) { -u16 pat_index =xe_cache_pat_idx (xe,XE_CACHE_WB ); -u32 ptes, ofs; - - ppgtt_ofs =NUM_KERNEL_PDE -1 ; -if (!is_migrate) { -u32 num_units =DIV_ROUND_UP (num_updates, -NUM_VMUSA_WRITES_PER_UNIT ); - -if (num_units > m->vm_update_sa.size) { - err = -ENOBUFS ; - goto err_bb; - } - sa_bo =drm_suballoc_new (&m->vm_update_sa, num_units, -GFP_KERNEL , true,0 ); -if (IS_ERR (sa_bo)) { - err =PTR_ERR (sa_bo); - goto err_bb; - } - - ppgtt_ofs =NUM_KERNEL_PDE + - (drm_suballoc_soffset (sa_bo) / -NUM_VMUSA_UNIT_PER_PAGE ); - page_ofs = (drm_suballoc_soffset (sa_bo) % -NUM_VMUSA_UNIT_PER_PAGE ) * -VM_SA_UPDATE_UNIT_SIZE ; - } - -/* Map our PT's to gtt */ - i =0 ; - j =0 ; - ptes = num_updates; - ofs = ppgtt_ofs *XE_PAGE_SIZE + page_ofs; -while (ptes) { -u32 chunk =min (MAX_PTE_PER_SDI , ptes); -u32 idx =0 ; - - bb->cs[bb->len++] =MI_STORE_DATA_IMM | -MI_SDI_NUM_QW (chunk); - bb->cs[bb->len++] = ofs; - bb->cs[bb->len++] =0 ;/* upper_32_bits */ - -for (; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (; j < pt_op->num_entries; ++j, ++current_update, ++idx) { -struct xe_vm *vm = pt_update->vops->vm; -struct xe_bo *pt_bo = updates[j].pt_bo; - -if (idx == chunk) - goto next_cmd; - -xe_tile_assert (tile,xe_bo_size (pt_bo) ==SZ_4K ); - -/* Map a PT at most once */ -if (pt_bo->update_index <0 ) - pt_bo->update_index = current_update; - - addr = vm->pt_ops->pte_encode_bo (pt_bo,0 , - pat_index,0 ); - bb->cs[bb->len++] =lower_32_bits (addr); - bb->cs[bb->len++] =upper_32_bits (addr); - } - - j =0 ; - } - -next_cmd: - ptes -= chunk; - ofs += chunk *sizeof (u64); - } - - bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; - update_idx = bb->len; - - addr =xe_migrate_vm_addr (ppgtt_ofs,0 ) + - (page_ofs /sizeof (u64)) *XE_PAGE_SIZE ; -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (j =0 ; j < pt_op->num_entries; ++j) { -struct xe_bo *pt_bo = updates[j].pt_bo; - -write_pgtable (tile, bb, addr + - pt_bo->update_index *XE_PAGE_SIZE , - pt_op, &updates[j], pt_update); - } - } - }else { -/* phys pages, no preamble required */ - bb->cs[bb->len++] =MI_BATCH_BUFFER_END ; - update_idx = bb->len; - -for (i =0 ; i < pt_update_ops->num_ops; ++i) { -struct xe_vm_pgtable_update_op *pt_op = - &pt_update_ops->pt_job_ops->ops[i]; -struct xe_vm_pgtable_update *updates = pt_op->entries; - -for (j =0 ; j < pt_op->num_entries; ++j) -write_pgtable (tile, bb,0 , pt_op, &updates[j], - pt_update); - } - } +int err; - job =xe_bb_create_migration_job (pt_update_ops->q, bb, -xe_migrate_batch_base (m, usm), - update_idx); + job =xe_sched_job_create (pt_update_ops->q,NULL );if (IS_ERR (job)) { err =PTR_ERR (job); - goto err_sa; + goto err_out; } -xe_sched_job_add_migrate_flush (job,MI_INVALIDATE_TLB ); +xe_tile_assert (tile, job->is_pt_job);if (ops->pre_commit) { pt_update->job = job;@@ -2104,6 +1889,12 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,if (is_migrate)mutex_lock (&m->job_mutex); + job->pt_update[0 ].vm = pt_update->vops->vm; + job->pt_update[0 ].tile = tile; + job->pt_update[0 ].ops = ops; + job->pt_update[0 ].pt_job_ops = +xe_pt_job_ops_get (pt_update_ops->pt_job_ops); +xe_sched_job_arm (job); fence =dma_fence_get (&job->drm.s_fence->finished);xe_sched_job_push (job);@@ -2111,17 +1902,11 @@ __xe_migrate_update_pgtables(struct xe_migrate *m,if (is_migrate)mutex_unlock (&m->job_mutex); -xe_bb_free (bb, fence); -drm_suballoc_free (sa_bo, fence); -return fence; err_job:xe_sched_job_put (job); -err_sa: -drm_suballoc_free (sa_bo,NULL ); -err_bb: -xe_bb_free (bb,NULL ); +err_out:return ERR_PTR (err); }drivers/gpu/drm/xe/xe_pt.c +0
-0drivers/gpu/drm/xe/xe_pt.c xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, · line 390
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-1@@ -387,7 +387,6 @@ xe_pt_new_shared(struct xe_walk_update *wupd, struct xe_pt *parent, entry->pt = parent; entry->flags =0 ; entry->qwords =0 ; - entry->pt_bo->update_index =-1 ;Patch LGTM
Quoted 5 lines
Reviewed-by Himal Prasad Ghimiray <[email protected]>Sashiko's comment regarding "Memory leak of
xe_pt_job_opsand `job->fence" needs to be handled in
https://patchwork.freedesktop.org/patch/751036/?series=149888&rev=8
No, once we call _arm(), run_job() is always called. Sashiko doesn't seem to understand this - it probably because the scheduler doesn't make this guarnetee but Xe usage of the scheduler ensure this.
Matt
Quoted 5 lines
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0entry->level = parent->level;if (alloc_entries) {
Matthew Brost
maintainerPATCH v6 14/24 drm/xe: Remove unused arguments from xe_migrate_pt_update_ops
In reply to Matthew BrostBoth populate and clear have unused void* ptr arguments, remove these.
--- drivers/gpu/drm/xe/xe_migrate.c | 4 ++-- drivers/gpu/drm/xe/xe_migrate.h | 7 ++----- drivers/gpu/drm/xe/xe_pt.c | 37 ++++++++++++--------------------- 3 files changed, 17 insertions(+), 31 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +2 -2
@@ -1812,11 +1812,11 @@ xe_migrate_update_pgtables_cpu_execute(struct xe_vm *vm, struct xe_tile *tile,if (pt_op->bind) ops->populate (tile, &update->pt_bo->vmap, -NULL , update->ofs, update->qwords, + update->ofs, update->qwords, update);else ops->clear (vm, tile, &update->pt_bo->vmap, -NULL , update->ofs, update->qwords, + update->ofs, update->qwords, update); } }
drivers/gpu/drm/xe/xe_migrate.h +2 -5
@@ -43,7 +43,6 @@ struct xe_migrate_pt_update_ops {* @populate: Populate a command buffer or page-table with ptes. * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. -* @pos: If @map is NULL, map into the memory to be populated. * @ofs: qword offset into @map, unused if @map is NULL. * @num_qwords: Number of qwords to write. * @update: Information about the PTEs to be inserted.
@@ -53,14 +52,13 @@ struct xe_migrate_pt_update_ops {* page-tables with PTEs. */ void (*populate)(struct xe_tile *tile,struct iosys_map *map, -void *pos,u32 ofs,u32 num_qwords, +u32 ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update);/** * @clear: Clear a command buffer or page-table with ptes. * @vm: VM being updated * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. -* @pos: If @map is NULL, map into the memory to be populated. * @ofs: qword offset into @map, unused if @map is NULL. * @num_qwords: Number of qwords to write. * @update: Information about the PTEs to be inserted.
@@ -70,8 +68,7 @@ struct xe_migrate_pt_update_ops {* page-tables with PTEs. */ void (*clear)(struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,void *pos,u32 ofs, -u32 num_qwords, +struct iosys_map *map,u32 ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update);/**
drivers/gpu/drm/xe/xe_pt.c +13 -24
@@ -1106,30 +1106,18 @@ bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm,static void xe_vm_populate_pgtable (struct xe_tile *tile,struct iosys_map *map, -void *data,u32 qword_ofs,u32 num_qwords, +u32 qword_ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update) {struct xe_pt_entry *ptes = update->pt_entries; -u64 *ptr = data;u32 i; -/* -* @qword_ofs is the absolute entry offset within the page table, while -* @ptes is indexed relative to @update->ofs (its first entry). The GPU -* path (write_pgtable) splits a single update into MAX_PTE_PER_SDI-sized -* chunks, calling this with an advancing @qword_ofs but a fresh @data -* pointer per chunk, so translate back into a @ptes index rather than -* assuming the chunk starts at ptes[0]. -*/ -for (i =0 ; i < num_qwords; i++) { -u32 idx = qword_ofs - update->ofs + i; +xe_assert (tile_to_xe (tile), map); +xe_assert (tile_to_xe (tile), !iosys_map_is_null (map)); -if (map) -xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) * -sizeof (u64), u64, ptes[idx].pte); -else - ptr[i] = ptes[idx].pte; - } +for (i =0 ; i < num_qwords; i++) +xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) * +sizeof (u64), u64, ptes[i].pte); } static void xe_pt_cancel_bind(struct xe_vma *vma,
@@ -2021,22 +2009,23 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile,static void xe_migrate_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,void *ptr, -u32 qword_ofs,u32 num_qwords, +struct iosys_map *map,u32 qword_ofs, +u32 num_qwords,const struct xe_vm_pgtable_update *update) {u64 empty =__xe_pt_empty_pte (tile, vm, update->level);int i; -if (map && map->is_iomem) +xe_assert (vm->xe, map); +xe_assert (vm->xe, !iosys_map_is_null (map)); + +if (map->is_iomem)for (i =0 ; i < num_qwords; ++i)xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) *sizeof (u64), u64, empty); -else if (map) +else memset64 (map->vaddr + qword_ofs *sizeof (u64), empty, num_qwords); -else -memset64 (ptr, empty, num_qwords); } static void xe_pt_abort_unbind(struct xe_vma *vma,
Signature · 1 line
2.34.1
RE: [PATCH v6 14/24] drm/xe: Remove unused arguments from xe_migrate_pt_update_ops
Quoted 41 lines · Matthew Brost
Both populate and clear have unused void* ptr arguments, remove these.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>---
drivers/gpu/drm/xe/xe_migrate.c| 4 ++--drivers/gpu/drm/xe/xe_migrate.h
| 7 ++-----drivers/gpu/drm/xe/xe_pt.c | 37 ++++++++++++--------------------- 3 files changed, 17 insertions(+), 31 deletions(-)(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.cindexe5c46e0fa960..22a442d13477100644drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -1812,11 +1812,11 @@xe_migrate_update_pgtables_cpu_execute(struct
xe_vm*vm, structxe_tile*tile,drivers/gpu/drm/xe/xe_migrate.c +0
-1if (pt_op->bind) ops->populate(tile, &update->pt_bo->vmap, - NULL, update->ofs, update-Quoted 1 line
qwords,
drivers/gpu/drm/xe/xe_migrate.c +1
-1+ update->ofs, update->qwords, update); else ops->clear(vm, tile, &update->pt_bo->vmap, - NULL, update->ofs, update-Quoted 1 line
qwords,
drivers/gpu/drm/xe/xe_migrate.c +1
-0+ update->ofs, update->qwords, update); } }(quoted diff) +0
-0
b/drivers/gpu/drm/xe/xe_migrate.hindex7a824654cb72..74ff6446309c100644
drivers/gpu/drm/xe/xe_migrate.h struct xe_migrate_pt_update_ops { · line 46
Quoted 5 lines
drivers/gpu/drm/xe/xe_migrate.h +0
-1@@ -43,7 +43,6 @@ struct xe_migrate_pt_update_ops {* @populate: Populate a command buffer or page-table with ptes. * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. -* @pos: If @map is NULL, map into the memory to be populated. * @ofs: qword offset into @map, unused if @map is NULL.
map can no longer be NULL. Maybe we could correct it in the patch 16 instead of here. The same apply to clear().
drivers/gpu/drm/xe/xe_migrate.h struct xe_migrate_pt_update_ops { · line 47
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.h +0
-0* @num_qwords: Number of qwords to write.
BTW, since both @ofs and @num_qwords are derived from @update, we could drop both and just keep @update. I am OK with either.
Rest of the patch LGTM.
Quoted 5 lines
drivers/gpu/drm/xe/xe_migrate.h +0
-0* @update: Information about the PTEs to be inserted. @@ -53,14 +52,13 @@ struct xe_migrate_pt_update_ops {* page-tables with PTEs. */ void (*populate)(struct xe_tile *tile, struct iosys_map *map,
drivers/gpu/drm/xe/xe_migrate.h struct xe_migrate_pt_update_ops { · line 62
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.h +1
-2- void *pos, u32 ofs, u32 num_qwords, + u32 ofs, u32 num_qwords, const struct xe_vm_pgtable_update *update);/** * @clear: Clear a command buffer or page-table with ptes. * @vm: VM being updated * @tile: The tile for the current operation. * @map: struct iosys_map into the memory to be populated. -* @pos: If @map is NULL, map into the memory to be populated. * @ofs: qword offset into @map, unused if @map is NULL.
The same as upper.
Quoted 87 lines
drivers/gpu/drm/xe/xe_migrate.h +1
-2* @num_qwords: Number of qwords to write. * @update: Information about the PTEs to be inserted. @@ -70,8 +68,7 @@ struct xe_migrate_pt_update_ops {* page-tables with PTEs. */ void (*clear)(struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,void *pos,u32 ofs, -u32 num_qwords, +struct iosys_map *map,u32 ofs,u32 num_qwords,const struct xe_vm_pgtable_update *update);/** drivers/gpu/drm/xe/xe_pt.c index renamed from drivers/gpu/drm/xe/xe_pt.c +0
-0
24190ba4f533..c17d5f96f00a100644drivers/gpu/drm/xe/xe_pt.c +1
-6@@ -1106,30 +1106,18 @@ bool xe_pt_zap_ptes_range(struct xe_tile *tile,struct xe_vm *vm, static void xe_vm_populate_pgtable(struct xe_tile *tile, struct iosys_map *map, - void *data, u32 qword_ofs, u32 num_qwords, + u32 qword_ofs, u32 num_qwords, const struct xe_vm_pgtable_update *update) { struct xe_pt_entry *ptes = update->pt_entries; -u64 *ptr = data;u32 i; -/* -* @qword_ofs is the absolute entry offset within the page table, while -* @ptes is indexed relative to @update->ofs (its first entry). The GPU -* path (write_pgtable) splits a single update into MAX_PTE_PER_SDI- sized
drivers/gpu/drm/xe/xe_pt.c +0
-1-* chunks, calling this with an advancing @qword_ofs but a fresh @data
drivers/gpu/drm/xe/xe_pt.c +5
-11-* pointer per chunk, so translate back into a @ptes index rather than -* assuming the chunk starts at ptes[0]. -*/ -for (i =0 ; i < num_qwords; i++) { -u32 idx = qword_ofs - update->ofs + i; +xe_assert (tile_to_xe (tile), map); +xe_assert (tile_to_xe (tile), !iosys_map_is_null (map)); -if (map) -xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) * -sizeof (u64), u64, ptes[idx].pte); -else - ptr[i] = ptes[idx].pte; - } +for (i =0 ; i < num_qwords; i++) +xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) * +sizeof (u64), u64, ptes[i].pte); } static void xe_pt_cancel_bind(struct xe_vma *vma, @@ -2021,22 +2009,23@@ static unsigned int xe_pt_stage_unbind(struct
xe_tile*tile,static void
xe_migrate_clear_pgtable_callback(structxe_vm*vm, structxe_tile*tile,drivers/gpu/drm/xe/xe_pt.c +2
-2- struct iosys_map *map, void *ptr, - u32 qword_ofs, u32 num_qwords, + struct iosys_map *map, u32 qword_ofs, + u32 num_qwords, const struct xe_vm_pgtable_update*update) {
drivers/gpu/drm/xe/xe_pt.c +5
-5u64 empty =__xe_pt_empty_pte (tile, vm, update->level);int i; -if (map && map->is_iomem) +xe_assert (vm->xe, map); +xe_assert (vm->xe, !iosys_map_is_null (map)); + +if (map->is_iomem)for (i =0 ; i < num_qwords; ++i)xe_map_wr (tile_to_xe (tile), map, (qword_ofs + i) *sizeof (u64), u64, empty); -else if (map) +else memset64 (map->vaddr + qword_ofs *sizeof (u64), empty, num_qwords); -else -memset64 (ptr, empty, num_qwords); } static void xe_pt_abort_unbind(struct xe_vma *vma, --2.34.1
Matthew Brost
maintainerPATCH v6 15/24 drm/xe: Make bind queues operate cross-tile
In reply to Matthew BrostSince bind jobs execute on the CPU rather than the GPU, maintaining a per-tile bind queue no longer provides value. Convert the driver to use a single bind queue shared across tiles. The primary change is routing all GT TLB invalidations through this unified bind queue.
--- drivers/gpu/drm/xe/xe_exec_queue.c | 142 +++++++++-------------- drivers/gpu/drm/xe/xe_exec_queue.h | 14 +-- drivers/gpu/drm/xe/xe_exec_queue_types.h | 20 +--- drivers/gpu/drm/xe/xe_pt.c | 22 ++-- drivers/gpu/drm/xe/xe_sync.c | 20 +--- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 15 ++- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 2 +- drivers/gpu/drm/xe/xe_vm.c | 65 +++++------ drivers/gpu/drm/xe/xe_vm_types.h | 2 +- 9 files changed, 125 insertions(+), 177 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +54 -88
@@ -142,9 +142,8 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q) {int i; -for (i =0 ; i <XE_EXEC_QUEUE_TLB_INVAL_COUNT ; ++i) -if (q->tlb_inval[i].dep_scheduler) -xe_dep_scheduler_fini (q->tlb_inval[i].dep_scheduler); +for_each_tlb_inval (q, i) +xe_dep_scheduler_fini (q->tlb_inval[i].dep_scheduler);if (xe_exec_queue_uses_pxp (q))xe_pxp_exec_queue_remove (gt_to_xe (q->gt)->pxp, q);
@@ -166,31 +165,34 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q)static int alloc_dep_schedulers (struct xe_device *xe,struct xe_exec_queue *q) { -struct xe_tile *tile =gt_to_tile (q->gt); -int i; +struct xe_tile *tile; +int i =0 , j; +u8 id; -for (i =0 ; i <XE_EXEC_QUEUE_TLB_INVAL_COUNT ; ++i) { -struct xe_dep_scheduler *dep_scheduler; -struct xe_gt *gt; -struct workqueue_struct *wq; +for_each_tile (tile, xe, id) { +for (j =0 ; j < (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); ++j, ++i) { +struct xe_dep_scheduler *dep_scheduler; +struct xe_gt *gt; +struct workqueue_struct *wq; -if (i ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ) - gt = tile->primary_gt; -else - gt = tile->media_gt; +if (j ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ) + gt = tile->primary_gt; +else + gt = tile->media_gt; -if (!gt) -continue ; +if (!gt) +continue ; - wq = gt->tlb_inval.job_wq; + wq = gt->tlb_inval.job_wq;#define MAX_TLB_INVAL_JOBS 16/* Picking a reasonable value */ - dep_scheduler =xe_dep_scheduler_create (xe, wq, q->name, -MAX_TLB_INVAL_JOBS ); -if (IS_ERR (dep_scheduler)) -return PTR_ERR (dep_scheduler); + dep_scheduler =xe_dep_scheduler_create (xe, wq, q->name, +MAX_TLB_INVAL_JOBS ); +if (IS_ERR (dep_scheduler)) +return PTR_ERR (dep_scheduler); - q->tlb_inval[i].dep_scheduler = dep_scheduler; + q->tlb_inval[i].dep_scheduler = dep_scheduler; + } }#undef MAX_TLB_INVAL_JOBS
@@ -224,7 +226,6 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe, q->ops = gt->exec_queue_ops;INIT_LIST_HEAD (&q->lr.link);INIT_LIST_HEAD (&q->vm_exec_queue_link); -INIT_LIST_HEAD (&q->multi_gt_link);INIT_LIST_HEAD (&q->hw_engine_group_link);INIT_LIST_HEAD (&q->pxp.link);spin_lock_init (&q->multi_queue.lock);
@@ -636,7 +637,6 @@ ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);void xe_exec_queue_destroy (struct kref *ref) { struct xe_exec_queue *q = container_of(ref, struct xe_exec_queue, refcount); -struct xe_exec_queue *eq, *next;int i;xe_assert (gt_to_xe (q->gt),atomic_read (&q->job_cnt) ==0 );
@@ -648,15 +648,9 @@ void xe_exec_queue_destroy(struct kref *ref)xe_pxp_exec_queue_remove (gt_to_xe (q->gt)->pxp, q);xe_exec_queue_last_fence_put_unlocked (q); -for_each_tlb_inval (i) +for_each_tlb_inval (q, i)xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, i); -if (!(q->flags &EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD )) { -list_for_each_entry_safe (eq, next, &q->multi_gt_list, - multi_gt_link) -xe_exec_queue_put (eq); - } -if (q->user_vm) {xe_vm_put (q->user_vm); q->user_vm =NULL ;
@@ -1343,7 +1337,6 @@ int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data,u64_to_user_ptr (args->instances);struct xe_hw_engine *hwe;struct xe_vm *vm; -struct xe_tile *tile;struct xe_exec_queue *q =NULL ;u32 logical_mask;u32 flags =0 ;
@@ -1392,31 +1385,16 @@ int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data,return -ENOENT ; } -for_each_tile (tile, xe, id) { -struct xe_exec_queue *new; - - flags |=EXEC_QUEUE_FLAG_VM ; -if (id) - flags |=EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD ; - - new =xe_exec_queue_create_bind (xe, tile, vm, flags, - args->extensions); -if (IS_ERR (new)) { -up_read (&vm->lock); -xe_vm_put (vm); - err =PTR_ERR (new); -if (q) - goto put_exec_queue; -return err; - } -if (id ==0 ) - q = new; -else -list_add_tail (&new->multi_gt_list, - &q->multi_gt_link); - } + flags |=EXEC_QUEUE_FLAG_VM ; + + q =xe_exec_queue_create_bind (xe,xe_device_get_root_tile (xe), + vm, flags, args->extensions);up_read (&vm->lock);xe_vm_put (vm); +if (IS_ERR (q)) { + err =PTR_ERR (q); +return err; + } }else { logical_mask = calc_validate_logical_mask(xe, eci, args->width,
@@ -1638,14 +1616,6 @@ void xe_exec_queue_update_run_ticks(struct xe_exec_queue *q)*/ void xe_exec_queue_kill (struct xe_exec_queue *q) { -struct xe_exec_queue *eq = q, *next; - -list_for_each_entry_safe (eq, next, &eq->multi_gt_list, - multi_gt_link) { - q->ops->kill (eq); -xe_vm_remove_compute_exec_queue (q->vm, eq); - } - q->ops->kill (q);xe_vm_remove_compute_exec_queue (q->vm, q); }
@@ -1806,42 +1776,40 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *q, struct xe_vm *vm,* xe_exec_queue_tlb_inval_last_fence_put() - Drop ref to last TLB invalidation fence * @q: The exec queue * @vm: The VM the engine does a bind for -* @type: Either primary or media GT +* @idx: Index of tlb invalidation */ void xe_exec_queue_tlb_inval_last_fence_put (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type) +unsigned int idx) {xe_exec_queue_last_fence_lockdep_assert (q, vm); -xe_assert (vm->xe, type ==XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT || - type ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT ); -xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, type); +xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, idx); }/** * xe_exec_queue_tlb_inval_last_fence_put_unlocked() - Drop ref to last TLB * invalidation fence unlocked * @q: The exec queue -* @type: Either primary or media GT +* @idx: Index of tlb invalidation * * Only safe to be called from xe_exec_queue_destroy(). */ void xe_exec_queue_tlb_inval_last_fence_put_unlocked (struct xe_exec_queue *q, -unsigned int type) +unsigned int idx) { -xe_assert (gt_to_xe (q->gt), type ==XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT || - type ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); +xe_assert (q->vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT ); -dma_fence_put (q->tlb_inval[type].last_fence); - q->tlb_inval[type].last_fence =NULL ; +dma_fence_put (q->tlb_inval[idx].last_fence); + q->tlb_inval[idx].last_fence =NULL ; }/** * xe_exec_queue_tlb_inval_last_fence_get() - Get last fence for TLB invalidation * @q: The exec queue * @vm: The VM the engine does a bind for -* @type: Either primary or media GT +* @idx: Index of tlb invalidation * * Get last fence, takes a ref *
@@ -1849,22 +1817,21 @@ void xe_exec_queue_tlb_inval_last_fence_put_unlocked(struct xe_exec_queue *q,*/ struct dma_fence *xe_exec_queue_tlb_inval_last_fence_get (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type) +unsigned int idx) {struct dma_fence *fence;xe_exec_queue_last_fence_lockdep_assert (q, vm); -xe_assert (vm->xe, type ==XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT || - type ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT );xe_assert (vm->xe, q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); -if (q->tlb_inval[type].last_fence && +if (q->tlb_inval[idx].last_fence &&test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , - &q->tlb_inval[type].last_fence->flags)) -xe_exec_queue_tlb_inval_last_fence_put (q, vm, type); + &q->tlb_inval[idx].last_fence->flags)) +xe_exec_queue_tlb_inval_last_fence_put (q, vm, idx); - fence = q->tlb_inval[type].last_fence ?:dma_fence_get_stub (); + fence = q->tlb_inval[idx].last_fence ?:dma_fence_get_stub ();dma_fence_get (fence);return fence; }
@@ -1874,26 +1841,25 @@ struct dma_fence *xe_exec_queue_tlb_inval_last_fence_get(struct xe_exec_queue *q* @q: The exec queue * @vm: The VM the engine does a bind for * @fence: The fence -* @type: Either primary or media GT +* @idx: Index of tlb invalidation * -* Set the last fence for the tlb invalidation type on the queue. Increases +* Set the last fence for the tlb invalidation client on the queue. Increases * reference count for fence, when closing queue * xe_exec_queue_tlb_inval_last_fence_put should be called. */ void xe_exec_queue_tlb_inval_last_fence_set (struct xe_exec_queue *q,struct xe_vm *vm,struct dma_fence *fence, -unsigned int type) +unsigned int idx) {xe_exec_queue_last_fence_lockdep_assert (q, vm); -xe_assert (vm->xe, type ==XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT || - type ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT );xe_assert (vm->xe, q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE ));xe_assert (vm->xe, !dma_fence_is_container (fence)); -xe_exec_queue_tlb_inval_last_fence_put (q, vm, type); - q->tlb_inval[type].last_fence =dma_fence_get (fence); +xe_exec_queue_tlb_inval_last_fence_put (q, vm, idx); + q->tlb_inval[idx].last_fence =dma_fence_get (fence); }/**
drivers/gpu/drm/xe/xe_exec_queue.h +7 -7
@@ -14,9 +14,9 @@ struct drm_file;struct xe_device ;struct xe_file ; -#define for_each_tlb_inval (__i) \ - for (__i = XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT; \ - __i <= XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT; ++__i) +#define for_each_tlb_inval (__q, __i) \ + for (__i = 0; __i < XE_EXEC_QUEUE_TLB_INVAL_COUNT; ++__i) \ + for_each_if((__q)->tlb_inval[__i].dep_scheduler) struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *vm, u32 logical_mask, u16 width,
@@ -141,19 +141,19 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *e, struct xe_vm *vm,void xe_exec_queue_tlb_inval_last_fence_put (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type); +unsigned int idx);void xe_exec_queue_tlb_inval_last_fence_put_unlocked (struct xe_exec_queue *q, -unsigned int type); +unsigned int idx);struct dma_fence *xe_exec_queue_tlb_inval_last_fence_get (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type); +unsigned int idx);void xe_exec_queue_tlb_inval_last_fence_set (struct xe_exec_queue *q,struct xe_vm *vm,struct dma_fence *fence, -unsigned int type); +unsigned int idx);void xe_exec_queue_update_run_ticks (struct xe_exec_queue *q);
drivers/gpu/drm/xe/xe_exec_queue_types.h +6 -14
@@ -137,16 +137,14 @@ struct xe_exec_queue {#define EXEC_QUEUE_FLAG_KERNEL BIT(0)/* for VM jobs. Caller needs to hold rpm ref when creating queue with this flag */ #define EXEC_QUEUE_FLAG_VM BIT(1) -/* child of VM queue for multi-tile VM jobs */ -#define EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD BIT(2)/* kernel exec_queue only, set priority to highest level */ -#define EXEC_QUEUE_FLAG_HIGH_PRIORITY BIT(3) +#define EXEC_QUEUE_FLAG_HIGH_PRIORITY BIT(2)/* flag to indicate low latency hint to guc */ -#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(4) +#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(3)/* for migration (kernel copy, clear, bind) jobs */ -#define EXEC_QUEUE_FLAG_MIGRATE BIT(5) +#define EXEC_QUEUE_FLAG_MIGRATE BIT(4)/* for programming COMMON_SLICE_CHICKEN3 on first submission */ -#define EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX BIT(6) +#define EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX BIT(5)/** * @flags: flags for this exec queue, should statically setup aside from ban
@@ -157,13 +155,6 @@ struct xe_exec_queue {/** @ban_reason: Bitmask of ban reasons (DRM_XE_EXEC_QUEUE_BAN_REASON_*) */ atomic_t ban_reason; -union { -/** @multi_gt_list: list head for VM bind engines if multi-GT */ -struct list_head multi_gt_list; -/** @multi_gt_link: link for VM bind engines if multi-GT */ -struct list_head multi_gt_link; - }; -union {/** @execlist: execlist backend specific state for exec queue */ struct xe_execlist_exec_queue *execlist;
@@ -230,7 +221,8 @@ struct xe_exec_queue {#define XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT 0#define XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT 1 -#define XE_EXEC_QUEUE_TLB_INVAL_COUNT (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT + 1) +#define XE_EXEC_QUEUE_TLB_INVAL_COUNT \ + ((XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT + 1) * 2)/** @tlb_inval: TLB invalidations exec queue state */ struct {
drivers/gpu/drm/xe/xe_pt.c +15 -7
@@ -2710,12 +2710,18 @@ static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops;#endif static struct xe_dep_scheduler *to_dep_scheduler (struct xe_exec_queue *q, -struct xe_gt *gt) +struct xe_tile *tile, +struct xe_gt *gt, +unsigned int *type) { +int tile_ofs = tile->id * (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); +if (xe_gt_is_media_type (gt)) -return q->tlb_inval[XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ].dep_scheduler; + *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ; +else + *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ; -return q->tlb_inval[XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ].dep_scheduler; +return q->tlb_inval[*type].dep_scheduler; }/**
@@ -2740,6 +2746,7 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)struct xe_tlb_inval_job *ijob =NULL , *mjob =NULL ;struct xe_range_fence *rfence;struct xe_vma_op *op; +unsigned int type;int err =0 , i;struct xe_migrate_pt_update update = { .ops = pt_update_ops->needs_svm_lock ?
@@ -2766,13 +2773,13 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)if (pt_update_ops->needs_invalidation) {struct xe_dep_scheduler *dep_scheduler = -to_dep_scheduler (q, tile->primary_gt); +to_dep_scheduler (q, tile, tile->primary_gt, &type); ijob =xe_tlb_inval_job_create (q, &tile->primary_gt->tlb_inval, dep_scheduler, vm, pt_update_ops->start, pt_update_ops->last, -XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); + type);if (IS_ERR (ijob)) { err =PTR_ERR (ijob); goto kill_vm_tile1;
@@ -2791,14 +2798,15 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) }if (tile->media_gt) { - dep_scheduler =to_dep_scheduler (q, tile->media_gt); + dep_scheduler =to_dep_scheduler (q, tile, + tile->media_gt, &type); mjob =xe_tlb_inval_job_create (q, &tile->media_gt->tlb_inval, dep_scheduler, vm, pt_update_ops->start, pt_update_ops->last, -XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ); + type);if (IS_ERR (mjob)) { err =PTR_ERR (mjob); goto free_ijob;
drivers/gpu/drm/xe/xe_sync.c +3 -17
@@ -345,15 +345,9 @@ xe_sync_in_fence_get(struct xe_sync_entry *sync, int num_sync,return ERR_PTR (-EOPNOTSUPP );if (q->flags &EXEC_QUEUE_FLAG_VM ) { -struct xe_exec_queue *__q; -struct xe_tile *tile; -u8 id; - -for_each_tile (tile, vm->xe, id) { + num_fence++; +for_each_tlb_inval (q, i) num_fence++; -for_each_tlb_inval (i) - num_fence++; - } fences =kmalloc_objs (*fences, num_fence);if (!fences)
@@ -361,17 +355,9 @@ xe_sync_in_fence_get(struct xe_sync_entry *sync, int num_sync, fences[current_fence++] =xe_exec_queue_last_fence_get (q, vm); -for_each_tlb_inval (i) +for_each_tlb_inval (q, i) fences[current_fence++] =xe_exec_queue_tlb_inval_last_fence_get (q, vm, i); -list_for_each_entry (__q, &q->multi_gt_list, - multi_gt_link) { - fences[current_fence++] = -xe_exec_queue_last_fence_get (__q, vm); -for_each_tlb_inval (i) - fences[current_fence++] = -xe_exec_queue_tlb_inval_last_fence_get (__q, vm, i); - }xe_assert (vm->xe, current_fence == num_fence); cf = dma_fence_array_create(num_fence, fences,
drivers/gpu/drm/xe/xe_tlb_inval_job.c +7 -8
@@ -39,8 +39,8 @@ struct xe_tlb_inval_job {u64 start;/** @end: End address to invalidate */ u64 end; -/** @type: GT type */ -int type; +/** @idx: Index of tlb invalidation */ +int idx;/** @fence_armed: Fence has been armed */ bool fence_armed; };
@@ -87,7 +87,7 @@ static const struct xe_dep_job_ops dep_job_ops = {* @vm: VM which TLB invalidation is being issued for * @start: Start address to invalidate * @end: End address to invalidate -* @type: GT type +* @idx: Index of tlb invalidation * * Create a TLB invalidation job and initialize internal fields. The caller is * responsible for releasing the creation reference.
@@ -97,7 +97,7 @@ static const struct xe_dep_job_ops dep_job_ops = {struct xe_tlb_inval_job *xe_tlb_inval_job_create (struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval,struct xe_dep_scheduler *dep_scheduler, -struct xe_vm *vm,u64 start,u64 end,int type) +struct xe_vm *vm,u64 start,u64 end,int idx) {struct xe_tlb_inval_job *job; struct drm_sched_entity *entity =
@@ -105,8 +105,7 @@ xe_tlb_inval_job_create(struct xe_exec_queue *q, struct xe_tlb_inval *tlb_inval,struct xe_tlb_inval_fence *ifence;int err; -xe_assert (vm->xe, type ==XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT || - type ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT ); job =kmalloc_obj (*job);if (!job)
@@ -120,7 +119,7 @@ xe_tlb_inval_job_create(struct xe_exec_queue *q, struct xe_tlb_inval *tlb_inval, job->fence_armed = false;xe_page_reclaim_list_init (&job->prl); job->dep.ops = &dep_job_ops; - job->type = type; + job->idx = idx;kref_init (&job->refcount);xe_exec_queue_get (q);/* Pairs with put in xe_tlb_inval_job_destroy */ xe_vm_get (vm);/* Pairs with put in xe_tlb_inval_job_destroy */
@@ -280,7 +279,7 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job,/* Let the upper layers fish this out */ xe_exec_queue_tlb_inval_last_fence_set (job->q, job->vm, &job->dep.drm.s_fence->finished, - job->type); + job->idx);xe_migrate_job_unlock (m, job->q);
drivers/gpu/drm/xe/xe_tlb_inval_job.h +1 -1
@@ -20,7 +20,7 @@ struct xe_vm;struct xe_tlb_inval_job *xe_tlb_inval_job_create (struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval,struct xe_dep_scheduler *dep_scheduler, -struct xe_vm *vm,u64 start,u64 end,int type); +struct xe_vm *vm,u64 start,u64 end,int idx);void xe_tlb_inval_job_add_page_reclaim (struct xe_tlb_inval_job *job,struct xe_page_reclaim_list *prl);
drivers/gpu/drm/xe/xe_vm.c +31 -34
120 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +1 -1
@@ -262,7 +262,7 @@ struct xe_vm {struct xe_device *xe;/* exec queue used for (un)binding vma's */ -struct xe_exec_queue *q[XE_MAX_TILES_PER_DEVICE ]; +struct xe_exec_queue *q;/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ struct ttm_lru_bulk_move lru_bulk_move;
Signature · 1 line
2.34.1
RE: [PATCH v6 15/24] drm/xe: Make bind queues operate cross-tile
Quoted 169 lines · Matthew Brost
Since bind jobs execute on the CPU rather than the GPU, maintaining a per-tile bind queue no longer provides value. Convert the driver to use a single bind queue shared across tiles. The primary change is routing all GT TLB invalidations through this unified bind queue.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_exec_queue.c | 142 +++++++++-------------- drivers/gpu/drm/xe/xe_exec_queue.h | 14 +-- drivers/gpu/drm/xe/xe_exec_queue_types.h | 20 +--- drivers/gpu/drm/xe/xe_pt.c | 22 ++-- drivers/gpu/drm/xe/xe_sync.c | 20 +--- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 15 ++- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 2 +- drivers/gpu/drm/xe/xe_vm.c | 65 +++++------ drivers/gpu/drm/xe/xe_vm_types.h | 2 +- 9 files changed, 125 insertions(+), 177 deletions(-)drivers/gpu/drm/xe/xe_exec_queue.c +2
-3@@ -142,9 +142,8 @@ static void __xe_exec_queue_free(struct xe_exec_queue *q) {int i; -for (i =0 ; i <XE_EXEC_QUEUE_TLB_INVAL_COUNT ; ++i) -if (q->tlb_inval[i].dep_scheduler) -xe_dep_scheduler_fini (q->tlb_inval[i].dep_scheduler); +for_each_tlb_inval (q, i) +xe_dep_scheduler_fini (q->tlb_inval[i].dep_scheduler);if (xe_exec_queue_uses_pxp (q)) xe_pxp_exec_queue_remove(gt_to_xe(q->gt)->pxp, q); @@ -166,31 +165,34 @@ static void
__xe_exec_queue_free(struct xe_exec_queue *q)static int
alloc_dep_schedulers(struct xe_device *xe, struct xe_exec_queue *q)
{drivers/gpu/drm/xe/xe_exec_queue.c +16
-15-struct xe_tile *tile =gt_to_tile (q->gt); -int i; +struct xe_tile *tile; +int i =0 , j; +u8 id; -for (i =0 ; i <XE_EXEC_QUEUE_TLB_INVAL_COUNT ; ++i) { -struct xe_dep_scheduler *dep_scheduler; -struct xe_gt *gt; -struct workqueue_struct *wq; +for_each_tile (tile, xe, id) { +for (j =0 ; j < (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); ++j, ++i) { +struct xe_dep_scheduler *dep_scheduler; +struct xe_gt *gt; +struct workqueue_struct *wq; -if (i ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ) - gt = tile->primary_gt; -else - gt = tile->media_gt; +if (j ==XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ) + gt = tile->primary_gt; +else + gt = tile->media_gt; -if (!gt) -continue ; +if (!gt) +continue ; - wq = gt->tlb_inval.job_wq; + wq = gt->tlb_inval.job_wq;#define MAX_TLB_INVAL_JOBS 16/* Picking a reasonable value */ - dep_scheduler = xe_dep_scheduler_create(xe, wq, q->name, -MAX_TLB_INVAL_JOBS);drivers/gpu/drm/xe/xe_exec_queue.c +1
-2-if (IS_ERR (dep_scheduler)) -return PTR_ERR (dep_scheduler); + dep_scheduler = xe_dep_scheduler_create(xe, wq, q-Quoted 1 line
name,
+
MAX_TLB_INVAL_JOBS);drivers/gpu/drm/xe/xe_exec_queue.c +4
-2+if (IS_ERR (dep_scheduler)) +return PTR_ERR (dep_scheduler); - q->tlb_inval[i].dep_scheduler = dep_scheduler; + q->tlb_inval[i].dep_scheduler = dep_scheduler; + } }#undef MAX_TLB_INVAL_JOBS@@ -224,7 +226,6 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe, q->ops = gt->exec_queue_ops;INIT_LIST_HEAD (&q->lr.link);INIT_LIST_HEAD (&q->vm_exec_queue_link); -INIT_LIST_HEAD (&q->multi_gt_link);INIT_LIST_HEAD (&q->hw_engine_group_link);INIT_LIST_HEAD (&q->pxp.link);drivers/gpu/drm/xe/xe_exec_queue.c +0
-0spin_lock_init (&q->multi_queue.lock);@@ -636,7 +637,6 @@
ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO); voidxe_exec_queue_destroy(struct kref *ref){
structxe_exec_queue*q = container_of(ref, structxe_exec_queue,
refcount);drivers/gpu/drm/xe/xe_exec_queue.c +0
-1-struct xe_exec_queue *eq, *next;int i; xe_assert(gt_to_xe(q->gt), atomic_read(&q->job_cnt) == 0); @@ -648,15 +648,9 @@ void
xe_exec_queue_destroy(struct kref *ref)drivers/gpu/drm/xe/xe_exec_queue.c +1
-8xe_pxp_exec_queue_remove (gt_to_xe (q->gt)->pxp, q);xe_exec_queue_last_fence_put_unlocked (q); -for_each_tlb_inval (i) +for_each_tlb_inval (q, i)xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, i); -if (!(q->flags &EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD )) { -list_for_each_entry_safe (eq, next, &q->multi_gt_list, - multi_gt_link) -xe_exec_queue_put (eq); - } -if (q->user_vm) {xe_vm_put (q->user_vm); q->user_vm =NULL ;@@ -1343,7 +1337,6 @@ int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data,u64_to_user_ptr (args->instances);struct xe_hw_engine *hwe;struct xe_vm *vm; -struct xe_tile *tile;struct xe_exec_queue *q =NULL ;u32 logical_mask;drivers/gpu/drm/xe/xe_exec_queue.c +0
-23u32 flags =0 ;@@ -1392,31 +1385,16 @@ int xe_exec_queue_create_ioctl(struct drm_device *dev, void *data, return -ENOENT ; } -for_each_tile (tile, xe, id) { -struct xe_exec_queue *new; - - flags |=EXEC_QUEUE_FLAG_VM ; -if (id) - flags |=EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD ; - - new =xe_exec_queue_create_bind (xe, tile, vm, flags, - args->extensions); -if (IS_ERR (new)) { -up_read (&vm->lock); -xe_vm_put (vm); - err =PTR_ERR (new); -if (q) - goto put_exec_queue; -return err; - } -if (id ==0 ) - q = new; -else -list_add_tail (&new->multi_gt_list, - &q->multi_gt_link); - }
drivers/gpu/drm/xe/xe_exec_queue.c int xe_exec_queue_create_ioctl(struct · line 1399
Quoted 10 lines
drivers/gpu/drm/xe/xe_exec_queue.c +7
-0+ flags |=EXEC_QUEUE_FLAG_VM ; + + q =xe_exec_queue_create_bind (xe,xe_device_get_root_tile (xe), + vm, flags, args->extensions);up_read (&vm->lock);xe_vm_put (vm); +if (IS_ERR (q)) { + err =PTR_ERR (q); +return err;
nit: return PTR_ERR(q);
Quoted 58 lines
drivers/gpu/drm/xe/xe_exec_queue.c +1
-0+ }drivers/gpu/drm/xe/xe_exec_queue.c +1
-8}else { logical_mask = calc_validate_logical_mask(xe, eci, args->width,@@ -1638,14 +1616,6 @@ void xe_exec_queue_update_run_ticks(structxe_exec_queue *q) */ void xe_exec_queue_kill (struct xe_exec_queue *q) { -struct xe_exec_queue *eq = q, *next; - -list_for_each_entry_safe (eq, next, &eq->multi_gt_list, - multi_gt_link) { - q->ops->kill (eq); -xe_vm_remove_compute_exec_queue (q->vm, eq); - } - q->ops->kill (q); xe_vm_remove_compute_exec_queue(q->vm, q); } @@ -1806,42 +1776,40 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *q,struct
xe_vm*vm,
*xe_exec_queue_tlb_inval_last_fence_put()- Drop ref to last TLB invalidation
fence
* @q: The exec queue
* @vm: The VM the engine does a bind for
- * @type: Either primary or media GT
+ * @idx: Index of tlb invalidation
*/
void xe_exec_queue_tlb_inval_last_fence_put(structxe_exec_queue*q,drivers/gpu/drm/xe/xe_exec_queue.c +1
-2struct xe_vm *vm, - unsigned int type) + unsigned int idx) {xe_exec_queue_last_fence_lockdep_assert (q ,vm ); - xe_assert(vm->xe, type == XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT||
drivers/gpu/drm/xe/xe_exec_queue.c +3
-3- type == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT ); -xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, type); +xe_exec_queue_tlb_inval_last_fence_put_unlocked (q, idx); }/** * xe_exec_queue_tlb_inval_last_fence_put_unlocked() - Drop ref to last TLB * invalidation fence unlocked * @q: The exec queue -* @type: Either primary or media GT +* @idx: Index of tlb invalidation * * Only safe to be called from xe_exec_queue_destroy(). */ void xe_exec_queue_tlb_inval_last_fence_put_unlocked(struct
xe_exec_queue*q,drivers/gpu/drm/xe/xe_exec_queue.c +1
-2- unsigned int type) + unsigned int idx) { - xe_assert(gt_to_xe(q->gt), type ==
XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT||
drivers/gpu/drm/xe/xe_exec_queue.c
Quoted 2 lines
drivers/gpu/drm/xe/xe_exec_queue.c +1
-1- type == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT); +xe_assert (q->vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT );
"gt_to_xe(q->gt)" is better than "q->vm->xe", as vm might be NULL.
Quoted 192 lines
drivers/gpu/drm/xe/xe_exec_queue.c +2
-2-dma_fence_put (q->tlb_inval[type].last_fence); - q->tlb_inval[type].last_fence =NULL ; +dma_fence_put (q->tlb_inval[idx].last_fence); + q->tlb_inval[idx].last_fence =NULL ; }/** * xe_exec_queue_tlb_inval_last_fence_get() - Get last fence for TLB invalidation
* @q: The exec queue
* @vm: The VM the engine does a bind for
- * @type: Either primary or media GT
+ * @idx: Index of tlb invalidation
*
* Get last fence, takes a ref
*drivers/gpu/drm/xe/xe_exec_queue.c +1
-2@@ -1849,22 +1817,21 @@ voidxe_exec_queue_tlb_inval_last_fence_put_unlocked(struct xe_exec_queue *q, */ struct dma_fence *xe_exec_queue_tlb_inval_last_fence_get (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type) +unsigned int idx) {struct dma_fence *fence;xe_exec_queue_last_fence_lockdep_assert (q, vm); - xe_assert(vm->xe, type == XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT||
drivers/gpu/drm/xe/xe_exec_queue.c +9
-9- type == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT );xe_assert (vm->xe, q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); -if (q->tlb_inval[type].last_fence && +if (q->tlb_inval[idx].last_fence &&test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , - &q->tlb_inval[type].last_fence->flags)) -xe_exec_queue_tlb_inval_last_fence_put (q, vm, type); + &q->tlb_inval[idx].last_fence->flags)) +xe_exec_queue_tlb_inval_last_fence_put (q, vm, idx); - fence = q->tlb_inval[type].last_fence ?:dma_fence_get_stub (); + fence = q->tlb_inval[idx].last_fence ?:dma_fence_get_stub ();dma_fence_get (fence);return fence; }@@ -1874,26 +1841,25 @@ struct dma_fence*xe_exec_queue_tlb_inval_last_fence_get(struct xe_exec_queue *q * @q: The exec queue * @vm: The VM the engine does a bind for * @fence: The fence -* @type: Either primary or media GT +* @idx: Index of tlb invalidation * -* Set the last fence for the tlb invalidation type on the queue. Increases +* Set the last fence for the tlb invalidation client on the queue. +Increases * reference count for fence, when closing queue * xe_exec_queue_tlb_inval_last_fence_put should be called. */ void xe_exec_queue_tlb_inval_last_fence_set (struct xe_exec_queue *q,struct xe_vm *vm,struct dma_fence *fence, -unsigned int type) +unsigned int idx) {xe_exec_queue_last_fence_lockdep_assert (q, vm); - xe_assert(vm->xe, type == XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT||
drivers/gpu/drm/xe/xe_exec_queue.c +3
-3- type == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT );xe_assert (vm->xe, q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE ));xe_assert (vm->xe, !dma_fence_is_container (fence)); -xe_exec_queue_tlb_inval_last_fence_put (q, vm, type); - q->tlb_inval[type].last_fence =dma_fence_get (fence); +xe_exec_queue_tlb_inval_last_fence_put (q, vm, idx); + q->tlb_inval[idx].last_fence =dma_fence_get (fence); }/** drivers/gpu/drm/xe/xe_exec_queue.h +3
-3@@ -14,9 +14,9 @@ struct drm_file;struct xe_device ;struct xe_file ; -#define for_each_tlb_inval (__i) \ - for (__i = XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT; \ - __i <= XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT; ++__i) +#define for_each_tlb_inval (__q, __i) \ + for (__i = 0; __i < XE_EXEC_QUEUE_TLB_INVAL_COUNT; ++__i) \ + for_each_if((__q)->tlb_inval[__i].dep_scheduler) struct xe_exec_queue *xe_exec_queue_create(struct xe_device *xe, struct xe_vm *vm,drivers/gpu/drm/xe/xe_exec_queue.h +4
-4u32 logical_mask, u16 width,@@ -141,19 +141,19 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *e, struct xe_vm *vm, void xe_exec_queue_tlb_inval_last_fence_put(struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type); +unsigned int idx);void xe_exec_queue_tlb_inval_last_fence_put_unlocked (struct xe_exec_queue *q, -unsigned int type); +unsigned int idx);struct dma_fence *xe_exec_queue_tlb_inval_last_fence_get (struct xe_exec_queue *q,struct xe_vm *vm, -unsigned int type); +unsigned int idx);void xe_exec_queue_tlb_inval_last_fence_set (struct xe_exec_queue *q,struct xe_vm *vm,struct dma_fence *fence, -unsigned int type); +unsigned int idx);void
xe_exec_queue_update_run_ticks(struct xe_exec_queue *q);drivers/gpu/drm/xe/xe_exec_queue_types.h +0
-0@@ -137,16 +137,14 @@ struct xe_exec_queue {#define EXEC_QUEUE_FLAG_KERNEL BIT(0)/* for VM jobs. Caller needs to hold rpm ref when creating queue with this flag */
#defineEXEC_QUEUE_FLAG_VMBIT(1)drivers/gpu/drm/xe/xe_exec_queue_types.h +4
-6-/* child of VM queue for multi-tile VM jobs */ -#define EXEC_QUEUE_FLAG_BIND_ENGINE_CHILD BIT(2)/* kernel exec_queue only, set priority to highest level */ -#define EXEC_QUEUE_FLAG_HIGH_PRIORITY BIT(3) +#define EXEC_QUEUE_FLAG_HIGH_PRIORITY BIT(2)/* flag to indicate low latency hint to guc */ -#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(4) +#define EXEC_QUEUE_FLAG_LOW_LATENCY BIT(3)/* for migration (kernel copy, clear, bind) jobs */ -#define EXEC_QUEUE_FLAG_MIGRATE BIT(5) +#define EXEC_QUEUE_FLAG_MIGRATE BIT(4)/* for programming COMMON_SLICE_CHICKEN3 on first submission */ -#define EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX BIT(6) +#define EXEC_QUEUE_FLAG_DISABLE_STATE_CACHE_PERF_FIX BIT(5)/** * @flags: flags for this exec queue, should statically setup aside from ban @@ -157,13 +155,6 @@ struct
xe_exec_queue{
/** @ban_reason: Bitmask of ban reasons
(DRM_XE_EXEC_QUEUE_BAN_REASON_*) */drivers/gpu/drm/xe/xe_exec_queue_types.h +0
-7atomic_t ban_reason; -union { -/** @multi_gt_list: list head for VM bind engines if multi-GT */ -struct list_head multi_gt_list; -/** @multi_gt_link: link for VM bind engines if multi-GT */ -struct list_head multi_gt_link; - }; -union {/** @execlist: execlist backend specific state for exec queue */ struct xe_execlist_exec_queue *execlist; @@ -230,7 +221,8@@ struct
xe_exec_queue{#define
XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT0
#defineXE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT1
-#defineXE_EXEC_QUEUE_TLB_INVAL_COUNT(XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT + 1)drivers/gpu/drm/xe/xe_exec_queue_types.h +2
-0+#define XE_EXEC_QUEUE_TLB_INVAL_COUNT \ + ((XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT + 1) * 2)/** @tlb_inval: TLB invalidations exec queue state */ struct {drivers/gpu/drm/xe/xe_pt.c index renamed from drivers/gpu/drm/xe/xe_pt.c +0
-0
c17d5f96f00a..bdeffd6f1828100644drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -2710,12 +2710,18 @@ static const struct xe_migrate_pt_update_ops
svm_userptr_migrate_ops; #endifstatic struct
xe_dep_scheduler*to_dep_scheduler(structxe_exec_queue*q,
drivers/gpu/drm/xe/xe_pt.c static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops; · line 2715
Quoted 4 lines
drivers/gpu/drm/xe/xe_pt.c +3
-1- struct xe_gt *gt) +struct xe_tile *tile, + struct xe_gt *gt, + unsigned int *type)
@tile looks redundant since it could get from gt_to_tile(). As this function is removed in later patch, so feel free to ignore it.
With upper 2 fixed:
Quoted 351 lines
{
drivers/gpu/drm/xe/xe_pt.c +2
-1+int tile_ofs = tile->id * (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); +if (xe_gt_is_media_type (gt)) - return q-Quoted 1 line
tlb_inval[XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT].dep_scheduler;
drivers/gpu/drm/xe/xe_pt.c +3
-1+ *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ; + else + *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ; - return q-Quoted 1 line
tlb_inval[XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT].dep_scheduler;
drivers/gpu/drm/xe/xe_pt.c +2
-0+return q->tlb_inval[*type].dep_scheduler; }/** @@ -2740,6 +2746,7 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)struct xe_tlb_inval_job *ijob =NULL , *mjob =NULL ;struct xe_range_fence *rfence;struct xe_vma_op *op; +unsigned int type;int err =0 , i;struct xe_migrate_pt_update update = {drivers/gpu/drm/xe/xe_pt.c +2
-2.ops = pt_update_ops->needs_svm_lock ?@@ -2766,13 +2773,13 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)if (pt_update_ops->needs_invalidation) {struct xe_dep_scheduler *dep_scheduler = -to_dep_scheduler (q, tile->primary_gt); +to_dep_scheduler (q, tile, tile->primary_gt, &type); ijob =xe_tlb_inval_job_create (q, &tile->primary_gt->tlb_inval, dep_scheduler, vm, pt_update_ops->start, pt_update_ops->last, -XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ); + type);if (IS_ERR (ijob)) {drivers/gpu/drm/xe/xe_pt.c +2
-2err =PTR_ERR (ijob); goto kill_vm_tile1;@@ -2791,14 +2798,15 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) }if (tile->media_gt) { - dep_scheduler =to_dep_scheduler (q, tile->media_gt); + dep_scheduler =to_dep_scheduler (q, tile, + tile->media_gt, &type); mjob = xe_tlb_inval_job_create(q, &tile->media_gt- >tlb_inval, dep_scheduler, vm, pt_update_ops->start, pt_update_ops->last, -
XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT);drivers/gpu/drm/xe/xe_pt.c +1
-0+ type);if (IS_ERR (mjob)) { err =PTR_ERR (mjob); goto free_ijob;drivers/gpu/drm/xe/xe_sync.c +2
-8@@ -345,15 +345,9 @@ xe_sync_in_fence_get(struct xe_sync_entry *sync,int num_sync, return ERR_PTR(-EOPNOTSUPP);if (q->flags &EXEC_QUEUE_FLAG_VM ) { -struct xe_exec_queue *__q; -struct xe_tile *tile; -u8 id; - -for_each_tile (tile, vm->xe, id) { + num_fence++; +for_each_tlb_inval (q, i) num_fence++; -for_each_tlb_inval (i) - num_fence++; - } fences =kmalloc_objs (*fences, num_fence);drivers/gpu/drm/xe/xe_sync.c +1
-9if (!fences)@@ -361,17 +355,9 @@ xe_sync_in_fence_get(struct xe_sync_entry *sync,int num_sync, fences[current_fence++] =xe_exec_queue_last_fence_get (q, vm); -for_each_tlb_inval (i) +for_each_tlb_inval (q, i) fences[current_fence++] =xe_exec_queue_tlb_inval_last_fence_get (q, vm, i); -list_for_each_entry (__q, &q->multi_gt_list, - multi_gt_link) { - fences[current_fence++] = -xe_exec_queue_last_fence_get (__q, vm); -for_each_tlb_inval (i) - fences[current_fence++] = -xe_exec_queue_tlb_inval_last_fence_get (__q, vm, i); - }drivers/gpu/drm/xe/xe_sync.c +0
-0xe_assert (vm->xe, current_fence == num_fence); cf = dma_fence_array_create(num_fence, fences, diff --git
a/drivers/gpu/drm/xe/xe_tlb_inval_job.cb/drivers/gpu/drm/xe/xe_tlb_inval_job.c
index04d21015cd5d..81f560068d3c100644drivers/gpu/drm/xe/xe_tlb_inval_job.c +4
-5@@ -39,8 +39,8 @@ struct xe_tlb_inval_job {u64 start;/** @end: End address to invalidate */ u64 end; -/** @type: GT type */ -int type; +/** @idx: Index of tlb invalidation */ +int idx;/** @fence_armed: Fence has been armed */ bool fence_armed; };@@ -87,7 +87,7 @@ static const struct xe_dep_job_ops dep_job_ops = {* @vm: VM which TLB invalidation is being issued for * @start: Start address to invalidate * @end: End address to invalidate -* @type: GT type +* @idx: Index of tlb invalidation * * Create a TLB invalidation job and initialize internal fields. The caller is * responsible for releasing the creation reference. @@ -97,7 +97,7 @@ static const struct xe_dep_job_ops dep_job_ops = {struct xe_tlb_inval_job *xe_tlb_inval_job_create (struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval,struct xe_dep_scheduler *dep_scheduler, -struct xe_vm *vm,u64 start,u64 end,int type) +struct xe_vm *vm,u64 start,u64 end,int idx) {struct xe_tlb_inval_job *job; struct drm_sched_entity *entity =@@ -105,8 +105,7 @@ xe_tlb_inval_job_create(struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval, struct xe_tlb_inval_fence *ifence;int err; - xe_assert(vm->xe, type == XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT||
drivers/gpu/drm/xe/xe_tlb_inval_job.c +2
-2- type == XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT); +xe_assert (vm->xe, idx <XE_EXEC_QUEUE_TLB_INVAL_COUNT ); job =kmalloc_obj (*job);if (!job)@@ -120,7 +119,7 @@ xe_tlb_inval_job_create(struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval, job->fence_armed = false;xe_page_reclaim_list_init (&job->prl); job->dep.ops = &dep_job_ops; - job->type = type; + job->idx = idx;kref_init (&job->refcount);xe_exec_queue_get (q);/* Pairs with put in xe_tlb_inval_job_destroy */
xe_vm_get(vm); /* Pairs with put inxe_tlb_inval_job_destroy*/drivers/gpu/drm/xe/xe_tlb_inval_job.c +1
-1@@ -280,7 +279,7 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job,/* Let the upper layers fish this out */ xe_exec_queue_tlb_inval_last_fence_set (job->q, job->vm, &job->dep.drm.s_fence->finished, - job->type); + job->idx);xe_migrate_job_unlock (m, job->q);drivers/gpu/drm/xe/xe_tlb_inval_job.h +1
-1@@ -20,7 +20,7 @@ struct xe_vm;struct xe_tlb_inval_job *xe_tlb_inval_job_create (struct xe_exec_queue *q,struct xe_tlb_inval *tlb_inval,struct xe_dep_scheduler *dep_scheduler, -struct xe_vm *vm,u64 start,u64 end,int type); +struct xe_vm *vm,u64 start,u64 end,int idx); void xe_tlb_inval_job_add_page_reclaim(struct xe_tlb_inval_job *job,struct
xe_page_reclaim_list*prl); diff --gita/drivers/gpu/drm/xe/xe_vm.cb/drivers/gpu/drm/xe/xe_vm.cindex13e984ac4e4f..433a0a681556100644drivers/gpu/drm/xe/xe_vm.c +1
-1@@ -1815,7 +1815,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) struct xe_exec_queue *q;u32 create_flags =EXEC_QUEUE_FLAG_VM ; -if (!vm->pt_root[id]) +if (!vm->pt_root[id] || vm->q)continue ;drivers/gpu/drm/xe/xe_vm.c +11
-17if (!xef)/* Not from userspace */ @@ -1826,7 +1826,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) err =PTR_ERR (q); goto err_close; } - vm->q[id] = q; + vm->q = q; } }@@ -1933,24 +1933,18 @@ void xe_vm_close_and_put(struct xe_vm *vm)if (xe_vm_in_fault_mode (vm))xe_svm_close (vm); -down_write (&vm->lock); -for_each_tile (tile, xe, id) { -if (vm->q[id]) { -int i; +if (vm->q) { +int i; -xe_exec_queue_last_fence_put (vm->q[id], vm); -for_each_tlb_inval (i) -xe_exec_queue_tlb_inval_last_fence_put (vm- >q[id], vm, i); - } - } -up_write (&vm->lock); +down_write (&vm->lock); +xe_exec_queue_last_fence_put (vm->q, vm); +for_each_tlb_inval (vm->q, i) +xe_exec_queue_tlb_inval_last_fence_put (vm->q, vm, i); +up_write (&vm->lock); -for_each_tile (tile, xe, id) { -if (vm->q[id]) { -xe_exec_queue_kill (vm->q[id]); -xe_exec_queue_put (vm->q[id]); - vm->q[id] =NULL ; - } +xe_exec_queue_kill (vm->q); +xe_exec_queue_put (vm->q); + vm->q =NULL ; }drivers/gpu/drm/xe/xe_vm.c +0
-0down_write (&vm->lock);@@ -2084,7 +2078,7 @@ u64 xe_vm_pdp4_descriptor(struct xe_vm *vm,struct
xe_tile*tile) static structxe_exec_queue*to_wait_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q){drivers/gpu/drm/xe/xe_vm.c +4
-7-return q ? q : vm->q[0 ]; +return q ? q : vm->q; } static struct xe_user_fence *@@ -3527,13 +3521,10 @@ static int vm_ops_setup_tile_args(struct xe_vm *vm, struct xe_vma_ops *vops)if (vops->pt_update_ops[id].q)continue ; -if (q) { +if (q) vops->pt_update_ops[id].q = q; -if (vm->pt_root[id] && !list_empty (&q- >multi_gt_list)) - q =list_next_entry (q, multi_gt_list); - }else { - vops->pt_update_ops[id].q = vm->q[id]; - } + else + vops->pt_update_ops[id].q = vm->q; }drivers/gpu/drm/xe/xe_vm.c +5
-5return number_tiles;@@ -3553,15 +3544,15 @@ static struct dma_fence *ops_execute(struct xe_vm *vm, if (number_tiles == 0)return ERR_PTR (-ENODATA ); -for_each_tile (tile, vm->xe, id) { +for_each_tile (tile, vm->xe, id) ++n_fence; -if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) -for_each_tlb_inval (i) - ++n_fence; +if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) { +for_each_tlb_inval (vops->pt_update_ops[0 ].q, i) + ++n_fence; } - fences =kmalloc_objs (*fences, n_fence); + fences =kcalloc (n_fence,sizeof (*fences),GFP_KERNEL );if (!fences) { fence =ERR_PTR (-ENOMEM );drivers/gpu/drm/xe/xe_vm.c +7
-3goto err_trace;@@ -3603,9 +3594,15 @@ static struct dma_fence *ops_execute(structxe_vm *vm, continue;xe_migrate_job_lock (tile->migrate, q); -for_each_tlb_inval (i) - fences[current_fence++] = -xe_exec_queue_tlb_inval_last_fence_get (q, vm, i); +for_each_tlb_inval (q, i) { +if (i >= (tile->id +1 ) *XE_MAX_GT_PER_TILE || + i < tile->id *XE_MAX_GT_PER_TILE ) +continue ; + + fences[current_fence++] = fence ? + xe_exec_queue_tlb_inval_last_fence_get(q,vm, i) :
drivers/gpu/drm/xe/xe_vm.c +3
-1+dma_fence_get_stub (); + }xe_migrate_job_unlock (tile->migrate, q); }@@ -4135,7 +4132,7 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) syncs_user =u64_to_user_ptr (args->syncs);for (num_syncs =0 ; num_syncs < args->num_syncs; num_syncs++) { -struct xe_exec_queue *__q = q ?: vm->q[0 ]; +struct xe_exec_queue *__q = q ?: vm->q; err = xe_sync_entry_parse(xe, xef, &syncs[num_syncs],drivers/gpu/drm/xe/xe_vm.c +0
-0&syncs_user[num_syncs],drivers/gpu/drm/xe/xe_vm_types.h +1
-2@@ -262,7 +262,7 @@ struct xe_vm {struct xe_device *xe;/* exec queue used for (un)binding vma's */ -struct xe_exec_queue *q[XE_MAX_TILES_PER_DEVICE ]; +struct xe_exec_queue *q;/** @lru_bulk_move: Bulk LRU move list for this VM's BOs */ struct ttm_lru_bulk_move lru_bulk_move; --2.34.1
Matthew Brost
maintainerPATCH v6 16/24 drm/xe: Add CPU bind layer
In reply to Matthew BrostWith CPU binds, it no longer makes sense to implement CPU bind handling in the migrate layer, as these operations are entirely decoupled from hardware. Introduce a dedicated CPU bind layer stored at the device level.
Since CPU binds are tile-independent, update the PT layer to generate a single bind job even when pages are mirrored across tiles.
This patch is large because the refactor touches multiple file / layers and ensures functional equivalence before and after the change.
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_cpu_bind.c | 298 +++++++++++++ drivers/gpu/drm/xe/xe_cpu_bind.h | 118 +++++ drivers/gpu/drm/xe/xe_device.c | 5 + drivers/gpu/drm/xe/xe_device_types.h | 4 + drivers/gpu/drm/xe/xe_exec_queue.c | 3 +- drivers/gpu/drm/xe/xe_guc_submit.c | 41 +- drivers/gpu/drm/xe/xe_migrate.c | 246 ----------- drivers/gpu/drm/xe/xe_migrate.h | 94 ---- drivers/gpu/drm/xe/xe_pt.c | 549 ++++++++++++------------ drivers/gpu/drm/xe/xe_pt.h | 8 +- drivers/gpu/drm/xe/xe_pt_types.h | 14 - drivers/gpu/drm/xe/xe_sched_job.c | 10 +- drivers/gpu/drm/xe/xe_sched_job_types.h | 11 +- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 13 +- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 2 - drivers/gpu/drm/xe/xe_vm.c | 155 ++----- drivers/gpu/drm/xe/xe_vm_types.h | 10 +- 18 files changed, 813 insertions(+), 769 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.c create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.h
drivers/gpu/drm/xe/Makefile +1 -0
@@ -35,6 +35,7 @@ $(obj)/generated/%_device_wa_oob.c $(obj)/generated/%_device_wa_oob.h: $(obj)/xe xe-y += xe_bb.o \ xe_bo.o \ xe_bo_evict.o \ + xe_cpu_bind.o \ xe_dep_scheduler.o \ xe_devcoredump.o \ xe_device.o \
drivers/gpu/drm/xe/xe_cpu_bind.c new file +298 -0
298 lines not on this page · raw message
drivers/gpu/drm/xe/xe_cpu_bind.h new file +118 -0
118 lines not on this page · raw message
drivers/gpu/drm/xe/xe_device.c +5 -0
@@ -27,6 +27,7 @@#include "xe_bo_evict.h" #include "xe_configfs.h" #include "xe_debugfs.h" +#include "xe_cpu_bind.h" #include "xe_defaults.h" #include "xe_devcoredump.h" #include "xe_device_sysfs.h"
@@ -1075,6 +1076,10 @@ int xe_device_probe(struct xe_device *xe)if (err)return err; + err =xe_cpu_bind_init (xe); +if (err) +return err; + err =xe_pagefault_init (xe);if (err)return err;
drivers/gpu/drm/xe/xe_device_types.h +4 -0
@@ -38,6 +38,7 @@struct drm_pagemap_shrinker ;struct intel_display ;struct intel_dg_nvm_dev ; +struct xe_cpu_bind ;struct xe_ggtt ;struct xe_i2c ;struct xe_pat_ops ;
@@ -559,6 +560,9 @@ struct xe_device {/** @sc: System Controller */ struct xe_sysctrl sc; +/** @cpu_bind: CPU bind object */ +struct xe_cpu_bind *cpu_bind; +/** @atomic_svm_timeslice_ms: Atomic SVM fault timeslice MS */ u32 atomic_svm_timeslice_ms;
drivers/gpu/drm/xe/xe_exec_queue.c +2 -1
@@ -14,6 +14,7 @@#include <uapi/drm/xe_drm.h> #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_dep_scheduler.h" #include "xe_device.h" #include "xe_gt.h"
@@ -1666,7 +1667,7 @@ static void xe_exec_queue_last_fence_lockdep_assert(struct xe_exec_queue *q, struct xe_vm *vm) {if (q->flags &EXEC_QUEUE_FLAG_MIGRATE ) { -xe_migrate_job_lock_assert (q); +xe_cpu_bind_job_lock_assert (q); }else if (q->flags &EXEC_QUEUE_FLAG_VM ) {lockdep_assert_held (&vm->lock); }else {
drivers/gpu/drm/xe/xe_guc_submit.c +32 -9
72 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.c +0 -246
294 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.h +0 -94
114 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.c +278 -271
895 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.h +4 -4
@@ -39,11 +39,11 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred);void xe_pt_clear (struct xe_device *xe,struct xe_pt *pt); -int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops); -struct dma_fence *xe_pt_update_ops_run (struct xe_tile *tile, +int xe_pt_update_ops_prepare (struct xe_device *xe,struct xe_vma_ops *vops); +struct dma_fence *xe_pt_update_ops_run (struct xe_device *xe,struct xe_vma_ops *vops); -void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops); -void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops); +void xe_pt_update_ops_fini (struct xe_device *xe,struct xe_vma_ops *vops); +void xe_pt_update_ops_abort (struct xe_device *xe,struct xe_vma_ops *vops);bool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm,
drivers/gpu/drm/xe/xe_pt_types.h +0 -14
@@ -120,8 +120,6 @@ struct xe_pt_job_ops {struct xe_vm_pgtable_update_ops {/** @pt_job_ops: PT update operations dynamic allocation*/ struct xe_pt_job_ops *pt_job_ops; -/** @q: exec queue for PT operations */ -struct xe_exec_queue *q;/** @prl: embedded page reclaim list */ struct xe_page_reclaim_list prl;/** @start: start address of ops */
@@ -134,18 +132,6 @@ struct xe_vm_pgtable_update_ops {bool needs_svm_lock;/** @needs_invalidation: Needs invalidation */ bool needs_invalidation; -/** -* @wait_vm_bookkeep: PT operations need to wait until VM is idle -* (bookkeep dma-resv slots are idle) and stage all future VM activity -* behind these operations (install PT operations into VM kernel -* dma-resv slot). -*/ -bool wait_vm_bookkeep; -/** -* @wait_vm_kernel: PT operations need to wait until VM kernel dma-resv -* slots are idle. -*/ - bool wait_vm_kernel; };#endif
drivers/gpu/drm/xe/xe_sched_job.c +6 -4
@@ -73,8 +73,9 @@ static void job_free(struct xe_sched_job *job)struct xe_exec_queue *q = job->q;bool is_migration =xe_sched_job_is_migration (q); -kmem_cache_free (xe_exec_queue_is_parallel (job->q) || is_migration ? - xe_sched_job_parallel_slab : xe_sched_job_slab, job); +kmem_cache_free (job->is_pt_job ||xe_exec_queue_is_parallel (job->q) || + is_migration ? xe_sched_job_parallel_slab : + xe_sched_job_slab, job); }static struct xe_device *job_to_xe (struct xe_sched_job *job)
@@ -127,10 +128,12 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,xe_assert (xe, batch_addr || q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); - job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration); + job =job_alloc (!batch_addr ||xe_exec_queue_is_parallel (q) || + is_migration);if (!job)return ERR_PTR (-ENOMEM ); + job->is_pt_job = !batch_addr; job->q = q; job->sample_timestamp =U64_MAX ;kref_init (&job->refcount);
@@ -143,7 +146,6 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,if (!batch_addr) { job->fence =dma_fence_get_stub (); - job->is_pt_job = true; }else {for (i =0 ; i < q->width; ++i) {struct dma_fence *fence =xe_lrc_alloc_seqno_fence ();
drivers/gpu/drm/xe/xe_sched_job_types.h +5 -6
@@ -14,7 +14,7 @@ struct dma_fence;struct dma_fence_chain ;struct xe_exec_queue ; -struct xe_migrate_pt_update_ops ; +struct xe_cpu_bind_pt_update_ops ;struct xe_pt_job_ops ;struct xe_tile ;struct xe_vm ;
@@ -25,12 +25,11 @@ struct xe_vm;struct xe_pt_update_args {/** @vm: VM which is being bound */ struct xe_vm *vm; -/** @tile: Tile which page tables belong to */ -struct xe_tile *tile; -/** @ops: Migrate PT update ops */ -const struct xe_migrate_pt_update_ops *ops; +/** @ops: CPU bind PT update ops */ +const struct xe_cpu_bind_pt_update_ops *ops; +#define XE_PT_UPDATE_JOB_OPS_COUNT 2/** @pt_job_ops: PT job ops state */ -struct xe_pt_job_ops *pt_job_ops; +struct xe_pt_job_ops *pt_job_ops[XE_PT_UPDATE_JOB_OPS_COUNT ]; };/**
drivers/gpu/drm/xe/xe_tlb_inval_job.c +6 -7
@@ -4,6 +4,7 @@*/ #include "xe_assert.h" +#include "xe_cpu_bind.h" #include "xe_dep_job_types.h" #include "xe_dep_scheduler.h" #include "xe_exec_queue.h"
@@ -12,7 +13,6 @@#include "xe_page_reclaim.h" #include "xe_tlb_inval.h" #include "xe_tlb_inval_job.h" -#include "xe_migrate.h" #include "xe_pm.h" #include "xe_vm.h"
@@ -218,7 +218,6 @@ int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job)/** * xe_tlb_inval_job_push() - TLB invalidation job push * @job: TLB invalidation job to push -* @m: The migration object being used * @fence: Dependency for TLB invalidation job * * Pushes a TLB invalidation job for execution, using @fence as a dependency.
@@ -230,11 +229,11 @@ int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job)* Return: Job's finished fence on success, cannot fail */ struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence) {struct xe_tlb_inval_fence *ifence =container_of (job->fence,typeof (*ifence), base); +struct xe_cpu_bind *cpu_bind =gt_to_xe (job->q->gt)->cpu_bind;if (!dma_fence_is_signaled (fence)) {void *ptr;
@@ -258,11 +257,11 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, job->fence_armed = true;/* -* We need the migration lock to protect the job's seqno and the spsc -* queue, only taken on migration queue, user queues protected dma-resv +* We need the cpu_bind lock to protect the job's seqno and the spsc +* queue, only taken on cpu_bind queue, user queues protected dma-resv * VM lock. */ -xe_migrate_job_lock (m, job->q); +xe_cpu_bind_job_lock (cpu_bind, job->q);/* Creation ref pairs with put in xe_tlb_inval_job_destroy */ xe_tlb_inval_fence_init (job->tlb_inval, ifence, false);
@@ -281,7 +280,7 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, &job->dep.drm.s_fence->finished, job->idx); -xe_migrate_job_unlock (m, job->q); +xe_cpu_bind_job_unlock (cpu_bind, job->q);/* * Not using job->fence, as it has its own dma-fence context, which does
drivers/gpu/drm/xe/xe_tlb_inval_job.h +0 -2
@@ -11,7 +11,6 @@struct dma_fence ;struct xe_dep_scheduler ;struct xe_exec_queue ; -struct xe_migrate ;struct xe_page_reclaim_list ;struct xe_tlb_inval ;struct xe_tlb_inval_job ;
@@ -28,7 +27,6 @@ void xe_tlb_inval_job_add_page_reclaim(struct xe_tlb_inval_job *job,int xe_tlb_inval_job_alloc_dep (struct xe_tlb_inval_job *job);struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence);void xe_tlb_inval_job_get (struct xe_tlb_inval_job *job);
drivers/gpu/drm/xe/xe_vm.c +45 -110
288 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +9 -1
@@ -317,7 +317,7 @@ struct xe_vm {* @rftree: range fence tree to track updates to page table structure. * Used to implement conflict tracking between independent bind engines. */ -struct xe_range_fence_tree rftree[XE_MAX_TILES_PER_DEVICE ]; +struct xe_range_fence_tree rftree;const struct xe_pt_ops *pt_ops;
@@ -557,6 +557,10 @@ struct xe_vma_ops {u32 num_syncs;/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ]; +/** @start: start address of ops */ +u64 start; +/** @last: last address of ops */ +u64 last;/** @flag: signify the properties within xe_vma_ops*/ #define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0)#define XE_VMA_OPS_FLAG_MADVISE BIT(1)
@@ -566,6 +570,10 @@ struct xe_vma_ops {#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5)#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6)#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7) +#define XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP BIT(8) +#define XE_VMA_OPS_FLAG_WAIT_VM_KERNEL BIT(9) +#define XE_VMA_OPS_FLAG_NEEDS_INVALIDATION BIT(10) +#define XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK BIT(11)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 16/24] drm/xe: Add CPU bind layer
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [High] Concurrent RB-tree corruption in xe_range_fence_insert() during PREFETCH operations.
- [High] Memory leak of pt_job_ops and deferred page table BOs when a PT job is dropped by the DRM scheduler. --
commit dab33f2788d31662c4784993dd54e7509cc455e6
Author: Matthew Brost <[email protected]>
drm/xe: Add CPU bind layer
This commit introduces a dedicated CPU bind layer at the device level, decoupling it from the migrate layer. It updates the PT layer to generate a single bind job when pages are mirrored across tiles. The patch includes a significant refactor to ensure functional equivalence.
drivers/gpu/drm/xe/xe_sched_job.c static void job_free(struct xe_sched_job *job) · line 79
Quoted 13 lines
drivers/gpu/drm/xe/xe_sched_job.c +3
-2@@ -73,8 +73,9 @@ static void job_free(struct xe_sched_job *job)struct xe_exec_queue *q = job->q;bool is_migration =xe_sched_job_is_migration (q); -kmem_cache_free (xe_exec_queue_is_parallel (job->q) || is_migration ? - xe_sched_job_parallel_slab : xe_sched_job_slab, job); +kmem_cache_free (job->is_pt_job ||xe_exec_queue_is_parallel (job->q) || + is_migration ? xe_sched_job_parallel_slab : + xe_sched_job_slab, job); }
[Severity: High]
This is a pre-existing issue, but does dropping a PT job leak the pt_job_ops and its deferred page table BOs?
When a page table update job is created (e.g. in xe_cpu_bind_update_pgtables_job()), a reference to pt_job_ops is acquired via xe_pt_job_ops_get(). This is normally released when the job executes in guc_exec_queue_run_job().
However, if the execution queue is destroyed before execution, the DRM scheduler aborts the job and calls the backend's .free_job callback (guc_exec_queue_free_job()). That callback invokes job_free() which frees the job memory, but fails to call xe_pt_job_ops_put() for PT jobs. This appears to permanently leak the xe_pt_job_ops structure and the BOs it tracks.
Quoted 3 lines
drivers/gpu/drm/xe/xe_pt.c +0
-0
[ ... ]
Quoted 1 line
drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -2821,67 +2824,61 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)
[ ... ]
drivers/gpu/drm/xe/xe_pt.c xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) · line 2852
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c +2
-3-if (xe_range_fence_insert (&vm->rftree[tile->id], rfence, +if (xe_range_fence_insert (&vm->rftree, rfence, &xe_range_fence_kfree_ops, - pt_update_ops->start, - pt_update_ops->last, fence)) + vops->start, vops->last, fence))
[Severity: High]
This isn't a bug introduced by this patch, but does xe_range_fence_insert() have a concurrent RB-tree corruption risk here?
In xe_vm_bind_ioctl(), if an operation does not modify the GPU VA (such as DRM_XE_VM_BIND_OP_PREFETCH on non-userptr memory), XE_VMA_OPS_FLAG_DOWNGRADE_LOCK is set and vm->lock is downgraded to a read lock (downgrade_write(&vm->lock)).
When multiple prefetch binds on the same VM execute concurrently, they will reach xe_pt_update_ops_run() and unconditionally call xe_range_fence_insert() to track the page table update. Since xe_range_fence_insert() modifies the RB-tree directly without any internal synchronization, this concurrent access under a read lock could lead to structural corruption of the shared vm->rftree.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
Re: [PATCH v6 16/24] drm/xe: Add CPU bind layer
In reply to Matthew BrostQuoted 378 lines · Matthew Brost
With CPU binds, it no longer makes sense to implement CPU bind handling in the migrate layer, as these operations are entirely decoupled from hardware. Introduce a dedicated CPU bind layer stored at the device level.
Since CPU binds are tile-independent, update the PT layer to generate a single bind job even when pages are mirrored across tiles.
This patch is large because the refactor touches multiple file / layers and ensures functional equivalence before and after the change.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_cpu_bind.c | 298 +++++++++++++ drivers/gpu/drm/xe/xe_cpu_bind.h | 118 +++++ drivers/gpu/drm/xe/xe_device.c | 5 + drivers/gpu/drm/xe/xe_device_types.h | 4 + drivers/gpu/drm/xe/xe_exec_queue.c | 3 +- drivers/gpu/drm/xe/xe_guc_submit.c | 41 +- drivers/gpu/drm/xe/xe_migrate.c | 246 ----------- drivers/gpu/drm/xe/xe_migrate.h | 94 ---- drivers/gpu/drm/xe/xe_pt.c | 549 ++++++++++++------------ drivers/gpu/drm/xe/xe_pt.h | 8 +- drivers/gpu/drm/xe/xe_pt_types.h | 14 - drivers/gpu/drm/xe/xe_sched_job.c | 10 +- drivers/gpu/drm/xe/xe_sched_job_types.h | 11 +- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 13 +- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 2 - drivers/gpu/drm/xe/xe_vm.c | 155 ++----- drivers/gpu/drm/xe/xe_vm_types.h | 10 +- 18 files changed, 813 insertions(+), 769 deletions(-)create mode 100644
drivers/gpu/drm/xe/xe_cpu_bind.ccreate mode 100644drivers/gpu/drm/xe/xe_cpu_bind.hdrivers/gpu/drm/xe/Makefile +1
-0@@ -35,6 +35,7 @@ $(obj)/generated/%_device_wa_oob.c $(obj)/generated/%_device_wa_oob.h: $(obj)/xe xe-y += xe_bb.o \ xe_bo.o \ xe_bo_evict.o \ + xe_cpu_bind.o \ xe_dep_scheduler.o \ xe_devcoredump.o \ xe_device.o \drivers/gpu/drm/xe/xe_cpu_bind.c new file +298
-0@@ -0,0 +1,298 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include <drm/drm_managed.h> +#include <linux/mutex.h> + +#include "xe_cpu_bind.h" +#include "xe_device_types.h" +#include "xe_exec_queue.h" +#include "xe_pt.h" +#include "xe_sched_job.h" +#include "xe_trace_bo.h" +#include "xe_vm.h" + +/** +* struct xe_cpu_bind - cpu_bind context. +*/ +struct xe_cpu_bind { +/** @xe: Xe device */ +struct xe_device *xe; +/** @q: Default exec queue used for kernel binds */ +struct xe_exec_queue *q; +/** @job_mutex: Timeline mutex for @q. */ +struct mutex job_mutex; +}; + +static bool is_cpu_bind_queue (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +return cpu_bind->q == q; +} + +static void xe_cpu_bind_fini (void *arg) +{ +struct xe_cpu_bind *cpu_bind = arg; + +mutex_destroy (&cpu_bind->job_mutex); +xe_exec_queue_put (cpu_bind->q); +} + +/** +* xe_cpu_bind_init() - Initialize a cpu_bind context +* @xe: &xe_device +* +* Return: 0 if successful, negative error code on failure +*/ +int xe_cpu_bind_init (struct xe_device *xe) +{ +struct xe_cpu_bind *cpu_bind = +drmm_kzalloc (&xe->drm,sizeof (*cpu_bind),GFP_KERNEL ); +struct xe_exec_queue *q; + +if (!cpu_bind) +return -ENOMEM ; + + q =xe_exec_queue_create_bind (xe,xe_device_get_root_tile (xe),NULL , +EXEC_QUEUE_FLAG_KERNEL | +EXEC_QUEUE_FLAG_MIGRATE ,0 ); +if (IS_ERR (q)) +return PTR_ERR (q); + + cpu_bind->xe = xe; + cpu_bind->q = q; + xe->cpu_bind = cpu_bind; + +mutex_init (&cpu_bind->job_mutex); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&cpu_bind->job_mutex); +fs_reclaim_release (GFP_KERNEL ); + +return devm_add_action_or_reset (cpu_bind->xe->drm.dev, xe_cpu_bind_fini, + cpu_bind); +} + +/** +* xe_cpu_bind_queue() - Get the bind queue from cpu_bind context. +* @cpu_bind: The cpu bind context. +* +* Return: Pointer to bind queue on success, error on failure +*/ +struct xe_exec_queue *xe_cpu_bind_queue (struct xe_cpu_bind *cpu_bind) +{ +return cpu_bind->q; +} + +/** +* xe_cpu_bind_update_pgtables_execute() - Update a VM's PTEs via the CPU +* @vm: The VM being updated +* @tile: The tile being updated +* @ops: The migrate PT update ops +* @pt_op: The VM PT update op +* @num_ops: The number of The VM PT update ops +* +* Execute the VM PT update ops array which results in a VM's PTEs being updated +* via the CPU. +*/ +void +xe_cpu_bind_update_pgtables_execute (struct xe_vm *vm,struct xe_tile *tile, +const struct xe_cpu_bind_pt_update_ops *ops, +struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops) +{ +u32 j, i; + +for (j =0 ; j < num_ops; ++j, ++pt_op) { +for (i =0 ; i < pt_op->num_entries; i++) { +const struct xe_vm_pgtable_update *update = + &pt_op->entries[i]; + +xe_assert (vm->xe, update); +xe_assert (vm->xe, update->pt_bo); +xe_assert (vm->xe, !iosys_map_is_null (&update->pt_bo->vmap)); + +if (pt_op->bind) + ops->populate (tile, &update->pt_bo->vmap, + update->ofs, update->qwords, + update); +else + ops->clear (vm, tile, &update->pt_bo->vmap, + update->ofs, update->qwords, + update); + } + } + +trace_xe_vm_cpu_bind (vm); +xe_device_wmb (vm->xe); +} + +static struct dma_fence * +xe_cpu_bind_update_pgtables_no_job (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +const struct xe_cpu_bind_pt_update_ops *ops = pt_update->ops; +struct xe_vm *vm = pt_update->vops->vm; +struct xe_tile *tile; +int err, id; + +if (ops->pre_commit) { + pt_update->job =NULL ; + err = ops->pre_commit (pt_update); +if (err) +return ERR_PTR (err); + } + +for_each_tile (tile, vm->xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &pt_update->vops->pt_update_ops[tile->id]; + +if (!pt_update_ops->pt_job_ops) +continue ; + +xe_cpu_bind_update_pgtables_execute (vm, tile, ops, + pt_update_ops->pt_job_ops->ops, + pt_update_ops->pt_job_ops->current_op); + } + +return dma_fence_get_stub (); +} + +static struct dma_fence * +xe_cpu_bind_update_pgtables_job (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +const struct xe_cpu_bind_pt_update_ops *ops = pt_update->ops; +struct xe_exec_queue *q = pt_update->vops->q; +struct xe_device *xe = cpu_bind->xe; +struct xe_sched_job *job; +struct dma_fence *fence; +struct xe_tile *tile; +int err, id; +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + + job =xe_sched_job_create (q,NULL ); +if (IS_ERR (job)) +return ERR_CAST (job); + +xe_assert (xe, job->is_pt_job); + +if (ops->pre_commit) { + pt_update->job = job; + err = ops->pre_commit (pt_update); +if (err) + goto err_job; + } + +if (is_cpu_bind) +mutex_lock (&cpu_bind->job_mutex); + + job->pt_update[0 ].vm = pt_update->vops->vm; + job->pt_update[0 ].ops = ops; +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &pt_update->vops->pt_update_ops[tile->id]; + + job->pt_update[0 ].pt_job_ops[tile->id] = +xe_pt_job_ops_get (pt_update_ops->pt_job_ops); + } + +xe_sched_job_arm (job); + fence =dma_fence_get (&job->drm.s_fence->finished); +xe_sched_job_push (job); + +if (is_cpu_bind) +mutex_unlock (&cpu_bind->job_mutex); + +return fence; + +err_job: +xe_sched_job_put (job); +return ERR_PTR (err); +} + +/** +* xe_cpu_bind_update_pgtables() - Pipelined page-table update +* @cpu_bind: The cpu bind context. +* @pt_update: PT update arguments +* +* Perform a pipelined page-table update. The update descriptors are typically +* built under the same lock critical section as a call to this function. If +* using the default engine for the updates, they will be performed in the +* order they grab the job_mutex. If different engines are used, external +* synchronization is needed for overlapping updates to maintain page-table +* consistency. Note that the meaning of "overlapping" is that the updates +* touch the same page-table, which might be a higher-level page-directory. +* If no pipelining is needed, then updates may be performed by the cpu. +* +* Return: A dma_fence that, when signaled, indicates the update completion. +*/ +struct dma_fence * +xe_cpu_bind_update_pgtables (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +struct dma_fence *fence; + + fence =xe_cpu_bind_update_pgtables_no_job (cpu_bind, pt_update); + +/* -ETIME indicates a job is needed, anything else is legit error */ +if (!IS_ERR (fence) ||PTR_ERR (fence) != -ETIME ) +return fence; + +return xe_cpu_bind_update_pgtables_job (cpu_bind, pt_update); +} + +/** +* xe_cpu_bind_job_lock() - Lock cpu_bind job lock +* @cpu_bind: The cpu bind context. +* @q: Queue associated with the operation which requires a lock +* +* Lock the cpu_bind job lock if the queue is a cpu bind queue, otherwise +* assert the VM's dma-resv is held (user queue's have own locking). +*/ +void xe_cpu_bind_job_lock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + +if (is_cpu_bind) +mutex_lock (&cpu_bind->job_mutex); +else +xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */ +} + +/** +* xe_cpu_bind_job_unlock() - Unlock cpu_bind job lock +* @cpu_bind: The cpu bind context. +* @q: Queue associated with the operation which requires a lock +* +* Unlock the cpu_bind job lock if the queue is a cpu bind queue, otherwise +* assert the VM's dma-resv is held (user queue's have own locking). +*/ +void xe_cpu_bind_job_unlock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + +if (is_cpu_bind) +mutex_unlock (&cpu_bind->job_mutex); +else +xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */ +} + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* xe_cpu_bind_job_lock_assert() - Assert cpu_bind job lock held of queue +* @q: cpu bind queue +*/ +void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q) +{ +struct xe_device *xe =gt_to_xe (q->gt); +struct xe_cpu_bind *cpu_bind = xe->cpu_bind; + +xe_assert (xe, q == cpu_bind->q); +lockdep_assert_held (&cpu_bind->job_mutex); +} +#endif drivers/gpu/drm/xe/xe_cpu_bind.h new file +19
-0@@ -0,0 +1,118 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _XE_CPU_BIND_H_ +#define _XE_CPU_BIND_H_ + +#include <linux/types.h> + +struct dma_fence ; +struct iosys_map ; +struct xe_cpu_bind ; +struct xe_cpu_bind_pt_update ; +struct xe_device ; +struct xe_tlb_inval_job ; +struct xe_tile ; +struct xe_vm ; +struct xe_vm_pgtable_update ;
drivers/gpu/drm/xe/xe_cpu_bind.h line 29
Quoted 10 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +10
-0+struct xe_vm_pgtable_update_op ; +struct xe_vma_ops ; + +/** +* struct xe_cpu_bind_pt_update_ops - Callbacks for the +* xe_cpu_bind_update_pgtables() function. +*/ +struct xe_cpu_bind_pt_update_ops { +/** +* @populate: Populate a command buffer or page-table with ptes.
with gpu copy removed doc is giving wrong info "command buffer"
Quoted 1,700 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +89
-0+* @tile: The tile for the current operation. +* @map: struct iosys_map into the memory to be populated. +* @ofs: qword offset into @map, unused if @map is NULL. +* @num_qwords: Number of qwords to write. +* @update: Information about the PTEs to be inserted. +* +* This interface is intended to be used as a callback into the +* page-table system to populate command buffers or shared +* page-tables with PTEs. +*/ +void (*populate)(struct xe_tile *tile,struct iosys_map *map, +u32 ofs,u32 num_qwords, +const struct xe_vm_pgtable_update *update); +/** +* @clear: Clear a command buffer or page-table with ptes. +* @vm: VM being updated +* @tile: The tile for the current operation. +* @map: struct iosys_map into the memory to be populated. +* @ofs: qword offset into @map, unused if @map is NULL. +* @num_qwords: Number of qwords to write. +* @update: Information about the PTEs to be inserted. +* +* This interface is intended to be used as a callback into the +* page-table system to populate command buffers or shared +* page-tables with PTEs. +*/ +void (*clear)(struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,u32 ofs,u32 num_qwords, +const struct xe_vm_pgtable_update *update); + +/** +* @pre_commit: Callback to be called just before arming the +* sched_job. +* @pt_update: Pointer to embeddable callback argument. +* +* Return: 0 on success, negative error code on error. +*/ +int (*pre_commit)(struct xe_cpu_bind_pt_update *pt_update); +}; + +/** +* struct xe_cpu_bind_pt_update - Argument to the struct +* xe_cpu_bind_pt_update_ops callbacks. +* +* Intended to be subclassed to support additional arguments if necessary. +*/ +struct xe_cpu_bind_pt_update { +/** @ops: Pointer to the struct xe_cpu_bind_pt_update_ops callbacks */ +const struct xe_cpu_bind_pt_update_ops *ops; +/** @vops: VMA operations */ +struct xe_vma_ops *vops; +/** @job: The job if a GPU page-table update. NULL otherwise */ +struct xe_sched_job *job; +/** +* @ijobs: The TLB invalidation jobs, individual instances can be NULL +*/ +#define XE_CPU_BIND_INVAL_JOB_COUNT 4 +struct xe_tlb_inval_job *ijobs[XE_CPU_BIND_INVAL_JOB_COUNT ]; +}; + +int xe_cpu_bind_init (struct xe_device *xe); + +struct xe_exec_queue *xe_cpu_bind_queue (struct xe_cpu_bind *cpu_bind); + +void +xe_cpu_bind_update_pgtables_execute (struct xe_vm *vm,struct xe_tile *tile, +const struct xe_cpu_bind_pt_update_ops *ops, +struct xe_vm_pgtable_update_op *pt_op, +u32 num_ops); + +struct dma_fence * +xe_cpu_bind_update_pgtables (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update); + +void xe_cpu_bind_job_lock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q); + +void xe_cpu_bind_job_unlock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q); +#else +static inline void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q) +{ +} +#endif + +#endif drivers/gpu/drm/xe/xe_device.c +5
-0@@ -27,6 +27,7 @@#include "xe_bo_evict.h" #include "xe_configfs.h" #include "xe_debugfs.h" +#include "xe_cpu_bind.h" #include "xe_defaults.h" #include "xe_devcoredump.h" #include "xe_device_sysfs.h" @@ -1075,6 +1076,10 @@ int xe_device_probe(struct xe_device *xe)if (err)return err; + err =xe_cpu_bind_init (xe); +if (err) +return err; + err =xe_pagefault_init (xe);if (err)return err;drivers/gpu/drm/xe/xe_device_types.h +4
-0@@ -38,6 +38,7 @@struct drm_pagemap_shrinker ;struct intel_display ;struct intel_dg_nvm_dev ; +struct xe_cpu_bind ;struct xe_ggtt ;struct xe_i2c ;struct xe_pat_ops ;@@ -559,6 +560,9 @@ struct xe_device {/** @sc: System Controller */ struct xe_sysctrl sc; +/** @cpu_bind: CPU bind object */ +struct xe_cpu_bind *cpu_bind; +/** @atomic_svm_timeslice_ms: Atomic SVM fault timeslice MS */ u32 atomic_svm_timeslice_ms;drivers/gpu/drm/xe/xe_exec_queue.c +2
-1@@ -14,6 +14,7 @@#include <uapi/drm/xe_drm.h> #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_dep_scheduler.h" #include "xe_device.h" #include "xe_gt.h" @@ -1666,7 +1667,7 @@ static void xe_exec_queue_last_fence_lockdep_assert(struct xe_exec_queue *q, struct xe_vm *vm) {if (q->flags &EXEC_QUEUE_FLAG_MIGRATE ) { -xe_migrate_job_lock_assert (q); +xe_cpu_bind_job_lock_assert (q); }else if (q->flags &EXEC_QUEUE_FLAG_VM ) {lockdep_assert_held (&vm->lock); }else {drivers/gpu/drm/xe/xe_guc_submit.c +32
-9@@ -19,6 +19,7 @@#include "abi/guc_klvs_abi.h" #include "xe_assert.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_devcoredump.h" #include "xe_device.h" #include "xe_exec_queue.h" @@ -38,7 +39,6 @@#include "xe_lrc.h" #include "xe_macros.h" #include "xe_map.h" -#include "xe_migrate.h" #include "xe_mocs.h" #include "xe_module.h" #include "xe_pm.h" @@ -1244,13 +1244,36 @@ static bool is_pt_job(struct xe_sched_job *job)return job->is_pt_job; } -static void run_pt_job (struct xe_sched_job *job) +static void run_pt_job (struct xe_device *xe,struct xe_sched_job *job) { -xe_migrate_update_pgtables_cpu_execute (job->pt_update[0 ].vm, - job->pt_update[0 ].tile, - job->pt_update[0 ].ops, - job->pt_update[0 ].pt_job_ops->ops, - job->pt_update[0 ].pt_job_ops->current_op); +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_pt_job_ops *pt_job_ops = + job->pt_update[0 ].pt_job_ops[id]; + +if (!pt_job_ops || !pt_job_ops->current_op) +continue ; + +xe_cpu_bind_update_pgtables_execute (job->pt_update[0 ].vm, tile, + job->pt_update[0 ].ops, + pt_job_ops->ops, + pt_job_ops->current_op); + } +} + +static void put_pt_job (struct xe_device *xe,struct xe_sched_job *job) +{ +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_pt_job_ops *pt_job_ops = + job->pt_update[0 ].pt_job_ops[id]; + +xe_pt_job_ops_put (pt_job_ops); + } } static struct dma_fence *@@ -1280,7 +1303,7 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)if (is_pt_job (job)) {xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); -run_pt_job (job); +run_pt_job (guc_to_xe (guc), job); }else {if (!exec_queue_registered (q))register_exec_queue (q,GUC_CONTEXT_NORMAL );@@ -1292,7 +1315,7 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job) }if (is_pt_job (job)) { -xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +put_pt_job (guc_to_xe (guc), job);dma_fence_put (job->fence);/* Drop ref from xe_sched_job_arm */ return NULL ; }drivers/gpu/drm/xe/xe_migrate.c +0
-246@@ -51,8 +51,6 @@struct xe_migrate {/** @q: Default exec queue used for migration */ struct xe_exec_queue *q; -/** @bind_q: Default exec queue used for binds */ -struct xe_exec_queue *bind_q;/** @tile: Backpointer to the tile this struct xe_migrate belongs to. */ struct xe_tile *tile;/** @job_mutex: Timeline mutex for @eng. */ @@ -110,7 +108,6 @@ static void xe_migrate_fini(void *arg)mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q); -xe_exec_queue_put (m->bind_q); } static inline u16 xe_migrate_pat_index(struct xe_device *xe,@@ -483,15 +480,6 @@ int xe_migrate_init(struct xe_migrate *m) goto err_out; } - m->bind_q =xe_exec_queue_create (xe, vm, logical_mask,1 , hwe0, -EXEC_QUEUE_FLAG_KERNEL | -EXEC_QUEUE_FLAG_HIGH_PRIORITY | -EXEC_QUEUE_FLAG_MIGRATE ,0 ); -if (IS_ERR (m->bind_q)) { - err =PTR_ERR (m->bind_q); - goto err_out; - } -/* * XXX: Currently only reserving 1 (likely slow) BCS instance on * PVC, may want to revisit if performance is needed. @@ -502,15 +490,6 @@ int xe_migrate_init(struct xe_migrate *m)EXEC_QUEUE_FLAG_MIGRATE | EXEC_QUEUE_FLAG_LOW_LATENCY, 0); }else { - m->bind_q =xe_exec_queue_create_class (xe, primary_gt, vm, -XE_ENGINE_CLASS_COPY , -EXEC_QUEUE_FLAG_KERNEL | -EXEC_QUEUE_FLAG_MIGRATE ,0 ); -if (IS_ERR (m->bind_q)) { - err =PTR_ERR (m->bind_q); - goto err_out; - } - m->q = xe_exec_queue_create_class(xe, primary_gt, vm, XE_ENGINE_CLASS_COPY, EXEC_QUEUE_FLAG_KERNEL |@@ -546,8 +525,6 @@ int xe_migrate_init(struct xe_migrate *m)return err; err_out: -if (!IS_ERR_OR_NULL (m->bind_q)) -xe_exec_queue_put (m->bind_q);xe_vm_close_and_put (vm);return err;@@ -1507,17 +1484,6 @@ static u32 blt_mem_set_cmd_len(struct xe_device *xe)return 7 ; } -/** -* xe_get_migrate_bind_queue() - Get the bind queue from migrate context. -* @migrate: Migrate context. -* -* Return: Pointer to bind queue on success, error on failure -*/ -struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate) -{ -return migrate->bind_q; -} -static void emit_clear_link_copy (struct xe_gt *gt,struct xe_bb *bb,u64 src_ofs,u32 size,u32 pitch) {@@ -1782,168 +1748,6 @@ struct migrate_test_params { container_of(_priv, struct migrate_test_params, base)#endif -/** -* xe_migrate_update_pgtables_cpu_execute() - Update a VM's PTEs via the CPU -* @vm: The VM being updated -* @tile: The tile being updated -* @ops: The migrate PT update ops -* @pt_ops: The VM PT update ops -* @num_ops: The number of The VM PT update ops -* -* Execute the VM PT update ops array which results in a VM's PTEs being updated -* via the CPU. -*/ -void -xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, -const struct xe_migrate_pt_update_ops *ops, -struct xe_vm_pgtable_update_op *pt_op, -u32 num_ops) -{ -u32 j, i; - -for (j =0 ; j < num_ops; ++j, ++pt_op) { -for (i =0 ; i < pt_op->num_entries; i++) { -const struct xe_vm_pgtable_update *update = - &pt_op->entries[i]; - -xe_tile_assert (tile, update); -xe_tile_assert (tile, update->pt_bo); -xe_tile_assert (tile, !iosys_map_is_null (&update->pt_bo->vmap)); - -if (pt_op->bind) - ops->populate (tile, &update->pt_bo->vmap, - update->ofs, update->qwords, - update); -else - ops->clear (vm, tile, &update->pt_bo->vmap, - update->ofs, update->qwords, - update); - } - } - -trace_xe_vm_cpu_bind (vm); -xe_device_wmb (vm->xe); -} - -static struct dma_fence * -xe_migrate_update_pgtables_cpu (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update) -{ - XE_TEST_DECLARE(struct migrate_test_params *test = -to_migrate_test_params - (xe_cur_kunit_priv(XE_TEST_LIVE_MIGRATE));) -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_vm *vm = pt_update->vops->vm; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &pt_update->vops->pt_update_ops[pt_update->tile_id]; -int err; - -if (XE_TEST_ONLY (test && test->force_gpu)) -return ERR_PTR (-ETIME ); - -if (ops->pre_commit) { - pt_update->job =NULL ; - err = ops->pre_commit (pt_update); -if (err) -return ERR_PTR (err); - } - -xe_migrate_update_pgtables_cpu_execute (vm, m->tile, ops, - pt_update_ops->pt_job_ops->ops, - pt_update_ops->num_ops); - -return dma_fence_get_stub (); -} - -static bool is_migrate_queue (struct xe_migrate *m,struct xe_exec_queue *q) -{ -return m->bind_q == q; -} - -static struct dma_fence * -__xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update, -struct xe_vm_pgtable_update_ops *pt_update_ops) -{ -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_tile *tile = m->tile; -struct xe_sched_job *job; -struct dma_fence *fence; -bool is_migrate =is_migrate_queue (m, pt_update_ops->q); -int err; - - job =xe_sched_job_create (pt_update_ops->q,NULL ); -if (IS_ERR (job)) { - err =PTR_ERR (job); - goto err_out; - } - -xe_tile_assert (tile, job->is_pt_job); - -if (ops->pre_commit) { - pt_update->job = job; - err = ops->pre_commit (pt_update); -if (err) - goto err_job; - } -if (is_migrate) -mutex_lock (&m->job_mutex); - - job->pt_update[0 ].vm = pt_update->vops->vm; - job->pt_update[0 ].tile = tile; - job->pt_update[0 ].ops = ops; - job->pt_update[0 ].pt_job_ops = -xe_pt_job_ops_get (pt_update_ops->pt_job_ops); - -xe_sched_job_arm (job); - fence =dma_fence_get (&job->drm.s_fence->finished); -xe_sched_job_push (job); - -if (is_migrate) -mutex_unlock (&m->job_mutex); - -return fence; - -err_job: -xe_sched_job_put (job); -err_out: -return ERR_PTR (err); -} - -/** -* xe_migrate_update_pgtables() - Pipelined page-table update -* @m: The migrate context. -* @pt_update: PT update arguments -* -* Perform a pipelined page-table update. The update descriptors are typically -* built under the same lock critical section as a call to this function. If -* using the default engine for the updates, they will be performed in the -* order they grab the job_mutex. If different engines are used, external -* synchronization is needed for overlapping updates to maintain page-table -* consistency. Note that the meaning of "overlapping" is that the updates -* touch the same page-table, which might be a higher-level page-directory. -* If no pipelining is needed, then updates may be performed by the cpu. -* -* Return: A dma_fence that, when signaled, indicates the update completion. -*/ -struct dma_fence * -xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update) - -{ -struct xe_vm_pgtable_update_ops *pt_update_ops = - &pt_update->vops->pt_update_ops[pt_update->tile_id]; -struct dma_fence *fence; - - fence =xe_migrate_update_pgtables_cpu (m, pt_update); - -/* -ETIME indicates a job is needed, anything else is legit error */ -if (!IS_ERR (fence) ||PTR_ERR (fence) != -ETIME ) -return fence; - -return __xe_migrate_update_pgtables (m, pt_update, pt_update_ops); -} -/** * xe_migrate_wait() - Complete all operations using the xe_migrate context * @m: Migrate context to wait for. @@ -2445,56 +2249,6 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo,return IS_ERR (fence) ?PTR_ERR (fence) :0 ; } -/** -* xe_migrate_job_lock() - Lock migrate job lock -* @m: The migration context. -* @q: Queue associated with the operation which requires a lock -* -* Lock the migrate job lock if the queue is a migration queue, otherwise -* assert the VM's dma-resv is held (user queue's have own locking). -*/ -void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q) -{ -bool is_migrate =is_migrate_queue (m, q); - -if (is_migrate) -mutex_lock (&m->job_mutex); -else -xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */ -} - -/** -* xe_migrate_job_unlock() - Unlock migrate job lock -* @m: The migration context. -* @q: Queue associated with the operation which requires a lock -* -* Unlock the migrate job lock if the queue is a migration queue, otherwise -* assert the VM's dma-resv is held (user queue's have own locking). -*/ -void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q) -{ -bool is_migrate =is_migrate_queue (m, q); - -if (is_migrate) -mutex_unlock (&m->job_mutex); -else -xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */ -} - -#if IS_ENABLED (CONFIG_PROVE_LOCKING ) -/** -* xe_migrate_job_lock_assert() - Assert migrate job lock held of queue -* @q: Migrate queue -*/ -void xe_migrate_job_lock_assert (struct xe_exec_queue *q) -{ -struct xe_migrate *m =gt_to_tile (q->gt)->migrate; - -xe_gt_assert (q->gt, q == m->bind_q); -lockdep_assert_held (&m->job_mutex); -} -#endif -#if IS_ENABLED (CONFIG_DRM_XE_KUNIT_TEST )#include "tests/xe_migrate.c" #endif drivers/gpu/drm/xe/xe_migrate.h +0
-94@@ -34,78 +34,6 @@ enum xe_migrate_copy_dir { XE_MIGRATE_COPY_TO_SRAM, }; -/** -* struct xe_migrate_pt_update_ops - Callbacks for the -* xe_migrate_update_pgtables() function. -*/ -struct xe_migrate_pt_update_ops { -/** -* @populate: Populate a command buffer or page-table with ptes. -* @tile: The tile for the current operation. -* @map: struct iosys_map into the memory to be populated. -* @ofs: qword offset into @map, unused if @map is NULL. -* @num_qwords: Number of qwords to write. -* @update: Information about the PTEs to be inserted. -* -* This interface is intended to be used as a callback into the -* page-table system to populate command buffers or shared -* page-tables with PTEs. -*/ -void (*populate)(struct xe_tile *tile,struct iosys_map *map, -u32 ofs,u32 num_qwords, -const struct xe_vm_pgtable_update *update); -/** -* @clear: Clear a command buffer or page-table with ptes. -* @vm: VM being updated -* @tile: The tile for the current operation. -* @map: struct iosys_map into the memory to be populated. -* @ofs: qword offset into @map, unused if @map is NULL. -* @num_qwords: Number of qwords to write. -* @update: Information about the PTEs to be inserted. -* -* This interface is intended to be used as a callback into the -* page-table system to populate command buffers or shared -* page-tables with PTEs. -*/ -void (*clear)(struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,u32 ofs,u32 num_qwords, -const struct xe_vm_pgtable_update *update); - -/** -* @pre_commit: Callback to be called just before arming the -* sched_job. -* @pt_update: Pointer to embeddable callback argument. -* -* Return: 0 on success, negative error code on error. -*/ -int (*pre_commit)(struct xe_migrate_pt_update *pt_update); -}; - -/** -* struct xe_migrate_pt_update - Argument to the -* struct xe_migrate_pt_update_ops callbacks. -* -* Intended to be subclassed to support additional arguments if necessary. -*/ -struct xe_migrate_pt_update { -/** @ops: Pointer to the struct xe_migrate_pt_update_ops callbacks */ -const struct xe_migrate_pt_update_ops *ops; -/** @vops: VMA operations */ -struct xe_vma_ops *vops; -/** @job: The job if a GPU page-table update. NULL otherwise */ -struct xe_sched_job *job; -/** -* @ijob: The TLB invalidation job for primary GT. NULL otherwise -*/ -struct xe_tlb_inval_job *ijob; -/** -* @mjob: The TLB invalidation job for media GT. NULL otherwise -*/ -struct xe_tlb_inval_job *mjob; -/** @tile_id: Tile ID of the update */ -u8 tile_id; -}; -struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m);@@ -143,7 +71,6 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo,struct xe_lrc *xe_migrate_lrc (struct xe_migrate *migrate);struct xe_exec_queue *xe_migrate_exec_queue (struct xe_migrate *migrate); -struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate);struct dma_fence *xe_migrate_vram_copy_chunk (struct xe_bo *vram_bo,u64 vram_offset,struct xe_bo *sysmem_bo,u64 sysmem_offset,u64 size,enum xe_migrate_copy_dir dir);@@ -162,29 +89,8 @@ struct dma_fence *xe_migrate_clear(struct xe_migrate *m,struct xe_vm *xe_migrate_get_vm (struct xe_migrate *m); -void -xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, -const struct xe_migrate_pt_update_ops *ops, -struct xe_vm_pgtable_update_op *pt_op, -u32 num_ops); - -struct dma_fence * -xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update); -void xe_migrate_wait (struct xe_migrate *m); -#if IS_ENABLED (CONFIG_PROVE_LOCKING ) -void xe_migrate_job_lock_assert (struct xe_exec_queue *q); -#else -static inline void xe_migrate_job_lock_assert (struct xe_exec_queue *q) -{ -} -#endif - -void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q); -void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q); -#if IS_ENABLED (CONFIG_DRM_XE_DEBUG_MEM ) int xe_migrate_debug_ccs_overlap(struct xe_migrate *m, struct xe_bo *scratch_bo,drivers/gpu/drm/xe/xe_pt.c +278
-271@@ -7,12 +7,12 @@#include "regs/xe_gtt_defs.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_device.h" #include "xe_drm_client.h" #include "xe_exec_queue.h" #include "xe_gt.h" #include "xe_gt_stats.h" -#include "xe_migrate.h" #include "xe_page_reclaim.h" #include "xe_pat.h" #include "xe_pt_types.h" @@ -1401,11 +1401,9 @@ static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op, }static int xe_pt_vm_dependencies (struct xe_sched_job *job, -struct xe_tlb_inval_job *ijob, -struct xe_tlb_inval_job *mjob, +struct xe_tlb_inval_job **ijobs,struct xe_vm *vm,struct xe_vma_ops *vops, -struct xe_vm_pgtable_update_ops *pt_update_ops,struct xe_range_fence_tree *rftree) {struct xe_range_fence *rtfence;@@ -1418,20 +1416,22 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job,if (!job && !no_in_syncs (vops->syncs, vops->num_syncs))return -ETIME ; -if (!job && !xe_exec_queue_is_idle (pt_update_ops->q)) +if (!job && !xe_exec_queue_is_idle (vops->q))return -ETIME ; -if (pt_update_ops->wait_vm_bookkeep || pt_update_ops->wait_vm_kernel) { - err =job_test_add_deps (job,xe_vm_resv (vm), - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_BOOKKEEP : -DMA_RESV_USAGE_KERNEL ); +if (vops->flags & (XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP | +XE_VMA_OPS_FLAG_WAIT_VM_KERNEL )) { +enum dma_resv_usage usage =DMA_RESV_USAGE_KERNEL ; + +if (vops->flags &XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ) + usage =DMA_RESV_USAGE_BOOKKEEP ; + + err =job_test_add_deps (job,xe_vm_resv (vm), usage);if (err)return err; } - rtfence =xe_range_fence_tree_first (rftree, pt_update_ops->start, - pt_update_ops->last); + rtfence =xe_range_fence_tree_first (rftree, vops->start, vops->last);while (rtfence) { fence = rtfence->fence;@@ -1449,9 +1449,8 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job,return err; } - rtfence =xe_range_fence_tree_next (rtfence, - pt_update_ops->start, - pt_update_ops->last); + rtfence =xe_range_fence_tree_next (rtfence, vops->start, + vops->last); }list_for_each_entry (op, &vops->list, link) {@@ -1464,14 +1463,11 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job, err =xe_sync_entry_add_deps (&vops->syncs[i], job);if (job) { -if (ijob) { - err =xe_tlb_inval_job_alloc_dep (ijob); -if (err) -return err; - } +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +if (!ijobs[i]) +continue ; -if (mjob) { - err =xe_tlb_inval_job_alloc_dep (mjob); + err =xe_tlb_inval_job_alloc_dep (ijobs[i]);if (err)return err; }@@ -1480,17 +1476,14 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job,return err; } -static int xe_pt_pre_commit (struct xe_migrate_pt_update *pt_update) +static int xe_pt_pre_commit (struct xe_cpu_bind_pt_update *pt_update) {struct xe_vma_ops *vops = pt_update->vops;struct xe_vm *vm = vops->vm; -struct xe_range_fence_tree *rftree = &vm->rftree[pt_update->tile_id]; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[pt_update->tile_id]; +struct xe_range_fence_tree *rftree = &vm->rftree; -return xe_pt_vm_dependencies (pt_update->job, pt_update->ijob, - pt_update->mjob, vm, pt_update->vops, - pt_update_ops, rftree); +return xe_pt_vm_dependencies (pt_update->job, pt_update->ijobs, + vm, vops, rftree); }#if IS_ENABLED (CONFIG_DRM_GPUSVM )@@ -1550,8 +1543,7 @@ static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)#endif -static int vma_check_userptr (struct xe_vm *vm,struct xe_vma *vma, -struct xe_vm_pgtable_update_ops *pt_update) +static int vma_check_userptr (struct xe_vm *vm,struct xe_vma *vma) {struct xe_userptr_vma *uvma;unsigned long notifier_seq;@@ -1581,8 +1573,7 @@ static int vma_check_userptr(struct xe_vm *vm, struct xe_vma *vma,return 0 ; } -static int op_check_svm_userptr (struct xe_vm *vm,struct xe_vma_op *op, -struct xe_vm_pgtable_update_ops *pt_update) +static int op_check_svm_userptr (struct xe_vm *vm,struct xe_vma_op *op) {int err =0 ;@@ -1593,13 +1584,13 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op,if (!op->map.immediate &&xe_vm_in_fault_mode (vm))break ; - err =vma_check_userptr (vm, op->map.vma, pt_update); + err =vma_check_userptr (vm, op->map.vma);break ;case DRM_GPUVA_OP_REMAP :if (op->remap.prev && !op->remap.skip_prev) - err =vma_check_userptr (vm, op->remap.prev, pt_update); + err =vma_check_userptr (vm, op->remap.prev);if (!err && op->remap.next && !op->remap.skip_next) - err =vma_check_userptr (vm, op->remap.next, pt_update); + err =vma_check_userptr (vm, op->remap.next);break ;case DRM_GPUVA_OP_UNMAP :break ;@@ -1619,7 +1610,7 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op, } } }else { - err =vma_check_userptr (vm,gpuva_to_vma (op->base.prefetch.va), pt_update); + err =vma_check_userptr (vm,gpuva_to_vma (op->base.prefetch.va)); }break ;#if IS_ENABLED (CONFIG_DRM_XE_GPUSVM )@@ -1645,12 +1636,10 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op,return err; } -static int xe_pt_svm_userptr_pre_commit (struct xe_migrate_pt_update *pt_update) +static int xe_pt_svm_userptr_pre_commit (struct xe_cpu_bind_pt_update *pt_update) {struct xe_vm *vm = pt_update->vops->vm;struct xe_vma_ops *vops = pt_update->vops; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[pt_update->tile_id];struct xe_vma_op *op;int err;@@ -1661,7 +1650,7 @@ static int xe_pt_svm_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)xe_pt_svm_userptr_notifier_lock (vm);list_for_each_entry (op, &vops->list, link) { - err =op_check_svm_userptr (vm, op, pt_update_ops); + err =op_check_svm_userptr (vm, op);if (err) {xe_pt_svm_userptr_notifier_unlock (vm);break ;@@ -2008,10 +1997,10 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, }static void -xe_migrate_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,u32 qword_ofs, -u32 num_qwords, -const struct xe_vm_pgtable_update *update) +xe_pt_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,u32 qword_ofs, +u32 num_qwords, +const struct xe_vm_pgtable_update *update) {u64 empty =__xe_pt_empty_pte (tile, vm, update->level);int i;@@ -2089,6 +2078,9 @@ to_pt_op(struct xe_vm_pgtable_update_ops *pt_update_ops, u32 op_idx)static u32 get_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) { +if (!pt_update_ops->pt_job_ops) +return 0 ; +return pt_update_ops->pt_job_ops->current_op; }@@ -2378,6 +2370,7 @@ static int unbind_range_prepare(struct xe_vm *vm,static int op_prepare (struct xe_vm *vm,struct xe_tile *tile, +struct xe_vma_ops *vops,struct xe_vm_pgtable_update_ops *pt_update_ops,struct xe_vma_op *op) {@@ -2394,7 +2387,7 @@ static int op_prepare(struct xe_vm *vm, err =bind_op_prepare (vm, tile, pt_update_ops, op->map.vma, op->map.invalidate_on_bind); - pt_update_ops->wait_vm_kernel = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_KERNEL ;break ;case DRM_GPUVA_OP_REMAP : {@@ -2408,12 +2401,12 @@ static int op_prepare(struct xe_vm *vm,if (!err && op->remap.prev && !op->remap.skip_prev) { err =bind_op_prepare (vm, tile, pt_update_ops, op->remap.prev, false); - pt_update_ops->wait_vm_bookkeep = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; }if (!err && op->remap.next && !op->remap.skip_next) { err =bind_op_prepare (vm, tile, pt_update_ops, op->remap.next, false); - pt_update_ops->wait_vm_bookkeep = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; }break ; }@@ -2449,7 +2442,7 @@ static int op_prepare(struct xe_vm *vm, } }else { err =bind_op_prepare (vm, tile, pt_update_ops, vma, false); - pt_update_ops->wait_vm_kernel = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_KERNEL ; }break ; }@@ -2483,18 +2476,8 @@ xe_pt_update_ops_init(struct xe_vm_pgtable_update_ops *pt_update_ops)xe_page_reclaim_list_init (&pt_update_ops->prl); } -/** -* xe_pt_update_ops_prepare() - Prepare PT update operations -* @tile: Tile of PT update operations -* @vops: VMA operationa -* -* Prepare PT update operations which includes updating internal PT state, -* allocate memory for page tables, populate page table being pruned in, and -* create PT update operations for leaf insertion / removal. -* -* Return: 0 on success, negative error code on error. -*/ -int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops) +static int __xe_pt_update_ops_prepare (struct xe_tile *tile, +struct xe_vma_ops *vops) {struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[tile->id];@@ -2513,7 +2496,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)return err;list_for_each_entry (op, &vops->list, link) { - err =op_prepare (vops->vm, tile, pt_update_ops, op); + err =op_prepare (vops->vm, tile, vops, pt_update_ops, op);if (err)return err;@@ -2522,6 +2505,16 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)xe_tile_assert (tile,get_current_op (pt_update_ops) <= pt_update_ops->num_ops); +/* Propagate individual tile state up to VMA operation */ +if (pt_update_ops->start < vops->start) + vops->start = pt_update_ops->start; +if (pt_update_ops->last > vops->last) + vops->last = pt_update_ops->last; +if (pt_update_ops->needs_invalidation) + vops->flags |=XE_VMA_OPS_FLAG_NEEDS_INVALIDATION ; +if (pt_update_ops->needs_svm_lock) + vops->flags |=XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK ; +#ifdef TEST_VM_OPS_ERROR if (vops->inject_error && vops->vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_PREPARE )@@ -2530,35 +2523,68 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)return 0 ; } -ALLOW_ERROR_INJECTION (xe_pt_update_ops_prepare,ERRNO ); -static void bind_op_commit (struct xe_vm *vm,struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma *vma,struct dma_fence *fence, -struct dma_fence *fence2,bool invalidate_on_bind) +/** +* xe_pt_update_ops_prepare() - Prepare PT update operations +* @xe: xe device. +* @vops: VMA operationa +* +* Prepare PT update operations which includes updating internal PT state, +* allocate memory for page tables, populate page table being pruned in, and +* create PT update operations for leaf insertion / removal. +* +* Return: 0 on success, negative error code on error. +*/ +int xe_pt_update_ops_prepare (struct xe_device *xe,struct xe_vma_ops *vops) { -xe_tile_assert (tile, !xe_vma_is_cpu_addr_mirror (vma)); +struct xe_tile *tile; +int id, err; + +for_each_tile (tile, xe, id) { +if (!vops->pt_update_ops[id].num_ops) +continue ; -if (!xe_vma_has_no_bo (vma) && !xe_vma_bo (vma)->vm) { -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); -if (fence2) -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence2, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); + err =__xe_pt_update_ops_prepare (tile, vops); +if (err) +return err; } + +return 0 ; +} +ALLOW_ERROR_INJECTION (xe_pt_update_ops_prepare,ERRNO ); + +static void vma_add_fences (struct xe_vma *vma,struct dma_fence **fences, +int fence_count,enum dma_resv_usage usage) +{ +int i; + +if (xe_vma_has_no_bo (vma) ||xe_vma_bo (vma)->vm) +return ; + +for (i =0 ; i < fence_count; ++i) +if (fences[i]) +dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, + fences[i], usage); +} + +static void bind_op_commit (struct xe_vm *vm,struct xe_vma *vma, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask, +bool invalidate_on_bind) +{ +xe_assert (vm->xe, !xe_vma_is_cpu_addr_mirror (vma)); + +vma_add_fences (vma, fences, fence_count, usage); +/* All WRITE_ONCE pair with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ -WRITE_ONCE (vma->tile_present, vma->tile_present |BIT (tile->id)); +WRITE_ONCE (vma->tile_present, vma->tile_present | tile_mask);if (invalidate_on_bind)WRITE_ONCE (vma->tile_invalidated, - vma->tile_invalidated |BIT (tile->id)); + vma->tile_invalidated | tile_mask);else WRITE_ONCE (vma->tile_invalidated, - vma->tile_invalidated & ~BIT (tile->id)); - vma->tile_staged &= ~BIT (tile->id); + vma->tile_invalidated & ~tile_mask); + vma->tile_staged &= ~tile_mask;if (xe_vma_is_userptr (vma)) {xe_svm_assert_held_read_or_inject_write (vm);to_userptr_vma (vma)->userptr.initial_bind = true;@@ -2568,31 +2594,21 @@ static void bind_op_commit(struct xe_vm *vm, struct xe_tile *tile,* Kick rebind worker if this bind triggers preempt fences and not in * the rebind worker */ -if (pt_update_ops->wait_vm_bookkeep && +if (usage ==DMA_RESV_USAGE_KERNEL &&xe_vm_in_preempt_fence_mode (vm) && !current->mm)xe_vm_queue_rebind_worker (vm); } -static void unbind_op_commit (struct xe_vm *vm,struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma *vma,struct dma_fence *fence, -struct dma_fence *fence2) +static void unbind_op_commit (struct xe_vm *vm,struct xe_vma *vma, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask) { -xe_tile_assert (tile, !xe_vma_is_cpu_addr_mirror (vma)); +xe_assert (vm->xe, !xe_vma_is_cpu_addr_mirror (vma)); -if (!xe_vma_has_no_bo (vma) && !xe_vma_bo (vma)->vm) { -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); -if (fence2) -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence2, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); - } - vma->tile_present &= ~BIT (tile->id); +vma_add_fences (vma, fences, fence_count, usage); + + vma->tile_present &= ~tile_mask;if (!vma->tile_present) {list_del_init (&vma->combined_links.rebind);if (xe_vma_is_userptr (vma)) {@@ -2607,21 +2623,19 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile,static void range_present_and_invalidated_tile (struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_id) +u8 tile_mask) {/* All WRITE_ONCE pair with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ lockdep_assert_held (&vm->svm.gpusvm.notifier_lock); -WRITE_ONCE (range->tile_present, range->tile_present |BIT (tile_id)); -WRITE_ONCE (range->tile_invalidated, range->tile_invalidated & ~BIT (tile_id)); +WRITE_ONCE (range->tile_present, range->tile_present | tile_mask); +WRITE_ONCE (range->tile_invalidated, range->tile_invalidated & ~tile_mask); } -static void op_commit (struct xe_vm *vm, -struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma_op *op,struct dma_fence *fence, -struct dma_fence *fence2) +static void op_commit (struct xe_vm *vm,struct xe_vma_op *op, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask) {xe_vm_assert_held (vm);@@ -2631,8 +2645,8 @@ static void op_commit(struct xe_vm *vm, (op->map.vma_flags & XE_VMA_SYSTEM_ALLOCATOR)) break; -bind_op_commit (vm, tile, pt_update_ops, op->map.vma, fence, - fence2, op->map.invalidate_on_bind); +bind_op_commit (vm, op->map.vma, fences, fence_count, usage, + tile_mask, op->map.invalidate_on_bind);break ;case DRM_GPUVA_OP_REMAP : {@@ -2641,14 +2655,15 @@ static void op_commit(struct xe_vm *vm,if (xe_vma_is_cpu_addr_mirror (old))break ; -unbind_op_commit (vm, tile, pt_update_ops, old, fence, fence2); +unbind_op_commit (vm, old, fences, fence_count, usage, + tile_mask);if (op->remap.prev && !op->remap.skip_prev) -bind_op_commit (vm, tile, pt_update_ops, op->remap.prev, - fence, fence2, false); +bind_op_commit (vm, op->remap.prev, fences, fence_count, + usage, tile_mask, false);if (op->remap.next && !op->remap.skip_next) -bind_op_commit (vm, tile, pt_update_ops, op->remap.next, - fence, fence2, false); +bind_op_commit (vm, op->remap.next, fences, fence_count, + usage, tile_mask, false);break ; }case DRM_GPUVA_OP_UNMAP :@@ -2656,8 +2671,8 @@ static void op_commit(struct xe_vm *vm,struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va);if (!xe_vma_is_cpu_addr_mirror (vma)) -unbind_op_commit (vm, tile, pt_update_ops, vma, fence, - fence2); +unbind_op_commit (vm, vma, fences, fence_count, + usage, tile_mask);break ; }case DRM_GPUVA_OP_PREFETCH :@@ -2669,10 +2684,11 @@ static void op_commit(struct xe_vm *vm,unsigned long i;xa_for_each (&op->prefetch_range.range, i, range) -range_present_and_invalidated_tile (vm, range, tile->id); +range_present_and_invalidated_tile (vm, range, + tile_mask); }else { -bind_op_commit (vm, tile, pt_update_ops, vma, fence, - fence2, false); +bind_op_commit (vm, vma, fences, fence_count, usage, + tile_mask, false); }break ; }@@ -2680,11 +2696,12 @@ static void op_commit(struct xe_vm *vm, {/* WRITE_ONCE pairs with READ_ONCE in xe_vm_has_valid_gpu_mapping() */ if (op->subop ==XE_VMA_SUBOP_MAP_RANGE ) -range_present_and_invalidated_tile (vm, op->map_range.range, tile->id); +range_present_and_invalidated_tile (vm, op->map_range.range, + tile_mask);else if (op->subop ==XE_VMA_SUBOP_UNMAP_RANGE )WRITE_ONCE (op->unmap_range.range->tile_present, op->unmap_range.range->tile_present & - ~BIT (tile->id)); + ~tile_mask);break ; }@@ -2693,40 +2710,25 @@ static void op_commit(struct xe_vm *vm, } } -static const struct xe_migrate_pt_update_ops migrate_ops = { +static const struct xe_cpu_bind_pt_update_ops cpu_bind_ops = { .populate = xe_vm_populate_pgtable, - .clear = xe_migrate_clear_pgtable_callback, + .clear = xe_pt_clear_pgtable_callback, .pre_commit = xe_pt_pre_commit, };#if IS_ENABLED (CONFIG_DRM_GPUSVM ) -static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops = { +static const struct xe_cpu_bind_pt_update_ops svm_userptr_cpu_bind_ops = { .populate = xe_vm_populate_pgtable, - .clear = xe_migrate_clear_pgtable_callback, + .clear = xe_pt_clear_pgtable_callback, .pre_commit = xe_pt_svm_userptr_pre_commit, };#else -static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops; +static const struct xe_cpu_bind_pt_update_ops svm_userptr_cpu_bind_ops;#endif -static struct xe_dep_scheduler *to_dep_scheduler (struct xe_exec_queue *q, -struct xe_tile *tile, -struct xe_gt *gt, -unsigned int *type) -{ -int tile_ofs = tile->id * (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); - -if (xe_gt_is_media_type (gt)) - *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ; -else - *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ; - -return q->tlb_inval[*type].dep_scheduler; -} -/** * xe_pt_update_ops_run() - Run PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operationa * * Run PT update operations which includes committing internal PT state changes, @@ -2736,82 +2738,83 @@ static struct xe_dep_scheduler *to_dep_scheduler(struct xe_exec_queue *q,* Return: fence on success, negative ERR_PTR on error. */ struct dma_fence * -xe_pt_update_ops_run (struct xe_tile *tile,struct xe_vma_ops *vops) +xe_pt_update_ops_run (struct xe_device *xe,struct xe_vma_ops *vops) {struct xe_vm *vm = vops->vm; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; -struct xe_exec_queue *q = pt_update_ops->q; -struct dma_fence *fence, *ifence =NULL , *mfence =NULL ; -struct xe_tlb_inval_job *ijob =NULL , *mjob =NULL ; +struct xe_exec_queue *q = vops->q; +struct dma_fence *fence; +struct dma_fence *ifences[XE_CPU_BIND_INVAL_JOB_COUNT ] = {};struct xe_range_fence *rfence; +enum dma_resv_usage usage =DMA_RESV_USAGE_BOOKKEEP ;struct xe_vma_op *op; -unsigned int type; -int err =0 , i; -struct xe_migrate_pt_update update = { - .ops = pt_update_ops->needs_svm_lock ? - &svm_userptr_migrate_ops : - &migrate_ops, +struct xe_tile *tile; +int err =0 , total_ops =0 , i, j; +u8 tile_mask =0 ; +bool needs_invalidation = vops->flags & +XE_VMA_OPS_FLAG_NEEDS_INVALIDATION ; +bool needs_svm_lock = vops->flags & +XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK ; +struct xe_cpu_bind_pt_update update = { + .ops = needs_svm_lock ? &svm_userptr_cpu_bind_ops : + &cpu_bind_ops, .vops = vops, - .tile_id = tile->id, };lockdep_assert_held (&vm->lock);xe_vm_assert_held (vm); -if (!get_current_op (pt_update_ops)) { -xe_tile_assert (tile,xe_vm_in_fault_mode (vm)); +for_each_tile (tile, xe, j) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[j]; + total_ops +=get_current_op (pt_update_ops); + } +if (!total_ops) { +xe_assert (xe,xe_vm_in_fault_mode (vm));return dma_fence_get_stub (); }#ifdef TEST_VM_OPS_ERROR if (vops->inject_error && - vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_RUN ) + xe->vm_inject_error_position ==FORCE_OP_ERROR_RUN )return ERR_PTR (-ENOSPC );#endif -if (pt_update_ops->needs_invalidation) { -struct xe_dep_scheduler *dep_scheduler = -to_dep_scheduler (q, tile, tile->primary_gt, &type); - - ijob =xe_tlb_inval_job_create (q, &tile->primary_gt->tlb_inval, - dep_scheduler, vm, - pt_update_ops->start, - pt_update_ops->last, - type); -if (IS_ERR (ijob)) { - err =PTR_ERR (ijob); - goto kill_vm_tile1; - } - update.ijob = ijob; -/* -* Only add page reclaim for the primary GT. Media GT does not have -* any PPC to flush, so enabling the PPC flush bit for media is -* effectively a NOP and provides no performance benefit nor -* interfere with primary GT. -*/ -if (xe_page_reclaim_list_valid (&pt_update_ops->prl)) { -xe_tlb_inval_job_add_page_reclaim (ijob, &pt_update_ops->prl); -/* Release ref from alloc, job will now handle it */ -xe_page_reclaim_list_invalidate (&pt_update_ops->prl); - } - -if (tile->media_gt) { - dep_scheduler =to_dep_scheduler (q, tile, - tile->media_gt, &type); - - mjob =xe_tlb_inval_job_create (q, - &tile->media_gt->tlb_inval, - dep_scheduler, vm, - pt_update_ops->start, - pt_update_ops->last, - type); -if (IS_ERR (mjob)) { - err =PTR_ERR (mjob); +if (needs_invalidation) { +for_each_tlb_inval (q, i) { +struct xe_dep_scheduler *dep_scheduler = + q->tlb_inval[i].dep_scheduler; +struct xe_tile *tile = + &xe->tiles[i /XE_MAX_GT_PER_TILE ]; +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[tile->id]; +struct xe_page_reclaim_list *prl = &pt_update_ops->prl; +struct xe_tlb_inval_job *ijob; +struct xe_gt *gt = i %XE_MAX_GT_PER_TILE ? + tile->media_gt : tile->primary_gt; + + ijob =xe_tlb_inval_job_create (q, >->tlb_inval, + dep_scheduler, + vm, vops->start, + vops->last, i); +if (IS_ERR (ijob)) { + err =PTR_ERR (ijob); goto free_ijob; } - update.mjob = mjob; + + update.ijobs[i] = ijob; + +/* +* Only add page reclaim for the primary GT. Media GT +* does not have any PPC to flush, so enabling the PPC +* flush bit for media is effectively a NOP and provides +* no performance benefit nor interfere with primary GT. +*/ +if (xe_page_reclaim_list_valid (prl)) { +xe_tlb_inval_job_add_page_reclaim (ijob, prl); +/* Release ref from alloc, job will now handle it */ +xe_page_reclaim_list_invalidate (prl); + } } }@@ -2821,67 +2824,61 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) goto free_ijob; } - fence =xe_migrate_update_pgtables (tile->migrate, &update); + fence =xe_cpu_bind_update_pgtables (xe->cpu_bind, &update);if (IS_ERR (fence)) { err =PTR_ERR (fence); goto free_rfence; }/* Point of no return - VM killed if failure after this */ -for (i =0 ; i <get_current_op (pt_update_ops); ++i) { -struct xe_vm_pgtable_update_op *pt_op = -to_pt_op (pt_update_ops, i); - -xe_pt_commit (pt_op->vma, pt_op->entries, - pt_op->num_entries, - &pt_update_ops->pt_job_ops->deferred); - pt_op->vma =NULL ;/* skip in xe_pt_update_ops_abort */ +for_each_tile (tile, xe, j) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[j]; + +for (i =0 ; i <get_current_op (pt_update_ops); ++i) { +struct xe_vm_pgtable_update_op *pt_op = +to_pt_op (pt_update_ops, i); + +xe_pt_commit (pt_op->vma, pt_op->entries, + pt_op->num_entries, + &pt_update_ops->pt_job_ops->deferred); + pt_op->vma =NULL ;/* skip in xe_pt_update_ops_abort */ + tile_mask |=BIT (tile->id); + } } -if (xe_range_fence_insert (&vm->rftree[tile->id], rfence, +if (xe_range_fence_insert (&vm->rftree, rfence, &xe_range_fence_kfree_ops, - pt_update_ops->start, - pt_update_ops->last, fence)) + vops->start, vops->last, fence))dma_fence_wait (fence, false); -if (ijob) - ifence =xe_tlb_inval_job_push (ijob, tile->migrate, fence); -if (mjob) - mfence =xe_tlb_inval_job_push (mjob, tile->migrate, fence); +if (vops->flags &XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ) + usage =DMA_RESV_USAGE_KERNEL ; -if (!mjob && !ijob) { -dma_resv_add_fence (xe_vm_resv (vm), fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); - -list_for_each_entry (op, &vops->list, link) -op_commit (vops->vm, tile, pt_update_ops, op, fence,NULL ); - }else if (ijob && !mjob) { -dma_resv_add_fence (xe_vm_resv (vm), ifence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); +if (!needs_invalidation) { +dma_resv_add_fence (xe_vm_resv (vm), fence, usage);list_for_each_entry (op, &vops->list, link) -op_commit (vops->vm, tile, pt_update_ops, op, ifence,NULL ); +op_commit (vops->vm, op, &fence,1 , usage, tile_mask); }else { -dma_resv_add_fence (xe_vm_resv (vm), ifence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +if (!update.ijobs[i]) +continue ; + + ifences[i] =xe_tlb_inval_job_push (update.ijobs[i], + fence); +xe_assert (xe, !IS_ERR_OR_NULL (ifences[i])); -dma_resv_add_fence (xe_vm_resv (vm), mfence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); +dma_resv_add_fence (xe_vm_resv (vm), ifences[i], usage); + }list_for_each_entry (op, &vops->list, link) -op_commit (vops->vm, tile, pt_update_ops, op, ifence, - mfence); +op_commit (vops->vm, op, ifences, +XE_CPU_BIND_INVAL_JOB_COUNT , usage, + tile_mask); } -if (pt_update_ops->needs_svm_lock) +if (needs_svm_lock)xe_pt_svm_userptr_notifier_unlock (vm);/* @@ -2891,21 +2888,18 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)if (!(q->flags &EXEC_QUEUE_FLAG_MIGRATE ))xe_exec_queue_last_fence_set (q, vm, fence); -xe_tlb_inval_job_put (mjob); -xe_tlb_inval_job_put (ijob); -dma_fence_put (ifence); -dma_fence_put (mfence); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +xe_tlb_inval_job_put (update.ijobs[i]); +dma_fence_put (ifences[i]); + }return fence; free_rfence:kfree (rfence); free_ijob: -xe_tlb_inval_job_put (mjob); -xe_tlb_inval_job_put (ijob); -kill_vm_tile1: -if (err != -EAGAIN && err != -ENODATA && tile->id) -xe_vm_kill (vops->vm, false); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) +xe_tlb_inval_job_put (update.ijobs[i]);return ERR_PTR (err); }@@ -2913,52 +2907,65 @@ ALLOW_ERROR_INJECTION(xe_pt_update_ops_run, ERRNO);/** * xe_pt_update_ops_fini() - Finish PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operations * * Finish PT update operations by committing to destroy page table memory */ -void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops) +void xe_pt_update_ops_fini (struct xe_device *xe,struct xe_vma_ops *vops) { -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; -xe_page_reclaim_entries_put (pt_update_ops->prl.entries); +if (!pt_update_ops->num_ops) +continue ; + +xe_page_reclaim_entries_put (pt_update_ops->prl.entries); + } }/** * xe_pt_update_ops_abort() - Abort PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operationa * * Abort PT update operations by unwinding internal PT state */ -void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops) +void xe_pt_update_ops_abort (struct xe_device *xe,struct xe_vma_ops *vops) { -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; -int i; +struct xe_tile *tile; +int id;lockdep_assert_held (&vops->vm->lock);xe_vm_assert_held (vops->vm); -for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) { -struct xe_vm_pgtable_update_op *pt_op = -to_pt_op (pt_update_ops, i); +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; +int i; -if (!pt_op->vma || i >=get_current_op (pt_update_ops)) -continue ; +for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) { +struct xe_vm_pgtable_update_op *pt_op = +to_pt_op (pt_update_ops, i); -if (pt_op->bind) -xe_pt_abort_bind (pt_op->vma, pt_op->entries, - pt_op->num_entries, - pt_op->rebind); -else -xe_pt_abort_unbind (pt_op->vma, pt_op->entries, - pt_op->num_entries); +if (!pt_op->vma || i >=get_current_op (pt_update_ops)) +continue ; + +if (pt_op->bind) +xe_pt_abort_bind (pt_op->vma, pt_op->entries, + pt_op->num_entries, + pt_op->rebind); +else +xe_pt_abort_unbind (pt_op->vma, pt_op->entries, + pt_op->num_entries); + } } -xe_pt_update_ops_fini (tile, vops); +xe_pt_update_ops_fini (xe, vops); }/** drivers/gpu/drm/xe/xe_pt.h +4
-4@@ -39,11 +39,11 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred);void xe_pt_clear (struct xe_device *xe,struct xe_pt *pt); -int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops); -struct dma_fence *xe_pt_update_ops_run (struct xe_tile *tile, +int xe_pt_update_ops_prepare (struct xe_device *xe,struct xe_vma_ops *vops); +struct dma_fence *xe_pt_update_ops_run (struct xe_device *xe,struct xe_vma_ops *vops); -void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops); -void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops); +void xe_pt_update_ops_fini (struct xe_device *xe,struct xe_vma_ops *vops); +void xe_pt_update_ops_abort (struct xe_device *xe,struct xe_vma_ops *vops);bool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma); bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm,drivers/gpu/drm/xe/xe_pt_types.h +0
-14@@ -120,8 +120,6 @@ struct xe_pt_job_ops {struct xe_vm_pgtable_update_ops {/** @pt_job_ops: PT update operations dynamic allocation*/ struct xe_pt_job_ops *pt_job_ops; -/** @q: exec queue for PT operations */ -struct xe_exec_queue *q;/** @prl: embedded page reclaim list */ struct xe_page_reclaim_list prl;/** @start: start address of ops */ @@ -134,18 +132,6 @@ struct xe_vm_pgtable_update_ops {bool needs_svm_lock;/** @needs_invalidation: Needs invalidation */ bool needs_invalidation; -/** -* @wait_vm_bookkeep: PT operations need to wait until VM is idle -* (bookkeep dma-resv slots are idle) and stage all future VM activity -* behind these operations (install PT operations into VM kernel -* dma-resv slot). -*/ -bool wait_vm_bookkeep; -/** -* @wait_vm_kernel: PT operations need to wait until VM kernel dma-resv -* slots are idle. -*/ - bool wait_vm_kernel; };#endif drivers/gpu/drm/xe/xe_sched_job.c +6
-4@@ -73,8 +73,9 @@ static void job_free(struct xe_sched_job *job)struct xe_exec_queue *q = job->q;bool is_migration =xe_sched_job_is_migration (q); -kmem_cache_free (xe_exec_queue_is_parallel (job->q) || is_migration ? - xe_sched_job_parallel_slab : xe_sched_job_slab, job); +kmem_cache_free (job->is_pt_job ||xe_exec_queue_is_parallel (job->q) || + is_migration ? xe_sched_job_parallel_slab : + xe_sched_job_slab, job); }static struct xe_device *job_to_xe (struct xe_sched_job *job)@@ -127,10 +128,12 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,xe_assert (xe, batch_addr || q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); - job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration); + job =job_alloc (!batch_addr ||xe_exec_queue_is_parallel (q) || + is_migration);if (!job)return ERR_PTR (-ENOMEM ); + job->is_pt_job = !batch_addr; job->q = q; job->sample_timestamp =U64_MAX ;kref_init (&job->refcount);@@ -143,7 +146,6 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,if (!batch_addr) { job->fence =dma_fence_get_stub (); - job->is_pt_job = true; }else {for (i =0 ; i < q->width; ++i) {struct dma_fence *fence =xe_lrc_alloc_seqno_fence ();drivers/gpu/drm/xe/xe_sched_job_types.h +1
-1@@ -14,7 +14,7 @@ struct dma_fence;struct dma_fence_chain ;struct xe_exec_queue ; -struct xe_migrate_pt_update_ops ; +struct xe_cpu_bind_pt_update_ops ;struct xe_pt_job_ops ;struct xe_tile ;struct xe_vm ;
drivers/gpu/drm/xe/xe_sched_job_types.h struct xe_vm; · line 30
Quoted 10 lines
drivers/gpu/drm/xe/xe_sched_job_types.h +3
-4@@ -25,12 +25,11 @@ struct xe_vm;struct xe_pt_update_args {/** @vm: VM which is being bound */ struct xe_vm *vm; -/** @tile: Tile which page tables belong to */ -struct xe_tile *tile; -/** @ops: Migrate PT update ops */ -const struct xe_migrate_pt_update_ops *ops; +/** @ops: CPU bind PT update ops */ +const struct xe_cpu_bind_pt_update_ops *ops; +#define XE_PT_UPDATE_JOB_OPS_COUNT 2
Nit: Instead of hardcoding 2 , shouldn't we use XE_MAX_TILES_PER_DEVICE
Quoted 413 lines
drivers/gpu/drm/xe/xe_sched_job_types.h +1
-1/** @pt_job_ops: PT job ops state */ -struct xe_pt_job_ops *pt_job_ops; +struct xe_pt_job_ops *pt_job_ops[XE_PT_UPDATE_JOB_OPS_COUNT ]; };/** drivers/gpu/drm/xe/xe_tlb_inval_job.c +6
-7@@ -4,6 +4,7 @@*/ #include "xe_assert.h" +#include "xe_cpu_bind.h" #include "xe_dep_job_types.h" #include "xe_dep_scheduler.h" #include "xe_exec_queue.h" @@ -12,7 +13,6 @@#include "xe_page_reclaim.h" #include "xe_tlb_inval.h" #include "xe_tlb_inval_job.h" -#include "xe_migrate.h" #include "xe_pm.h" #include "xe_vm.h" @@ -218,7 +218,6 @@ int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job)/** * xe_tlb_inval_job_push() - TLB invalidation job push * @job: TLB invalidation job to push -* @m: The migration object being used * @fence: Dependency for TLB invalidation job * * Pushes a TLB invalidation job for execution, using @fence as a dependency. @@ -230,11 +229,11 @@ int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job)* Return: Job's finished fence on success, cannot fail */ struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence) {struct xe_tlb_inval_fence *ifence =container_of (job->fence,typeof (*ifence), base); +struct xe_cpu_bind *cpu_bind =gt_to_xe (job->q->gt)->cpu_bind;if (!dma_fence_is_signaled (fence)) {void *ptr;@@ -258,11 +257,11 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, job->fence_armed = true;/* -* We need the migration lock to protect the job's seqno and the spsc -* queue, only taken on migration queue, user queues protected dma-resv +* We need the cpu_bind lock to protect the job's seqno and the spsc +* queue, only taken on cpu_bind queue, user queues protected dma-resv * VM lock. */ -xe_migrate_job_lock (m, job->q); +xe_cpu_bind_job_lock (cpu_bind, job->q);/* Creation ref pairs with put in xe_tlb_inval_job_destroy */ xe_tlb_inval_fence_init (job->tlb_inval, ifence, false);@@ -281,7 +280,7 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, &job->dep.drm.s_fence->finished, job->idx); -xe_migrate_job_unlock (m, job->q); +xe_cpu_bind_job_unlock (cpu_bind, job->q);/* * Not using job->fence, as it has its own dma-fence context, which does drivers/gpu/drm/xe/xe_tlb_inval_job.h +0
-2@@ -11,7 +11,6 @@struct dma_fence ;struct xe_dep_scheduler ;struct xe_exec_queue ; -struct xe_migrate ;struct xe_page_reclaim_list ;struct xe_tlb_inval ;struct xe_tlb_inval_job ;@@ -28,7 +27,6 @@ void xe_tlb_inval_job_add_page_reclaim(struct xe_tlb_inval_job *job,int xe_tlb_inval_job_alloc_dep (struct xe_tlb_inval_job *job);struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence);void xe_tlb_inval_job_get (struct xe_tlb_inval_job *job);drivers/gpu/drm/xe/xe_vm.c +45
-110@@ -24,6 +24,7 @@#include "regs/xe_gtt_defs.h" #include "xe_assert.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_device.h" #include "xe_drm_client.h" #include "xe_exec_queue.h" @@ -795,8 +796,6 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)struct xe_vma *vma, *next;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;lockdep_assert_held (&vm->lock);@@ -804,12 +803,9 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker) list_empty(&vm->rebind_list))return 0 ; -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe->cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ;xe_vm_assert_held (vm);list_for_each_entry (vma, &vm->rebind_list, combined_links.rebind) {@@ -854,21 +850,16 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_mastruct dma_fence *fence =NULL ;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;lockdep_assert_held (&vm->lock);xe_vm_assert_held (vm);xe_assert (vm->xe,xe_vm_in_fault_mode (vm)); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); - vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ; -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe->cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT | +XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; err =xe_vm_ops_add_rebind (&vops, vma, tile_mask);if (err)@@ -944,8 +935,6 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm,struct dma_fence *fence =NULL ;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;lockdep_assert_held (&range->lock);@@ -954,13 +943,10 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm,xe_assert (vm->xe,xe_vm_in_fault_mode (vm));xe_assert (vm->xe,xe_vma_is_cpu_addr_mirror (vma)); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); - vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ; -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe->cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT | +XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask);if (err)@@ -1027,8 +1013,6 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm,struct dma_fence *fence =NULL ;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;lockdep_assert_held (&range->lock);@@ -1039,12 +1023,9 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm,if (!range->tile_present)return dma_fence_get_stub (); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe->cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; err =xe_vm_ops_add_range_unbind (&vops, range);if (err)@@ -1715,9 +1696,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)init_rwsem (&vm->exec_queues.lock);xe_vm_init_prove_locking (xe, vm); - -for_each_tile (tile, xe, id) -xe_range_fence_tree_init (&vm->rftree[id]); +xe_range_fence_tree_init (&vm->rftree); vm->pt_ops = &xelp_pt_ops;@@ -1859,8 +1838,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)xe_svm_fini (vm); err_no_resv:mutex_destroy (&vm->snap_mutex); -for_each_tile (tile, xe, id) -xe_range_fence_tree_fini (&vm->rftree[id]); +xe_range_fence_tree_fini (&vm->rftree);ttm_lru_bulk_move_fini (&xe->ttm, &vm->lru_bulk_move);if (vm->xef)xe_file_put (vm->xef);@@ -1916,10 +1894,8 @@ void xe_vm_close_and_put(struct xe_vm *vm) {LIST_HEAD (contested);struct xe_device *xe = vm->xe; -struct xe_tile *tile;struct xe_vma *vma, *next_vma;struct drm_gpuva *gpuva, *next; -u8 id;xe_assert (xe, !vm->preempt.num_exec_queues);@@ -2011,8 +1987,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)xe_vm_clear_fault_entries (vm); -for_each_tile (tile, xe, id) -xe_range_fence_tree_fini (&vm->rftree[id]); +xe_range_fence_tree_fini (&vm->rftree);xe_vm_put (vm); }@@ -3509,23 +3484,16 @@ static void trace_xe_vm_ops_execute(struct xe_vma_ops *vops)static int vm_ops_setup_tile_args (struct xe_vm *vm,struct xe_vma_ops *vops) { -struct xe_exec_queue *q = vops->q;struct xe_tile *tile;int number_tiles =0 ;u8 id; -for_each_tile (tile, vm->xe, id) { +for_each_tile (tile, vm->xe, id)if (vops->pt_update_ops[id].num_ops) ++number_tiles; -if (vops->pt_update_ops[id].q) -continue ; - -if (q) - vops->pt_update_ops[id].q = q; -else - vops->pt_update_ops[id].q = vm->q; - } +if (!vops->q) + vops->q = vm->q;return number_tiles; }@@ -3533,22 +3501,17 @@ static int vm_ops_setup_tile_args(struct xe_vm *vm, struct xe_vma_ops *vops)static struct dma_fence *ops_execute (struct xe_vm *vm,struct xe_vma_ops *vops) { -struct xe_tile *tile; +struct xe_device *xe = vm->xe;struct dma_fence *fence =NULL ;struct dma_fence **fences =NULL ;struct dma_fence_array *cf =NULL ; -int number_tiles =0 , current_fence =0 , n_fence =0 , err, i; -u8 id; +int current_fence =0 , n_fence =1 , err, i; - number_tiles =vm_ops_setup_tile_args (vm, vops); -if (number_tiles ==0 ) +if (!vm_ops_setup_tile_args (vm, vops))return ERR_PTR (-ENODATA ); -for_each_tile (tile, vm->xe, id) - ++n_fence; -if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) { -for_each_tlb_inval (vops->pt_update_ops[0 ].q, i) +for_each_tlb_inval (vops->q, i) ++n_fence; }@@ -3564,69 +3527,39 @@ static struct dma_fence *ops_execute(struct xe_vm *vm, goto err_out; } -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - - err =xe_pt_update_ops_prepare (tile, vops); -if (err) { - fence =ERR_PTR (err); - goto err_out; - } + err =xe_pt_update_ops_prepare (xe, vops); +if (err) { + fence =ERR_PTR (err); + goto err_out; }trace_xe_vm_ops_execute (vops); -for_each_tile (tile, vm->xe, id) { -struct xe_exec_queue *q = vops->pt_update_ops[tile->id].q; - - fence =NULL ; -if (!vops->pt_update_ops[id].num_ops) - goto collect_fences; - - fence =xe_pt_update_ops_run (tile, vops); -if (IS_ERR (fence)) - goto err_out; - -collect_fences: - fences[current_fence++] = fence ?:dma_fence_get_stub (); -if (vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ) -continue ; - -xe_migrate_job_lock (tile->migrate, q); -for_each_tlb_inval (q, i) { -if (i >= (tile->id +1 ) *XE_MAX_GT_PER_TILE || - i < tile->id *XE_MAX_GT_PER_TILE ) -continue ; + fence =xe_pt_update_ops_run (xe, vops); +if (IS_ERR (fence)) + goto err_out; + fences[current_fence++] = fence; - fences[current_fence++] = fence ? -xe_exec_queue_tlb_inval_last_fence_get (q, vm, i) : -dma_fence_get_stub (); - } -xe_migrate_job_unlock (tile->migrate, q); +if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) { +xe_cpu_bind_job_lock (xe->cpu_bind, vops->q); +for_each_tlb_inval (vops->q, i) + fences[current_fence++] = +xe_exec_queue_tlb_inval_last_fence_get (vops->q, + vm, i); +xe_cpu_bind_job_unlock (xe->cpu_bind, vops->q); } -xe_assert (vm->xe, current_fence == n_fence); +xe_assert (xe, current_fence == n_fence);dma_fence_array_init (cf, n_fence, fences,dma_fence_context_alloc (1 ),1 ); fence = &cf->base; -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - -xe_pt_update_ops_fini (tile, vops); - } +xe_pt_update_ops_fini (xe, vops);return fence; err_out: -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - -xe_pt_update_ops_abort (tile, vops); - } +xe_pt_update_ops_abort (xe, vops);while (current_fence)dma_fence_put (fences[--current_fence]);kfree (fences);@@ -3938,6 +3871,8 @@ static void xe_vma_ops_init(struct xe_vma_ops *vops, struct xe_vm *vm, vops->syncs = syncs; vops->num_syncs = num_syncs; vops->flags =0 ; + vops->start = ~0x0ull ; + vops->last =0x0ull ; } static int xe_vm_bind_ioctl_validate_bo(struct xe_device *xe, struct xe_bo *bo,drivers/gpu/drm/xe/xe_vm_types.h +1
-1@@ -317,7 +317,7 @@ struct xe_vm {* @rftree: range fence tree to track updates to page table structure. * Used to implement conflict tracking between independent bind engines. */ -struct xe_range_fence_tree rftree[XE_MAX_TILES_PER_DEVICE ]; +struct xe_range_fence_tree rftree;const struct xe_pt_ops *pt_ops;
drivers/gpu/drm/xe/xe_vm_types.h struct xe_vma_ops { · line 563
Quoted 7 lines
drivers/gpu/drm/xe/xe_vm_types.h +4
-0@@ -557,6 +557,10 @@ struct xe_vma_ops {u32 num_syncs;/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ]; +/** @start: start address of ops */ +u64 start; +/** @last: last address of ops */ +u64 last;
This was very lengthy and complex patch. Have tried my best to go through it in detail and see no obvious issues. Issues flagged via sashiko needs addressal, but are not introduced in the patch.
With above Nits address, Patch LTGM
Quoted 14 lines
drivers/gpu/drm/xe/xe_vm_types.h +4
-0/** @flag: signify the properties within xe_vma_ops*/ #define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0)#define XE_VMA_OPS_FLAG_MADVISE BIT(1)@@ -566,6 +570,10 @@ struct xe_vma_ops {#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5)#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6)#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7) +#define XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP BIT(8) +#define XE_VMA_OPS_FLAG_WAIT_VM_KERNEL BIT(9) +#define XE_VMA_OPS_FLAG_NEEDS_INVALIDATION BIT(10) +#define XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK BIT(11)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */
RE: [PATCH v6 16/24] drm/xe: Add CPU bind layer
In reply to Matthew BrostQuoted 93 lines · Matthew Brost
With CPU binds, it no longer makes sense to implement CPU bind handling in the migrate layer, as these operations are entirely decoupled from hardware. Introduce a dedicated CPU bind layer stored at the device level.
Since CPU binds are tile-independent, update the PT layer to generate a single bind job even when pages are mirrored across tiles.
This patch is large because the refactor touches multiple file / layers and ensures functional equivalence before and after the change.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_cpu_bind.c | 298 +++++++++++++ drivers/gpu/drm/xe/xe_cpu_bind.h | 118 +++++ drivers/gpu/drm/xe/xe_device.c | 5 + drivers/gpu/drm/xe/xe_device_types.h | 4 + drivers/gpu/drm/xe/xe_exec_queue.c | 3 +- drivers/gpu/drm/xe/xe_guc_submit.c | 41 +- drivers/gpu/drm/xe/xe_migrate.c | 246 ----------- drivers/gpu/drm/xe/xe_migrate.h | 94 ---- drivers/gpu/drm/xe/xe_pt.c | 549 ++++++++++++------------ drivers/gpu/drm/xe/xe_pt.h | 8 +- drivers/gpu/drm/xe/xe_pt_types.h | 14 - drivers/gpu/drm/xe/xe_sched_job.c | 10 +- drivers/gpu/drm/xe/xe_sched_job_types.h | 11 +- drivers/gpu/drm/xe/xe_tlb_inval_job.c | 13 +- drivers/gpu/drm/xe/xe_tlb_inval_job.h | 2 - drivers/gpu/drm/xe/xe_vm.c | 155 ++----- drivers/gpu/drm/xe/xe_vm_types.h | 10 +- 18 files changed, 813 insertions(+), 769 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.c create mode 100644 drivers/gpu/drm/xe/xe_cpu_bind.hdrivers/gpu/drm/xe/Makefile +0
-0@@ -35,6 +35,7 @@ $(obj)/generated/%_device_wa_oob.c$(obj)/generated/%_device_wa_oob.h: $(obj)/xe
xe-y += xe_bb.o \drivers/gpu/drm/xe/Makefile +1
-0xe_bo.o \ xe_bo_evict.o \ + xe_cpu_bind.o \ xe_dep_scheduler.o \ xe_devcoredump.o \ xe_device.o \drivers/gpu/drm/xe/xe_cpu_bind.c new file +35
-0@@ -0,0 +1,298 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include <drm/drm_managed.h> +#include <linux/mutex.h> + +#include "xe_cpu_bind.h" +#include "xe_device_types.h" +#include "xe_exec_queue.h" +#include "xe_pt.h" +#include "xe_sched_job.h" +#include "xe_trace_bo.h" +#include "xe_vm.h" + +/** +* struct xe_cpu_bind - cpu_bind context. +*/ +struct xe_cpu_bind { +/** @xe: Xe device */ +struct xe_device *xe; +/** @q: Default exec queue used for kernel binds */ +struct xe_exec_queue *q; +/** @job_mutex: Timeline mutex for @q. */ +struct mutex job_mutex; +}; + +static bool is_cpu_bind_queue (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +return cpu_bind->q == q; +} + +static void xe_cpu_bind_fini (void *arg)
drivers/gpu/drm/xe/xe_cpu_bind.c line 45
Quoted 10 lines
drivers/gpu/drm/xe/xe_cpu_bind.c new file +10
-0+{ +struct xe_cpu_bind *cpu_bind = arg; + +mutex_destroy (&cpu_bind->job_mutex); +xe_exec_queue_put (cpu_bind->q); +} + +/** +* xe_cpu_bind_init() - Initialize a cpu_bind context +* @xe: &xe_device
Nit: " @xe: &struct xe_device"
Quoted 31 lines
drivers/gpu/drm/xe/xe_cpu_bind.c new file +13
-0+* +* Return: 0 if successful, negative error code on failure +*/ +int xe_cpu_bind_init (struct xe_device *xe) +{ +struct xe_cpu_bind *cpu_bind = +drmm_kzalloc (&xe->drm,sizeof (*cpu_bind),GFP_KERNEL ); +struct xe_exec_queue *q; + +if (!cpu_bind) +return -ENOMEM ; + + q = xe_exec_queue_create_bind(xe, xe_device_get_root_tile(xe),NULL,
drivers/gpu/drm/xe/xe_cpu_bind.c new file +16
-0+EXEC_QUEUE_FLAG_KERNEL | + EXEC_QUEUE_FLAG_MIGRATE, 0); +if (IS_ERR (q)) +return PTR_ERR (q); + + cpu_bind->xe = xe; + cpu_bind->q = q; + xe->cpu_bind = cpu_bind; + +mutex_init (&cpu_bind->job_mutex); + +fs_reclaim_acquire (GFP_KERNEL ); +might_lock (&cpu_bind->job_mutex); +fs_reclaim_release (GFP_KERNEL ); + + return devm_add_action_or_reset(cpu_bind->xe->drm.dev,
xe_cpu_bind_fini,
drivers/gpu/drm/xe/xe_cpu_bind.c line 82
Quoted 8 lines
drivers/gpu/drm/xe/xe_cpu_bind.c new file +8
-0+ cpu_bind); +} + +/** +* xe_cpu_bind_queue() - Get the bind queue from cpu_bind context. +* @cpu_bind: The cpu bind context. +* +* Return: Pointer to bind queue on success, error on failure
Nit: " Return: Pointer to the bind queue. "
drivers/gpu/drm/xe/xe_cpu_bind.c line 93
Quoted 11 lines
+ */
drivers/gpu/drm/xe/xe_cpu_bind.c new file +10
-0+struct xe_exec_queue *xe_cpu_bind_queue (struct xe_cpu_bind *cpu_bind) +{ +return cpu_bind->q; +} + +/** +* xe_cpu_bind_update_pgtables_execute() - Update a VM's PTEs via the CPU +* @vm: The VM being updated +* @tile: The tile being updated +* @ops: The migrate PT update ops
Nit: " @ops: The CPU bind PT update ops"
Quoted 130 lines
+ * @pt_op: The VM PT update op
+ * @num_ops: The number of The VM PT update ops
+ *
+ * Execute the VM PT update ops array which results in a VM's PTEs being updated
+ * via the CPU.
+ */drivers/gpu/drm/xe/xe_cpu_bind.c new file +3
-0+void +xe_cpu_bind_update_pgtables_execute (struct xe_vm *vm,struct xe_tile *tile, + const struct xe_cpu_bind_pt_update_ops*ops,
drivers/gpu/drm/xe/xe_cpu_bind.c new file +13
-0+ struct xe_vm_pgtable_update_op *pt_op, + u32 num_ops) +{ + u32 j, i; + + for (j = 0; j < num_ops; ++j, ++pt_op) { + for (i = 0; i < pt_op->num_entries; i++) { + const struct xe_vm_pgtable_update *update = + &pt_op->entries[i]; + + xe_assert(vm->xe, update); + xe_assert(vm->xe, update->pt_bo); + xe_assert(vm->xe, !iosys_map_is_null(&update-Quoted 1 line
pt_bo->vmap));drivers/gpu/drm/xe/xe_cpu_bind.c new file +41
-0+ +if (pt_op->bind) + ops->populate (tile, &update->pt_bo->vmap, + update->ofs, update->qwords, + update); +else + ops->clear (vm, tile, &update->pt_bo->vmap, + update->ofs, update->qwords, + update); + } + } + +trace_xe_vm_cpu_bind (vm); +xe_device_wmb (vm->xe); +} + +static struct dma_fence * +xe_cpu_bind_update_pgtables_no_job (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +const struct xe_cpu_bind_pt_update_ops *ops = pt_update->ops; +struct xe_vm *vm = pt_update->vops->vm; +struct xe_tile *tile; +int err, id; + +if (ops->pre_commit) { + pt_update->job =NULL ; + err = ops->pre_commit (pt_update); +if (err) +return ERR_PTR (err); + } + +for_each_tile (tile, vm->xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &pt_update->vops->pt_update_ops[tile->id]; + +if (!pt_update_ops->pt_job_ops) +continue ; + + xe_cpu_bind_update_pgtables_execute(vm, tile, ops, + pt_update_ops-Quoted 1 line
pt_job_ops->ops,drivers/gpu/drm/xe/xe_cpu_bind.c new file +1
-0+ pt_update_ops-Quoted 1 line
pt_job_ops->current_op);drivers/gpu/drm/xe/xe_cpu_bind.c new file +61
-0+ } + +return dma_fence_get_stub (); +} + +static struct dma_fence * +xe_cpu_bind_update_pgtables_job (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +const struct xe_cpu_bind_pt_update_ops *ops = pt_update->ops; +struct xe_exec_queue *q = pt_update->vops->q; +struct xe_device *xe = cpu_bind->xe; +struct xe_sched_job *job; +struct dma_fence *fence; +struct xe_tile *tile; +int err, id; +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + + job =xe_sched_job_create (q,NULL ); +if (IS_ERR (job)) +return ERR_CAST (job); + +xe_assert (xe, job->is_pt_job); + +if (ops->pre_commit) { + pt_update->job = job; + err = ops->pre_commit (pt_update); +if (err) + goto err_job; + } + +if (is_cpu_bind) +mutex_lock (&cpu_bind->job_mutex); + + job->pt_update[0 ].vm = pt_update->vops->vm; + job->pt_update[0 ].ops = ops; +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &pt_update->vops->pt_update_ops[tile->id]; + + job->pt_update[0 ].pt_job_ops[tile->id] = +xe_pt_job_ops_get (pt_update_ops->pt_job_ops); + } + +xe_sched_job_arm (job); + fence =dma_fence_get (&job->drm.s_fence->finished); +xe_sched_job_push (job); + +if (is_cpu_bind) +mutex_unlock (&cpu_bind->job_mutex); + +return fence; + +err_job: +xe_sched_job_put (job); +return ERR_PTR (err); +} + +/** +* xe_cpu_bind_update_pgtables() - Pipelined page-table update +* @cpu_bind: The cpu bind context.
drivers/gpu/drm/xe/xe_cpu_bind.c line 228
Quoted 10 lines
drivers/gpu/drm/xe/xe_cpu_bind.c new file +10
-0+* @pt_update: PT update arguments +* +* Perform a pipelined page-table update. The update descriptors are typically +* built under the same lock critical section as a call to this function. If +* using the default engine for the updates, they will be performed in the +* order they grab the job_mutex. If different engines are used, external +* synchronization is needed for overlapping updates to maintain page-table +* consistency. Note that the meaning of "overlapping" is that the updates +* touch the same page-table, which might be a higher-level page-directory. +* If no pipelining is needed, then updates may be performed by the cpu.
This doc needs to be updated with cpu bind.
Quoted 99 lines
+ *
+ * Return: Adma_fencethat, when signaled, indicates the update completion.
+ */drivers/gpu/drm/xe/xe_cpu_bind.c new file +32
-0+struct dma_fence * +xe_cpu_bind_update_pgtables (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update) +{ +struct dma_fence *fence; + + fence =xe_cpu_bind_update_pgtables_no_job (cpu_bind, pt_update); + +/* -ETIME indicates a job is needed, anything else is legit error */ +if (!IS_ERR (fence) ||PTR_ERR (fence) != -ETIME ) +return fence; + +return xe_cpu_bind_update_pgtables_job (cpu_bind, pt_update); +} + +/** +* xe_cpu_bind_job_lock() - Lock cpu_bind job lock +* @cpu_bind: The cpu bind context. +* @q: Queue associated with the operation which requires a lock +* +* Lock the cpu_bind job lock if the queue is a cpu bind queue, otherwise +* assert the VM's dma-resv is held (user queue's have own locking). +*/ +void xe_cpu_bind_job_lock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + +if (is_cpu_bind) +mutex_lock (&cpu_bind->job_mutex); +else +xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */
drivers/gpu/drm/xe/xe_cpu_bind.c new file +19
-0+} + +/** +* xe_cpu_bind_job_unlock() - Unlock cpu_bind job lock +* @cpu_bind: The cpu bind context. +* @q: Queue associated with the operation which requires a lock +* +* Unlock the cpu_bind job lock if the queue is a cpu bind queue, otherwise +* assert the VM's dma-resv is held (user queue's have own locking). +*/ +void xe_cpu_bind_job_unlock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q) +{ +bool is_cpu_bind =is_cpu_bind_queue (cpu_bind, q); + +if (is_cpu_bind) +mutex_unlock (&cpu_bind->job_mutex); +else +xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */
drivers/gpu/drm/xe/xe_cpu_bind.c new file +16
-0+} + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +/** +* xe_cpu_bind_job_lock_assert() - Assert cpu_bind job lock held of queue +* @q: cpu bind queue +*/ +void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q) +{ +struct xe_device *xe =gt_to_xe (q->gt); +struct xe_cpu_bind *cpu_bind = xe->cpu_bind; + +xe_assert (xe, q == cpu_bind->q); +lockdep_assert_held (&cpu_bind->job_mutex); +} +#endif drivers/gpu/drm/xe/xe_cpu_bind.h new file +19
-0@@ -0,0 +1,118 @@ +/* SPDX-License-Identifier: MIT */ +/* +* Copyright © 2026 Intel Corporation +*/ + +#ifndef _XE_CPU_BIND_H_ +#define _XE_CPU_BIND_H_ + +#include <linux/types.h> + +struct dma_fence ; +struct iosys_map ; +struct xe_cpu_bind ; +struct xe_cpu_bind_pt_update ; +struct xe_device ; +struct xe_tlb_inval_job ; +struct xe_tile ; +struct xe_vm ; +struct xe_vm_pgtable_update ;
drivers/gpu/drm/xe/xe_cpu_bind.h line 29
Quoted 10 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +10
-0+struct xe_vm_pgtable_update_op ; +struct xe_vma_ops ; + +/** +* struct xe_cpu_bind_pt_update_ops - Callbacks for the +* xe_cpu_bind_update_pgtables() function. +*/ +struct xe_cpu_bind_pt_update_ops { +/** +* @populate: Populate a command buffer or page-table with ptes.
Nit: " @populate: Populate a page-table with ptes."
drivers/gpu/drm/xe/xe_cpu_bind.h line 32
Quoted 3 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +3
-0+* @tile: The tile for the current operation. +* @map: struct iosys_map into the memory to be populated. +* @ofs: qword offset into @map, unused if @map is NULL.
Nit: " @ofs: qword offset into @map."
drivers/gpu/drm/xe/xe_cpu_bind.h line 38
Quoted 6 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +6
-0+* @num_qwords: Number of qwords to write. +* @update: Information about the PTEs to be inserted. +* +* This interface is intended to be used as a callback into the +* page-table system to populate command buffers or shared +* page-tables with PTEs.
Nit: " page-table system to populate shared page-tables with PTEs. "
drivers/gpu/drm/xe/xe_cpu_bind.h line 44
Quoted 6 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +6
-0+*/ +void (*populate)(struct xe_tile *tile,struct iosys_map *map, +u32 ofs,u32 num_qwords, +const struct xe_vm_pgtable_update *update); +/** +* @clear: Clear a command buffer or page-table with ptes.
Nit: " @clear: Clear a page-table's ptes."
drivers/gpu/drm/xe/xe_cpu_bind.h line 47
Quoted 3 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +3
-0+* @vm: VM being updated +* @tile: The tile for the current operation. +* @map: struct iosys_map into the memory to be populated.
Nit: " @map: struct iosys_map into the page-table to be cleared. "
drivers/gpu/drm/xe/xe_cpu_bind.h line 48
Quoted 1 line
drivers/gpu/drm/xe/xe_cpu_bind.h new file +1
-0+* @ofs: qword offset into @map, unused if @map is NULL.
Nit: " @ofs: qword offset into @map. "
drivers/gpu/drm/xe/xe_cpu_bind.h line 50
Quoted 2 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +2
-0+* @num_qwords: Number of qwords to write. +* @update: Information about the PTEs to be inserted.
Nit: " @update: Information about the PTEs to be cleared. "
drivers/gpu/drm/xe/xe_cpu_bind.h line 38
Quoted 4 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +4
-0+* +* This interface is intended to be used as a callback into the +* page-table system to populate command buffers or shared +* page-tables with PTEs.
Nit: " page-table system to clear PTEs from shared page-tables. "
Quoted 24 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +23
-0+*/ +void (*clear)(struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,u32 ofs,u32 num_qwords, +const struct xe_vm_pgtable_update *update); + +/** +* @pre_commit: Callback to be called just before arming the +* sched_job. +* @pt_update: Pointer to embeddable callback argument. +* +* Return: 0 on success, negative error code on error. +*/ +int (*pre_commit)(struct xe_cpu_bind_pt_update *pt_update); +}; + +/** +* struct xe_cpu_bind_pt_update - Argument to the struct +* xe_cpu_bind_pt_update_ops callbacks. +* +* Intended to be subclassed to support additional arguments if necessary. +*/ +struct xe_cpu_bind_pt_update { +/** @ops: Pointer to the struct xe_cpu_bind_pt_update_ops callbacks */
drivers/gpu/drm/xe/xe_cpu_bind.h line 81
Quoted 4 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +4
-0+const struct xe_cpu_bind_pt_update_ops *ops; +/** @vops: VMA operations */ +struct xe_vma_ops *vops; +/** @job: The job if a GPU page-table update. NULL otherwise */
Nit: " @job: The bind job, or NULL if the update is issued immediately "
Quoted 991 lines
drivers/gpu/drm/xe/xe_cpu_bind.h new file +15
-0+struct xe_sched_job *job; +/** +* @ijobs: The TLB invalidation jobs, individual instances can be NULL +*/ +#define XE_CPU_BIND_INVAL_JOB_COUNT 4 +struct xe_tlb_inval_job *ijobs[XE_CPU_BIND_INVAL_JOB_COUNT ]; +}; + +int xe_cpu_bind_init (struct xe_device *xe); + +struct xe_exec_queue *xe_cpu_bind_queue (struct xe_cpu_bind *cpu_bind); + +void +xe_cpu_bind_update_pgtables_execute (struct xe_vm *vm,struct xe_tile *tile, + const struct xe_cpu_bind_pt_update_ops*ops,
drivers/gpu/drm/xe/xe_cpu_bind.h new file +22
-0+struct xe_vm_pgtable_update_op *pt_op, + u32 num_ops); + +struct dma_fence * +xe_cpu_bind_update_pgtables (struct xe_cpu_bind *cpu_bind, +struct xe_cpu_bind_pt_update *pt_update); + +void xe_cpu_bind_job_lock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q); + +void xe_cpu_bind_job_unlock (struct xe_cpu_bind *cpu_bind, +struct xe_exec_queue *q); + +#if IS_ENABLED (CONFIG_PROVE_LOCKING ) +void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q); +#else +static inline void xe_cpu_bind_job_lock_assert (struct xe_exec_queue *q) +{ +} +#endif + +#endif drivers/gpu/drm/xe/xe_device.c +5
-0@@ -27,6 +27,7 @@#include "xe_bo_evict.h" #include "xe_configfs.h" #include "xe_debugfs.h" +#include "xe_cpu_bind.h" #include "xe_defaults.h" #include "xe_devcoredump.h" #include "xe_device_sysfs.h" @@ -1075,6 +1076,10 @@ int xe_device_probe(struct xe_device *xe)if (err)return err; + err =xe_cpu_bind_init (xe); +if (err) +return err; + err =xe_pagefault_init (xe);if (err)return err;drivers/gpu/drm/xe/xe_device_types.h +4
-0@@ -38,6 +38,7 @@struct drm_pagemap_shrinker ;struct intel_display ;struct intel_dg_nvm_dev ; +struct xe_cpu_bind ;struct xe_ggtt ;struct xe_i2c ;struct xe_pat_ops ;@@ -559,6 +560,9 @@ struct xe_device {/** @sc: System Controller */ struct xe_sysctrl sc; +/** @cpu_bind: CPU bind object */ +struct xe_cpu_bind *cpu_bind; +/** @atomic_svm_timeslice_ms: Atomic SVM fault timeslice MS */ u32 atomic_svm_timeslice_ms;drivers/gpu/drm/xe/xe_exec_queue.c +2
-1@@ -14,6 +14,7 @@#include <uapi/drm/xe_drm.h> #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_dep_scheduler.h" #include "xe_device.h" #include "xe_gt.h" @@ -1666,7 +1667,7 @@ static void xe_exec_queue_last_fence_lockdep_assert(struct xe_exec_queue *q, struct xe_vm *vm) {if (q->flags &EXEC_QUEUE_FLAG_MIGRATE ) { -xe_migrate_job_lock_assert (q); +xe_cpu_bind_job_lock_assert (q); }else if (q->flags &EXEC_QUEUE_FLAG_VM ) {lockdep_assert_held (&vm->lock);drivers/gpu/drm/xe/xe_exec_queue.c +0
-0}else {drivers/gpu/drm/xe/xe_guc_submit.c +30
-7@@ -19,6 +19,7 @@#include "abi/guc_klvs_abi.h" #include "xe_assert.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_devcoredump.h" #include "xe_device.h" #include "xe_exec_queue.h" @@ -38,7 +39,6 @@#include "xe_lrc.h" #include "xe_macros.h" #include "xe_map.h" -#include "xe_migrate.h" #include "xe_mocs.h" #include "xe_module.h" #include "xe_pm.h" @@ -1244,13 +1244,36 @@ static bool is_pt_job(struct xe_sched_job *job)return job->is_pt_job; } -static void run_pt_job (struct xe_sched_job *job) +static void run_pt_job (struct xe_device *xe,struct xe_sched_job *job) { -xe_migrate_update_pgtables_cpu_execute (job->pt_update[0 ].vm, - job->pt_update[0 ].tile, - job->pt_update[0 ].ops, - job->pt_update[0 ].pt_job_ops- >ops, - job->pt_update[0 ].pt_job_ops- >current_op); +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_pt_job_ops *pt_job_ops = + job->pt_update[0 ].pt_job_ops[id]; + +if (!pt_job_ops || !pt_job_ops->current_op) +continue ; + +xe_cpu_bind_update_pgtables_execute (job- >pt_update[0].vm, tile, + job->pt_update[0 ].ops, + pt_job_ops->ops, + pt_job_ops->current_op); + } +} + +static void put_pt_job (struct xe_device *xe,struct xe_sched_job *job) +{ +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_pt_job_ops *pt_job_ops = + job->pt_update[0 ].pt_job_ops[id]; + +xe_pt_job_ops_put (pt_job_ops); + }}
static struct
dma_fence*drivers/gpu/drm/xe/xe_guc_submit.c +1
-1@@ -1280,7 +1303,7 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job)if (is_pt_job (job)) {xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q)); -run_pt_job (job); +run_pt_job (guc_to_xe (guc), job); }else {drivers/gpu/drm/xe/xe_guc_submit.c +0
-0if (!exec_queue_registered (q)) register_exec_queue(q,
GUC_CONTEXT_NORMAL);drivers/gpu/drm/xe/xe_guc_submit.c +1
-1@@ -1292,7 +1315,7 @@ guc_exec_queue_run_job(struct drm_sched_job *drm_job) }if (is_pt_job (job)) { -xe_pt_job_ops_put (job->pt_update[0 ].pt_job_ops); +put_pt_job (guc_to_xe (guc), job);dma_fence_put (job->fence);/* Drop ref from
xe_sched_job_arm*/drivers/gpu/drm/xe/xe_guc_submit.c +0
-0return NULL ; }drivers/gpu/drm/xe/xe_migrate.c +0
-5@@ -51,8 +51,6 @@struct xe_migrate {/** @q: Default exec queue used for migration */ struct xe_exec_queue *q; -/** @bind_q: Default exec queue used for binds */ -struct xe_exec_queue *bind_q;/** @tile: Backpointer to the tile this struct xe_migrate belongs to. */ struct xe_tile *tile;/** @job_mutex: Timeline mutex for @eng. */ @@ -110,7 +108,6 @@ static void xe_migrate_fini(void *arg)mutex_destroy (&m->job_mutex);xe_vm_close_and_put (m->q->vm);xe_exec_queue_put (m->q); -xe_exec_queue_put (m->bind_q); } static inline u16 xe_migrate_pat_index(struct xe_device *xe,@@ -483,15 +480,6 @@ int xe_migrate_init(struct xe_migrate *m) goto err_out; } - m->bind_q =xe_exec_queue_create (xe, vm, logical_mask,1 , hwe0, - EXEC_QUEUE_FLAG_KERNEL|
-EXEC_QUEUE_FLAG_HIGH_PRIORITY|
-EXEC_QUEUE_FLAG_MIGRATE, 0);drivers/gpu/drm/xe/xe_migrate.c +0
-8-if (IS_ERR (m->bind_q)) { - err =PTR_ERR (m->bind_q); - goto err_out; - } -/* * XXX: Currently only reserving 1 (likely slow) BCS instance on * PVC, may want to revisit if performance is needed. @@ -502,15 +490,6 @@ int xe_migrate_init(struct xe_migrate *m)EXEC_QUEUE_FLAG_MIGRATE |EXEC_QUEUE_FLAG_LOW_LATENCY , 0); }else { - m->bind_q = xe_exec_queue_create_class(xe, primary_gt, vm, - XE_ENGINE_CLASS_COPY, -
EXEC_QUEUE_FLAG_KERNEL|
-EXEC_QUEUE_FLAG_MIGRATE, 0);drivers/gpu/drm/xe/xe_migrate.c +0
-5-if (IS_ERR (m->bind_q)) { - err =PTR_ERR (m->bind_q); - goto err_out; - } - m->q =xe_exec_queue_create_class (xe, primary_gt, vm,XE_ENGINE_CLASS_COPY , EXEC_QUEUE_FLAG_KERNEL|
drivers/gpu/drm/xe/xe_migrate.c +0
-13@@ -546,8 +525,6 @@ int xe_migrate_init(struct xe_migrate *m)return err; err_out: -if (!IS_ERR_OR_NULL (m->bind_q)) -xe_exec_queue_put (m->bind_q);xe_vm_close_and_put (vm);return err;@@ -1507,17 +1484,6 @@ static u32 blt_mem_set_cmd_len(struct xe_device *xe)return 7 ; } -/** -* xe_get_migrate_bind_queue() - Get the bind queue from migrate context. -* @migrate: Migrate context. -* -* Return: Pointer to bind queue on success, error on failure -*/ -struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate) -{ -return migrate->bind_q; -} - static void emit_clear_link_copy(struct xe_gt *gt, struct xe_bb *bb, u64 src_ofs,drivers/gpu/drm/xe/xe_migrate.c +0
-2u32 size, u32 pitch) {@@ -1782,168 +1748,6 @@ struct migrate_test_params { container_of(_priv, struct migrate_test_params, base)#endif -/** -* xe_migrate_update_pgtables_cpu_execute() - Update a VM's PTEs via the CPU
- * @vm: The VM being updated
- * @tile: The tile being updated
- * @ops: The migrate PT update ops
- * @pt_ops: The VM PT update ops
- * @num_ops: The number of The VM PT update ops
- *
- * Execute the VM PT update ops array which results in a VM's PTEs being updated
- * via the CPU.
- */drivers/gpu/drm/xe/xe_migrate.c +0
-2-void -xe_migrate_update_pgtables_cpu_execute(struct xe_vm *vm, struct xe_tile*tile,
drivers/gpu/drm/xe/xe_migrate.c +0
-1- const struct xe_migrate_pt_update_ops*ops,
drivers/gpu/drm/xe/xe_migrate.c +0
-13- struct xe_vm_pgtable_update_op *pt_op, - u32 num_ops) -{ - u32 j, i; - - for (j = 0; j < num_ops; ++j, ++pt_op) { - for (i = 0; i < pt_op->num_entries; i++) { - const struct xe_vm_pgtable_update *update = - &pt_op->entries[i]; - - xe_tile_assert(tile, update); - xe_tile_assert(tile, update->pt_bo); - xe_tile_assert(tile, !iosys_map_is_null(&update-Quoted 1 line
pt_bo->vmap));drivers/gpu/drm/xe/xe_migrate.c +0
-41- -if (pt_op->bind) - ops->populate (tile, &update->pt_bo->vmap, - update->ofs, update->qwords, - update); -else - ops->clear (vm, tile, &update->pt_bo->vmap, - update->ofs, update->qwords, - update); - } - } - -trace_xe_vm_cpu_bind (vm); -xe_device_wmb (vm->xe); -} - -static struct dma_fence * -xe_migrate_update_pgtables_cpu (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update) -{ - XE_TEST_DECLARE(struct migrate_test_params *test = -to_migrate_test_params - (xe_cur_kunit_priv(XE_TEST_LIVE_MIGRATE));) -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_vm *vm = pt_update->vops->vm; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &pt_update->vops->pt_update_ops[pt_update->tile_id]; -int err; - -if (XE_TEST_ONLY (test && test->force_gpu)) -return ERR_PTR (-ETIME ); - -if (ops->pre_commit) { - pt_update->job =NULL ; - err = ops->pre_commit (pt_update); -if (err) -return ERR_PTR (err); - } - - xe_migrate_update_pgtables_cpu_execute(vm, m->tile, ops, - pt_update_ops->pt_job_ops-Quoted 1 line
ops,
drivers/gpu/drm/xe/xe_migrate.c +0
-14- pt_update_ops->num_ops); - -return dma_fence_get_stub (); -} - -static bool is_migrate_queue (struct xe_migrate *m,struct xe_exec_queue *q) -{ -return m->bind_q == q; -} - -static struct dma_fence * -__xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update, - struct xe_vm_pgtable_update_ops
*pt_update_ops)drivers/gpu/drm/xe/xe_migrate.c +0
-78-{ -const struct xe_migrate_pt_update_ops *ops = pt_update->ops; -struct xe_tile *tile = m->tile; -struct xe_sched_job *job; -struct dma_fence *fence; -bool is_migrate =is_migrate_queue (m, pt_update_ops->q); -int err; - - job =xe_sched_job_create (pt_update_ops->q,NULL ); -if (IS_ERR (job)) { - err =PTR_ERR (job); - goto err_out; - } - -xe_tile_assert (tile, job->is_pt_job); - -if (ops->pre_commit) { - pt_update->job = job; - err = ops->pre_commit (pt_update); -if (err) - goto err_job; - } -if (is_migrate) -mutex_lock (&m->job_mutex); - - job->pt_update[0 ].vm = pt_update->vops->vm; - job->pt_update[0 ].tile = tile; - job->pt_update[0 ].ops = ops; - job->pt_update[0 ].pt_job_ops = -xe_pt_job_ops_get (pt_update_ops->pt_job_ops); - -xe_sched_job_arm (job); - fence =dma_fence_get (&job->drm.s_fence->finished); -xe_sched_job_push (job); - -if (is_migrate) -mutex_unlock (&m->job_mutex); - -return fence; - -err_job: -xe_sched_job_put (job); -err_out: -return ERR_PTR (err); -} - -/** -* xe_migrate_update_pgtables() - Pipelined page-table update -* @m: The migrate context. -* @pt_update: PT update arguments -* -* Perform a pipelined page-table update. The update descriptors are typically -* built under the same lock critical section as a call to this function. If -* using the default engine for the updates, they will be performed in the -* order they grab the job_mutex. If different engines are used, external -* synchronization is needed for overlapping updates to maintain page-table -* consistency. Note that the meaning of "overlapping" is that the updates -* touch the same page-table, which might be a higher-level page-directory. -* If no pipelining is needed, then updates may be performed by the cpu. -* -* Return: A dma_fence that, when signaled, indicates the update completion. -*/ -struct dma_fence * -xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update) - -{ -struct xe_vm_pgtable_update_ops *pt_update_ops = - &pt_update->vops->pt_update_ops[pt_update->tile_id]; -struct dma_fence *fence; - - fence =xe_migrate_update_pgtables_cpu (m, pt_update); - -/* -ETIME indicates a job is needed, anything else is legit error */ -if (!IS_ERR (fence) ||PTR_ERR (fence) != -ETIME ) -return fence; - - return __xe_migrate_update_pgtables(m, pt_update,
pt_update_ops);drivers/gpu/drm/xe/xe_migrate.c +0
-18-} -/** * xe_migrate_wait() - Complete all operations using the xe_migrate context * @m: Migrate context to wait for. @@ -2445,56 +2249,6 @@ int xe_migrate_access_memory(struct xe_migrate *m, struct xe_bo *bo, return IS_ERR(fence) ? PTR_ERR(fence) : 0; } -/** -* xe_migrate_job_lock() - Lock migrate job lock -* @m: The migration context. -* @q: Queue associated with the operation which requires a lock -* -* Lock the migrate job lock if the queue is a migration queue, otherwise -* assert the VM's dma-resv is held (user queue's have own locking). -*/ -void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q) -{ -bool is_migrate =is_migrate_queue (m, q); - -if (is_migrate) -mutex_lock (&m->job_mutex); -else -xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */
drivers/gpu/drm/xe/xe_migrate.c +0
-18-} - -/** -* xe_migrate_job_unlock() - Unlock migrate job lock -* @m: The migration context. -* @q: Queue associated with the operation which requires a lock -* -* Unlock the migrate job lock if the queue is a migration queue, otherwise -* assert the VM's dma-resv is held (user queue's have own locking). -*/ -void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q) -{ -bool is_migrate =is_migrate_queue (m, q); - -if (is_migrate) -mutex_unlock (&m->job_mutex); -else -xe_vm_assert_held (q->user_vm);/* User queues VM's should be locked */
drivers/gpu/drm/xe/xe_migrate.c +0
-16-} - -#if IS_ENABLED (CONFIG_PROVE_LOCKING ) -/** -* xe_migrate_job_lock_assert() - Assert migrate job lock held of queue -* @q: Migrate queue -*/ -void xe_migrate_job_lock_assert (struct xe_exec_queue *q) -{ -struct xe_migrate *m =gt_to_tile (q->gt)->migrate; - -xe_gt_assert (q->gt, q == m->bind_q); -lockdep_assert_held (&m->job_mutex); -} -#endif -#if IS_ENABLED (CONFIG_DRM_XE_KUNIT_TEST )#include "tests/xe_migrate.c" #endif drivers/gpu/drm/xe/xe_migrate.h +0
-73@@ -34,78 +34,6 @@ enum xe_migrate_copy_dir {XE_MIGRATE_COPY_TO_SRAM, }; -/** -* struct xe_migrate_pt_update_ops - Callbacks for the -* xe_migrate_update_pgtables() function. -*/ -struct xe_migrate_pt_update_ops { -/** -* @populate: Populate a command buffer or page-table with ptes. -* @tile: The tile for the current operation. -* @map: struct iosys_map into the memory to be populated. -* @ofs: qword offset into @map, unused if @map is NULL. -* @num_qwords: Number of qwords to write. -* @update: Information about the PTEs to be inserted. -* -* This interface is intended to be used as a callback into the -* page-table system to populate command buffers or shared -* page-tables with PTEs. -*/ -void (*populate)(struct xe_tile *tile,struct iosys_map *map, -u32 ofs,u32 num_qwords, -const struct xe_vm_pgtable_update *update); -/** -* @clear: Clear a command buffer or page-table with ptes. -* @vm: VM being updated -* @tile: The tile for the current operation. -* @map: struct iosys_map into the memory to be populated. -* @ofs: qword offset into @map, unused if @map is NULL. -* @num_qwords: Number of qwords to write. -* @update: Information about the PTEs to be inserted. -* -* This interface is intended to be used as a callback into the -* page-table system to populate command buffers or shared -* page-tables with PTEs. -*/ -void (*clear)(struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,u32 ofs,u32 num_qwords, -const struct xe_vm_pgtable_update *update); - -/** -* @pre_commit: Callback to be called just before arming the -* sched_job. -* @pt_update: Pointer to embeddable callback argument. -* -* Return: 0 on success, negative error code on error. -*/ -int (*pre_commit)(struct xe_migrate_pt_update *pt_update); -}; - -/** -* struct xe_migrate_pt_update - Argument to the -* struct xe_migrate_pt_update_ops callbacks. -* -* Intended to be subclassed to support additional arguments if necessary. -*/ -struct xe_migrate_pt_update { -/** @ops: Pointer to the struct xe_migrate_pt_update_ops callbacks */ -const struct xe_migrate_pt_update_ops *ops; -/** @vops: VMA operations */ -struct xe_vma_ops *vops; -/** @job: The job if a GPU page-table update. NULL otherwise */ -struct xe_sched_job *job; -/** -* @ijob: The TLB invalidation job for primary GT. NULL otherwise -*/ -struct xe_tlb_inval_job *ijob; -/** -* @mjob: The TLB invalidation job for media GT. NULL otherwise -*/ -struct xe_tlb_inval_job *mjob; -/** @tile_id: Tile ID of the update */ -u8 tile_id; -}; -struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m);@@ -143,7 +71,6 @@ void xe_migrate_ccs_rw_copy_clear(struct xe_bo *src_bo, struct xe_lrc *xe_migrate_lrc(struct xe_migrate *migrate);struct xe_exec_queue *xe_migrate_exec_queue (struct xe_migrate *migrate); -struct xe_exec_queue *xe_migrate_bind_queue (struct xe_migrate *migrate); struct dma_fence *xe_migrate_vram_copy_chunk(struct xe_bo *vram_bo, u64 vram_offset,struct
xe_bo*sysmem_bo, u64sysmem_offset,
u64 size, enumxe_migrate_copy_dirdir);drivers/gpu/drm/xe/xe_migrate.h +0
-21@@ -162,29 +89,8 @@ struct dma_fence *xe_migrate_clear(structxe_migrate *m, struct xe_vm *xe_migrate_get_vm(struct xe_migrate *m); -void -xe_migrate_update_pgtables_cpu_execute (struct xe_vm *vm,struct xe_tile *tile, -const struct xe_migrate_pt_update_ops *ops, -struct xe_vm_pgtable_update_op *pt_op, -u32 num_ops); - -struct dma_fence * -xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_migrate_pt_update *pt_update); - void xe_migrate_wait(struct xe_migrate *m); -#if IS_ENABLED (CONFIG_PROVE_LOCKING ) -void xe_migrate_job_lock_assert (struct xe_exec_queue *q); -#else -static inline void xe_migrate_job_lock_assert (struct xe_exec_queue *q) -{ -} -#endif - -void xe_migrate_job_lock (struct xe_migrate *m,struct xe_exec_queue *q); -void xe_migrate_job_unlock (struct xe_migrate *m,struct xe_exec_queue *q); -#if
IS_ENABLED(CONFIG_DRM_XE_DEBUG_MEM)
int xe_migrate_debug_ccs_overlap(structxe_migrate*m,drivers/gpu/drm/xe/xe_migrate.h +0
-0struct xe_bo *scratch_bo,drivers/gpu/drm/xe/xe_pt.c +2
-4@@ -7,12 +7,12 @@#include "regs/xe_gtt_defs.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_device.h" #include "xe_drm_client.h" #include "xe_exec_queue.h" #include "xe_gt.h" #include "xe_gt_stats.h" -#include "xe_migrate.h" #include "xe_page_reclaim.h" #include "xe_pat.h" #include "xe_pt_types.h" @@ -1401,11 +1401,9 @@ static int op_add_deps(struct xe_vm *vm, struct xe_vma_op *op, }static int xe_pt_vm_dependencies (struct xe_sched_job *job, -struct xe_tlb_inval_job *ijob, -struct xe_tlb_inval_job *mjob, +struct xe_tlb_inval_job **ijobs,struct xe_vm *vm,struct xe_vma_ops *vops, -struct xe_vm_pgtable_update_ops *pt_update_ops,struct xe_range_fence_tree *rftree){
drivers/gpu/drm/xe/xe_pt.c +1
-4struct xe_range_fence *rtfence;@@ -1418,20 +1416,22 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job, if (!job && !no_in_syncs(vops->syncs, vops->num_syncs)) return -ETIME; -if (!job && !xe_exec_queue_is_idle (pt_update_ops->q)) +if (!job && !xe_exec_queue_is_idle (vops->q))return -ETIME ; -if (pt_update_ops->wait_vm_bookkeep || pt_update_ops- >wait_vm_kernel) { - err = job_test_add_deps(job, xe_vm_resv(vm), - pt_update_ops-Quoted 1 line
wait_vm_bookkeep?drivers/gpu/drm/xe/xe_pt.c +11
-7- DMA_RESV_USAGE_BOOKKEEP : - DMA_RESV_USAGE_KERNEL); +if (vops->flags & (XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP | +XE_VMA_OPS_FLAG_WAIT_VM_KERNEL )) { +enum dma_resv_usage usage =DMA_RESV_USAGE_KERNEL ; + +if (vops->flags &XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ) + usage =DMA_RESV_USAGE_BOOKKEEP ; + + err =job_test_add_deps (job,xe_vm_resv (vm), usage);if (err)return err; } - rtfence =xe_range_fence_tree_first (rftree, pt_update_ops->start, - pt_update_ops->last); + rtfence =xe_range_fence_tree_first (rftree, vops->start, vops->last);while (rtfence) { fence = rtfence->fence;@@ -1449,9 +1449,8 @@ static int xe_pt_vm_dependencies(structxe_sched_job *job, return err; } - rtfence =xe_range_fence_tree_next (rtfence, - pt_update_ops->start, - pt_update_ops->last); + rtfence =xe_range_fence_tree_next (rtfence, vops->start, + vops->last); }drivers/gpu/drm/xe/xe_pt.c +4
-7list_for_each_entry (op, &vops->list, link) {@@ -1464,14 +1463,11 @@ static int xe_pt_vm_dependencies(structxe_sched_job *job, err =xe_sync_entry_add_deps (&vops->syncs[i], job);if (job) { -if (ijob) { - err =xe_tlb_inval_job_alloc_dep (ijob); -if (err) -return err; - } +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +if (!ijobs[i]) +continue ; -if (mjob) { - err =xe_tlb_inval_job_alloc_dep (mjob); + err =xe_tlb_inval_job_alloc_dep (ijobs[i]);if (err)return err;drivers/gpu/drm/xe/xe_pt.c +4
-7}@@ -1480,17 +1476,14 @@ static int xe_pt_vm_dependencies(structxe_sched_job *job, return err; } -static int xe_pt_pre_commit (struct xe_migrate_pt_update *pt_update) +static int xe_pt_pre_commit (struct xe_cpu_bind_pt_update *pt_update) {struct xe_vma_ops *vops = pt_update->vops;struct xe_vm *vm = vops->vm; -struct xe_range_fence_tree *rftree = &vm->rftree[pt_update->tile_id]; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[pt_update->tile_id]; +struct xe_range_fence_tree *rftree = &vm->rftree; -return xe_pt_vm_dependencies (pt_update->job, pt_update->ijob, - pt_update->mjob, vm, pt_update->vops, - pt_update_ops, rftree); +return xe_pt_vm_dependencies (pt_update->job, pt_update->ijobs, + vm, vops, rftree); }#if
IS_ENABLED(CONFIG_DRM_GPUSVM)drivers/gpu/drm/xe/xe_pt.c +1
-2@@ -1550,8 +1543,7 @@ static bool xe_pt_userptr_inject_eagain(struct xe_userptr_vma *uvma)#endif -static int vma_check_userptr (struct xe_vm *vm,struct xe_vma *vma, -struct xe_vm_pgtable_update_ops *pt_update) +static int vma_check_userptr (struct xe_vm *vm,struct xe_vma *vma) {struct xe_userptr_vma *uvma;drivers/gpu/drm/xe/xe_pt.c +1
-2unsigned long notifier_seq;@@ -1581,8 +1573,7 @@ static int vma_check_userptr(struct xe_vm *vm,struct xe_vma *vma, return 0; } -static int op_check_svm_userptr (struct xe_vm *vm,struct xe_vma_op *op, -struct xe_vm_pgtable_update_ops *pt_update) +static int op_check_svm_userptr (struct xe_vm *vm,struct xe_vma_op *op) {drivers/gpu/drm/xe/xe_pt.c +3
-3int err =0 ;@@ -1593,13 +1584,13 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op, if (!op->map.immediate && xe_vm_in_fault_mode(vm)) break; - err =vma_check_userptr (vm, op->map.vma, pt_update); + err =vma_check_userptr (vm, op->map.vma);break ;case DRM_GPUVA_OP_REMAP :if (op->remap.prev && !op->remap.skip_prev) - err =vma_check_userptr (vm, op->remap.prev, pt_update); + err =vma_check_userptr (vm, op->remap.prev);if (!err && op->remap.next && !op->remap.skip_next) - err =vma_check_userptr (vm, op->remap.next, pt_update); + err =vma_check_userptr (vm, op->remap.next);drivers/gpu/drm/xe/xe_pt.c +1
-1break ;case DRM_GPUVA_OP_UNMAP :break ;@@ -1619,7 +1610,7 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op, } } }else { - err =vma_check_userptr (vm,gpuva_to_vma (op- >base.prefetch.va), pt_update); + err =vma_check_userptr (vm,gpuva_to_vma (op- >base.prefetch.va));drivers/gpu/drm/xe/xe_pt.c +1
-3}break ;#if IS_ENABLED (CONFIG_DRM_XE_GPUSVM )@@ -1645,12 +1636,10 @@ static int op_check_svm_userptr(struct xe_vm *vm, struct xe_vma_op *op, return err; } -static int xe_pt_svm_userptr_pre_commit (struct xe_migrate_pt_update *pt_update) +static int xe_pt_svm_userptr_pre_commit (struct xe_cpu_bind_pt_update *pt_update) {struct xe_vm *vm = pt_update->vops->vm;struct xe_vma_ops *vops = pt_update->vops; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[pt_update->tile_id];drivers/gpu/drm/xe/xe_pt.c +1
-1struct xe_vma_op *op;int err;@@ -1661,7 +1650,7 @@ static int xe_pt_svm_userptr_pre_commit(struct xe_migrate_pt_update *pt_update)xe_pt_svm_userptr_notifier_lock (vm );list_for_each_entry (op, &vops->list, link) { - err =op_check_svm_userptr (vm, op, pt_update_ops); + err =op_check_svm_userptr (vm, op);if (err) {xe_pt_svm_userptr_notifier_unlock (vm);drivers/gpu/drm/xe/xe_pt.c +4
-4break ;@@ -2008,10 +1997,10 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, }static void -xe_migrate_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, -struct iosys_map *map,u32 qword_ofs, -u32 num_qwords, -const struct xe_vm_pgtable_update *update) +xe_pt_clear_pgtable_callback (struct xe_vm *vm,struct xe_tile *tile, +struct iosys_map *map,u32 qword_ofs, +u32 num_qwords, +const struct xe_vm_pgtable_update *update) {drivers/gpu/drm/xe/xe_pt.c +4
-0u64 empty =__xe_pt_empty_pte (tile, vm, update->level);int i;@@ -2089,6 +2078,9 @@ to_pt_op(struct xe_vm_pgtable_update_ops *pt_update_ops, u32 op_idx)static u32 get_current_op (struct xe_vm_pgtable_update_ops *pt_update_ops) { +if (!pt_update_ops->pt_job_ops) +return 0 ; +return pt_update_ops->pt_job_ops->current_op; }@@ -2378,6 +2370,7 @@ static int unbind_range_prepare(struct xe_vm *vm, static int op_prepare(struct xe_vm *vm,struct xe_tile *tile, +struct xe_vma_ops *vops,struct xe_vm_pgtable_update_ops *pt_update_ops,struct xe_vma_op *op){
drivers/gpu/drm/xe/xe_pt.c +1
-1@@ -2394,7 +2387,7 @@ static int op_prepare(struct xe_vm *vm, err =bind_op_prepare (vm, tile, pt_update_ops, op- >map.vma, op->map.invalidate_on_bind); - pt_update_ops->wait_vm_kernel = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_KERNEL ;break ;case DRM_GPUVA_OP_REMAP :drivers/gpu/drm/xe/xe_pt.c +2
-2{@@ -2408,12 +2401,12 @@ static int op_prepare(struct xe_vm *vm,if (!err && op->remap.prev && !op->remap.skip_prev) { err =bind_op_prepare (vm, tile, pt_update_ops, op->remap.prev, false); - pt_update_ops->wait_vm_bookkeep = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; }if (!err && op->remap.next && !op->remap.skip_next) { err =bind_op_prepare (vm, tile, pt_update_ops, op->remap.next, false); - pt_update_ops->wait_vm_bookkeep = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; }drivers/gpu/drm/xe/xe_pt.c +1
-1break ; }@@ -2449,7 +2442,7 @@ static int op_prepare(struct xe_vm *vm, } }else { err =bind_op_prepare (vm, tile, pt_update_ops, vma, false); - pt_update_ops->wait_vm_kernel = true; + vops->flags |=XE_VMA_OPS_FLAG_WAIT_VM_KERNEL ; }drivers/gpu/drm/xe/xe_pt.c +2
-12break ; }@@ -2483,18 +2476,8 @@ xe_pt_update_ops_init(struct xe_vm_pgtable_update_ops *pt_update_ops) xe_page_reclaim_list_init(&pt_update_ops->prl); } -/** -* xe_pt_update_ops_prepare() - Prepare PT update operations -* @tile: Tile of PT update operations -* @vops: VMA operationa -* -* Prepare PT update operations which includes updating internal PT state, -* allocate memory for page tables, populate page table being pruned in, and -* create PT update operations for leaf insertion / removal. -* -* Return: 0 on success, negative error code on error. -*/ -int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops) +static int __xe_pt_update_ops_prepare (struct xe_tile *tile, +struct xe_vma_ops *vops) { struct xe_vm_pgtable_update_ops *pt_update_ops =drivers/gpu/drm/xe/xe_pt.c +1
-1&vops->pt_update_ops[tile->id];@@ -2513,7 +2496,7 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)return err;list_for_each_entry (op, &vops->list, link) { - err =op_prepare (vops->vm, tile, pt_update_ops, op); + err =op_prepare (vops->vm, tile, vops, pt_update_ops, op);if (err)drivers/gpu/drm/xe/xe_pt.c +10
-0return err;@@ -2522,6 +2505,16 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops) xe_tile_assert(tile, get_current_op(pt_update_ops) <= pt_update_ops->num_ops); +/* Propagate individual tile state up to VMA operation */ +if (pt_update_ops->start < vops->start) + vops->start = pt_update_ops->start; +if (pt_update_ops->last > vops->last) + vops->last = pt_update_ops->last; +if (pt_update_ops->needs_invalidation) + vops->flags |=XE_VMA_OPS_FLAG_NEEDS_INVALIDATION ; +if (pt_update_ops->needs_svm_lock) + vops->flags |=XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK ; +#ifdef TEST_VM_OPS_ERROR if (vops->inject_error &&
vops->vm->xe->vm_inject_error_position==FORCE_OP_ERROR_PREPARE)drivers/gpu/drm/xe/xe_pt.c +0
-0@@ -2530,35 +2523,68 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)return 0 ; }
drivers/gpu/drm/xe/xe_pt.c int xe_pt_update_ops_prepare(struct xe_tile · line 2531
Quoted 10 lines
drivers/gpu/drm/xe/xe_pt.c +4
-5-ALLOW_ERROR_INJECTION (xe_pt_update_ops_prepare,ERRNO ); -static void bind_op_commit (struct xe_vm *vm,struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma *vma,struct dma_fence *fence, -struct dma_fence *fence2,bool invalidate_on_bind) +/** +* xe_pt_update_ops_prepare() - Prepare PT update operations +* @xe: xe device. +* @vops: VMA operationa
s/operationa/operations
Quoted 1,075 lines
drivers/gpu/drm/xe/xe_pt.c +14
-9+* +* Prepare PT update operations which includes updating internal PT state, +* allocate memory for page tables, populate page table being pruned in, and +* create PT update operations for leaf insertion / removal. +* +* Return: 0 on success, negative error code on error. +*/ +int xe_pt_update_ops_prepare (struct xe_device *xe,struct xe_vma_ops *vops) { -xe_tile_assert (tile, !xe_vma_is_cpu_addr_mirror (vma)); +struct xe_tile *tile; +int id, err; + +for_each_tile (tile, xe, id) { +if (!vops->pt_update_ops[id].num_ops) +continue ; -if (!xe_vma_has_no_bo (vma) && !xe_vma_bo (vma)->vm) { -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); -if (fence2) - dma_resv_add_fence(xe_vma_bo(vma)- >ttm.base.resv, fence2, - pt_update_ops-Quoted 1 line
wait_vm_bookkeep?drivers/gpu/drm/xe/xe_pt.c +19
-2- DMA_RESV_USAGE_KERNEL : - DMA_RESV_USAGE_BOOKKEEP); + err =__xe_pt_update_ops_prepare (tile, vops); +if (err) +return err; } + +return 0 ; +} +ALLOW_ERROR_INJECTION (xe_pt_update_ops_prepare,ERRNO ); + +static void vma_add_fences (struct xe_vma *vma,struct dma_fence **fences, +int fence_count,enum dma_resv_usage usage) +{ +int i; + +if (xe_vma_has_no_bo (vma) ||xe_vma_bo (vma)->vm) +return ; + +for (i =0 ; i < fence_count; ++i) +if (fences[i]) + dma_resv_add_fence(xe_vma_bo(vma)-Quoted 1 line
ttm.base.resv,
drivers/gpu/drm/xe/xe_pt.c +12
-0+ fences[i], usage); +} + +static void bind_op_commit (struct xe_vm *vm,struct xe_vma *vma, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask, +bool invalidate_on_bind) +{ +xe_assert (vm->xe, !xe_vma_is_cpu_addr_mirror (vma)); + +vma_add_fences (vma, fences, fence_count, usage); +/* All WRITE_ONCE pair with READ_ONCE in
xe_vm_has_valid_gpu_mapping()*/drivers/gpu/drm/xe/xe_pt.c +9
-18-WRITE_ONCE (vma->tile_present, vma->tile_present |BIT (tile->id)); +WRITE_ONCE (vma->tile_present, vma->tile_present | tile_mask);if (invalidate_on_bind)WRITE_ONCE (vma->tile_invalidated, - vma->tile_invalidated |BIT (tile->id)); + vma->tile_invalidated | tile_mask);else WRITE_ONCE (vma->tile_invalidated, - vma->tile_invalidated & ~BIT (tile->id)); - vma->tile_staged &= ~BIT (tile->id); + vma->tile_invalidated & ~tile_mask); + vma->tile_staged &= ~tile_mask;if (xe_vma_is_userptr (vma)) {xe_svm_assert_held_read_or_inject_write (vm);to_userptr_vma (vma)->userptr.initial_bind = true;@@ -2568,31 +2594,21 @@ static void bind_op_commit(struct xe_vm *vm,struct xe_tile *tile, * Kick rebind worker if this bind triggers preempt fences and not in * the rebind worker */ -if (pt_update_ops->wait_vm_bookkeep && +if (usage ==DMA_RESV_USAGE_KERNEL &&xe_vm_in_preempt_fence_mode (vm) && !current->mm)xe_vm_queue_rebind_worker (vm); } -static void unbind_op_commit (struct xe_vm *vm,struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma *vma,struct dma_fence *fence, -struct dma_fence *fence2) +static void unbind_op_commit(struct xe_vm *vm, struct xe_vma *vma, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask) { -xe_tile_assert (tile, !xe_vma_is_cpu_addr_mirror (vma)); +xe_assert (vm->xe, !xe_vma_is_cpu_addr_mirror (vma)); -if (!xe_vma_has_no_bo (vma) && !xe_vma_bo (vma)->vm) { -dma_resv_add_fence (xe_vma_bo (vma)->ttm.base.resv, fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); -if (fence2) - dma_resv_add_fence(xe_vma_bo(vma)- >ttm.base.resv, fence2, - pt_update_ops-Quoted 1 line
wait_vm_bookkeep?drivers/gpu/drm/xe/xe_pt.c +4
-5- DMA_RESV_USAGE_KERNEL : - DMA_RESV_USAGE_BOOKKEEP); - } - vma->tile_present &= ~BIT (tile->id); +vma_add_fences (vma, fences, fence_count, usage); + + vma->tile_present &= ~tile_mask;if (!vma->tile_present) {list_del_init (&vma->combined_links.rebind);if (xe_vma_is_userptr (vma)) {@@ -2607,21 +2623,19 @@ static void unbind_op_commit(struct xe_vm *vm, struct xe_tile *tile, static void range_present_and_invalidated_tile(struct xe_vm *vm,struct xe_svm_range *range, -u8 tile_id) +u8 tile_mask) {/* All WRITE_ONCE pair with READ_ONCE in
xe_vm_has_valid_gpu_mapping()*/drivers/gpu/drm/xe/xe_pt.c +0
-2lockdep_assert_held (&vm->svm.gpusvm.notifier_lock); -WRITE_ONCE (range->tile_present, range->tile_present |BIT (tile_id)); - WRITE_ONCE(range->tile_invalidated, range->tile_invalidated &~BIT(tile_id));
drivers/gpu/drm/xe/xe_pt.c +2
-0+WRITE_ONCE (range->tile_present, range->tile_present | tile_mask); + WRITE_ONCE(range->tile_invalidated, range->tile_invalidated &~tile_mask);
}drivers/gpu/drm/xe/xe_pt.c +5
-7-static void op_commit (struct xe_vm *vm, -struct xe_tile *tile, -struct xe_vm_pgtable_update_ops *pt_update_ops, -struct xe_vma_op *op,struct dma_fence *fence, -struct dma_fence *fence2) +static void op_commit (struct xe_vm *vm,struct xe_vma_op *op, +struct dma_fence **fences,int fence_count, +enum dma_resv_usage usage,u8 tile_mask) {xe_vm_assert_held (vm);@@ -2631,8 +2645,8 @@ static void op_commit(struct xe_vm *vm, (op->map.vma_flags & XE_VMA_SYSTEM_ALLOCATOR)) break; -bind_op_commit (vm, tile, pt_update_ops, op->map.vma, fence, - fence2, op->map.invalidate_on_bind); +bind_op_commit (vm, op->map.vma, fences, fence_count, usage, + tile_mask, op->map.invalidate_on_bind);break ;drivers/gpu/drm/xe/xe_pt.c +5
-5case DRM_GPUVA_OP_REMAP : {@@ -2641,14 +2655,15 @@ static void op_commit(struct xe_vm *vm,if (xe_vma_is_cpu_addr_mirror (old))break ; -unbind_op_commit (vm, tile, pt_update_ops, old, fence, fence2); +unbind_op_commit (vm, old, fences, fence_count, usage, + tile_mask);if (op->remap.prev && !op->remap.skip_prev) -bind_op_commit (vm, tile, pt_update_ops, op- >remap.prev, - fence, fence2, false); +bind_op_commit (vm, op->remap.prev, fences, fence_count, + usage, tile_mask, false);if (op->remap.next && !op->remap.skip_next) -bind_op_commit (vm, tile, pt_update_ops, op- >remap.next, - fence, fence2, false); + bind_op_commit(vm, op->remap.next, fences,
fence_count,drivers/gpu/drm/xe/xe_pt.c +3
-2+ usage, tile_mask, false);break ; }case DRM_GPUVA_OP_UNMAP :@@ -2656,8 +2671,8 @@ static void op_commit(struct xe_vm *vm,struct xe_vma *vma =gpuva_to_vma (op->base.unmap.va);if (!xe_vma_is_cpu_addr_mirror (vma)) -unbind_op_commit (vm, tile, pt_update_ops, vma, fence, - fence2); +unbind_op_commit (vm, vma, fences, fence_count, + usage, tile_mask);break ; }drivers/gpu/drm/xe/xe_pt.c +3
-3case DRM_GPUVA_OP_PREFETCH :@@ -2669,10 +2684,11 @@ static void op_commit(struct xe_vm *vm,unsigned long i;xa_for_each (&op->prefetch_range.range, i, range) -range_present_and_invalidated_tile (vm, range, tile->id); +range_present_and_invalidated_tile (vm, range, + tile_mask); }else { -bind_op_commit (vm, tile, pt_update_ops, vma, fence, - fence2, false); + bind_op_commit(vm, vma, fences, fence_count,usage,
drivers/gpu/drm/xe/xe_pt.c +1
-0+ tile_mask, false); }break ; }@@ -2680,11 +2696,12 @@ static void op_commit(struct xe_vm *vm, {/* WRITE_ONCE pairs with READ_ONCE in
xe_vm_has_valid_gpu_mapping()*/drivers/gpu/drm/xe/xe_pt.c +0
-1if (op->subop ==XE_VMA_SUBOP_MAP_RANGE ) - range_present_and_invalidated_tile(vm, op-Quoted 1 line
map_range.range,
tile->id);drivers/gpu/drm/xe/xe_pt.c +1
-0+ range_present_and_invalidated_tile(vm, op-Quoted 1 line
map_range.range,
drivers/gpu/drm/xe/xe_pt.c +8
-22+ tile_mask); else if (op->subop == XE_VMA_SUBOP_UNMAP_RANGE) WRITE_ONCE(op->unmap_range.range->tile_present, op->unmap_range.range->tile_present & - ~BIT (tile->id)); + ~tile_mask);break ; }@@ -2693,40 +2710,25 @@ static void op_commit(struct xe_vm *vm, } } -static const struct xe_migrate_pt_update_ops migrate_ops = { +static const struct xe_cpu_bind_pt_update_ops cpu_bind_ops = { .populate = xe_vm_populate_pgtable, - .clear = xe_migrate_clear_pgtable_callback, + .clear = xe_pt_clear_pgtable_callback, .pre_commit = xe_pt_pre_commit, };#if IS_ENABLED (CONFIG_DRM_GPUSVM ) -static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops = { +static const struct xe_cpu_bind_pt_update_ops svm_userptr_cpu_bind_ops = { .populate = xe_vm_populate_pgtable, - .clear = xe_migrate_clear_pgtable_callback, + .clear = xe_pt_clear_pgtable_callback, .pre_commit = xe_pt_svm_userptr_pre_commit, };#else -static const struct xe_migrate_pt_update_ops svm_userptr_migrate_ops; +static const struct xe_cpu_bind_pt_update_ops svm_userptr_cpu_bind_ops;#endif -static struct xe_dep_scheduler *to_dep_scheduler (struct xe_exec_queue *q, -struct xe_tile *tile, -struct xe_gt *gt, -unsigned int *type) -{ -int tile_ofs = tile->id * (XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT +1 ); - -if (xe_gt_is_media_type (gt)) - *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_MEDIA_GT ; -else - *type = tile_ofs +XE_EXEC_QUEUE_TLB_INVAL_PRIMARY_GT ; - -return q->tlb_inval[*type].dep_scheduler; -} -/** * xe_pt_update_ops_run() - Run PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operationa * * Run PT update operations which includes committing internal PT state
changes,drivers/gpu/drm/xe/xe_pt.c +23
-32@@ -2736,82 +2738,83 @@ static struct xe_dep_scheduler*to_dep_scheduler(struct xe_exec_queue *q, * Return: fence on success, negative ERR_PTR on error. */ struct dma_fence * -xe_pt_update_ops_run (struct xe_tile *tile,struct xe_vma_ops *vops) +xe_pt_update_ops_run (struct xe_device *xe,struct xe_vma_ops *vops) {struct xe_vm *vm = vops->vm; -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; -struct xe_exec_queue *q = pt_update_ops->q; -struct dma_fence *fence, *ifence =NULL , *mfence =NULL ; -struct xe_tlb_inval_job *ijob =NULL , *mjob =NULL ; +struct xe_exec_queue *q = vops->q; +struct dma_fence *fence; +struct dma_fence *ifences[XE_CPU_BIND_INVAL_JOB_COUNT ] = {};struct xe_range_fence *rfence; +enum dma_resv_usage usage =DMA_RESV_USAGE_BOOKKEEP ;struct xe_vma_op *op; -unsigned int type; -int err =0 , i; -struct xe_migrate_pt_update update = { - .ops = pt_update_ops->needs_svm_lock ? - &svm_userptr_migrate_ops : - &migrate_ops, +struct xe_tile *tile; +int err =0 , total_ops =0 , i, j; +u8 tile_mask =0 ; +bool needs_invalidation = vops->flags & +XE_VMA_OPS_FLAG_NEEDS_INVALIDATION ; +bool needs_svm_lock = vops->flags & +XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK ; +struct xe_cpu_bind_pt_update update = { + .ops = needs_svm_lock ? &svm_userptr_cpu_bind_ops : + &cpu_bind_ops, .vops = vops, - .tile_id = tile->id, };lockdep_assert_held (&vm->lock);xe_vm_assert_held (vm); -if (!get_current_op (pt_update_ops)) { -xe_tile_assert (tile,xe_vm_in_fault_mode (vm)); +for_each_tile (tile, xe, j) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[j]; + total_ops +=get_current_op (pt_update_ops); + } +if (!total_ops) { +xe_assert (xe,xe_vm_in_fault_mode (vm));return dma_fence_get_stub (); }#ifdef TEST_VM_OPS_ERROR if (vops->inject_error && - vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_RUN ) + xe->vm_inject_error_position ==FORCE_OP_ERROR_RUN )return ERR_PTR (-ENOSPC );#endif -if (pt_update_ops->needs_invalidation) { -struct xe_dep_scheduler *dep_scheduler = -to_dep_scheduler (q, tile, tile->primary_gt, &type); - - ijob =xe_tlb_inval_job_create (q, &tile->primary_gt->tlb_inval, - dep_scheduler, vm, - pt_update_ops->start, - pt_update_ops->last, - type); -if (IS_ERR (ijob)) { - err =PTR_ERR (ijob); - goto kill_vm_tile1; - } - update.ijob = ijob; -/* -* Only add page reclaim for the primary GT. Media GT does not have
drivers/gpu/drm/xe/xe_pt.c +0
-6-* any PPC to flush, so enabling the PPC flush bit for media is -* effectively a NOP and provides no performance benefit nor -* interfere with primary GT. -*/ -if (xe_page_reclaim_list_valid (&pt_update_ops->prl)) { - xe_tlb_inval_job_add_page_reclaim(ijob,
&pt_update_ops->prl);drivers/gpu/drm/xe/xe_pt.c +0
-2-/* Release ref from alloc, job will now handle it */ - xe_page_reclaim_list_invalidate(&pt_update_ops-Quoted 1 line
prl);
drivers/gpu/drm/xe/xe_pt.c +0
-5- } - -if (tile->media_gt) { - dep_scheduler = to_dep_scheduler(q, tile, - tile->media_gt,&type);
drivers/gpu/drm/xe/xe_pt.c +0
-3- - mjob = xe_tlb_inval_job_create(q, - &tile->media_gt-Quoted 1 line
tlb_inval,drivers/gpu/drm/xe/xe_pt.c +9
-6- dep_scheduler, vm, - pt_update_ops->start, - pt_update_ops->last, - type); -if (IS_ERR (mjob)) { - err =PTR_ERR (mjob); +if (needs_invalidation) { +for_each_tlb_inval (q, i) { +struct xe_dep_scheduler *dep_scheduler = + q->tlb_inval[i].dep_scheduler; +struct xe_tile *tile = + &xe->tiles[i /XE_MAX_GT_PER_TILE ]; +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[tile->id]; + struct xe_page_reclaim_list *prl = &pt_update_ops-Quoted 1 line
prl;
drivers/gpu/drm/xe/xe_pt.c +18
-1+struct xe_tlb_inval_job *ijob; +struct xe_gt *gt = i %XE_MAX_GT_PER_TILE ? + tile->media_gt : tile->primary_gt; + + ijob =xe_tlb_inval_job_create (q, >->tlb_inval, + dep_scheduler, + vm, vops->start, + vops->last, i); +if (IS_ERR (ijob)) { + err =PTR_ERR (ijob); goto free_ijob; } - update.mjob = mjob; + + update.ijobs[i] = ijob; + +/* +* Only add page reclaim for the primary GT. Media GT +* does not have any PPC to flush, so enabling the PPC +* flush bit for media is effectively a NOP and provides +* no performance benefit nor interfere with primary GT.
drivers/gpu/drm/xe/xe_pt.c +4
-0+*/ +if (xe_page_reclaim_list_valid (prl)) { +xe_tlb_inval_job_add_page_reclaim (ijob, prl); +/* Release ref from alloc, job will now handle it */
drivers/gpu/drm/xe/xe_pt.c +15
-9+xe_page_reclaim_list_invalidate (prl); + } } }@@ -2821,67 +2824,61 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) goto free_ijob; } - fence =xe_migrate_update_pgtables (tile->migrate, &update); + fence =xe_cpu_bind_update_pgtables (xe->cpu_bind, &update);if (IS_ERR (fence)) { err =PTR_ERR (fence); goto free_rfence; }/* Point of no return - VM killed if failure after this */ -for (i =0 ; i <get_current_op (pt_update_ops); ++i) { -struct xe_vm_pgtable_update_op *pt_op = -to_pt_op (pt_update_ops, i); - -xe_pt_commit (pt_op->vma, pt_op->entries, - pt_op->num_entries, - &pt_update_ops->pt_job_ops->deferred); - pt_op->vma =NULL ;/* skip in xe_pt_update_ops_abort */ +for_each_tile (tile, xe, j) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[j]; + +for (i =0 ; i <get_current_op (pt_update_ops); ++i) { +struct xe_vm_pgtable_update_op *pt_op = +to_pt_op (pt_update_ops, i); + +xe_pt_commit (pt_op->vma, pt_op->entries, + pt_op->num_entries, + &pt_update_ops->pt_job_ops->deferred); + pt_op->vma =NULL ;/* skip in
xe_pt_update_ops_abort*/drivers/gpu/drm/xe/xe_pt.c +6
-15+ tile_mask |=BIT (tile->id); + } } -if (xe_range_fence_insert (&vm->rftree[tile->id], rfence, +if (xe_range_fence_insert (&vm->rftree, rfence, &xe_range_fence_kfree_ops, - pt_update_ops->start, - pt_update_ops->last, fence)) + vops->start, vops->last, fence))dma_fence_wait (fence, false); -if (ijob) - ifence =xe_tlb_inval_job_push (ijob, tile->migrate, fence); -if (mjob) - mfence =xe_tlb_inval_job_push (mjob, tile->migrate, fence); +if (vops->flags &XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ) + usage =DMA_RESV_USAGE_KERNEL ; -if (!mjob && !ijob) { -dma_resv_add_fence (xe_vm_resv (vm), fence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); - -list_for_each_entry (op, &vops->list, link) - op_commit(vops->vm, tile, pt_update_ops, op, fence,NULL);
drivers/gpu/drm/xe/xe_pt.c +2
-6- }else if (ijob && !mjob) { -dma_resv_add_fence (xe_vm_resv (vm), ifence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); +if (!needs_invalidation) { +dma_resv_add_fence (xe_vm_resv (vm), fence, usage);list_for_each_entry (op, &vops->list, link) - op_commit(vops->vm, tile, pt_update_ops, op,ifence, NULL);
drivers/gpu/drm/xe/xe_pt.c +1
-0+ op_commit(vops->vm, op, &fence, 1, usage,
tile_mask);drivers/gpu/drm/xe/xe_pt.c +8
-8}else { -dma_resv_add_fence (xe_vm_resv (vm), ifence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +if (!update.ijobs[i]) +continue ; + + ifences[i] =xe_tlb_inval_job_push (update.ijobs[i], + fence); +xe_assert (xe, !IS_ERR_OR_NULL (ifences[i])); -dma_resv_add_fence (xe_vm_resv (vm), mfence, - pt_update_ops->wait_vm_bookkeep ? -DMA_RESV_USAGE_KERNEL : -DMA_RESV_USAGE_BOOKKEEP ); + dma_resv_add_fence(xe_vm_resv(vm), ifences[i],usage);
drivers/gpu/drm/xe/xe_pt.c +1
-1+ }list_for_each_entry (op, &vops->list, link) - op_commit(vops->vm, tile, pt_update_ops, op,ifence,
drivers/gpu/drm/xe/xe_pt.c +10
-11- mfence); +op_commit (vops->vm, op, ifences, +XE_CPU_BIND_INVAL_JOB_COUNT , usage, + tile_mask); } -if (pt_update_ops->needs_svm_lock) +if (needs_svm_lock)xe_pt_svm_userptr_notifier_unlock (vm);/* @@ -2891,21 +2888,18 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)if (!(q->flags &EXEC_QUEUE_FLAG_MIGRATE ))xe_exec_queue_last_fence_set (q, vm, fence); -xe_tlb_inval_job_put (mjob); -xe_tlb_inval_job_put (ijob); -dma_fence_put (ifence); -dma_fence_put (mfence); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) { +xe_tlb_inval_job_put (update.ijobs[i]); +dma_fence_put (ifences[i]); + }return fence; free_rfence:kfree (rfence); free_ijob: -xe_tlb_inval_job_put (mjob); -xe_tlb_inval_job_put (ijob); -kill_vm_tile1: -if (err != -EAGAIN && err != -ENODATA && tile->id) -xe_vm_kill (vops->vm, false); +for (i =0 ; i <XE_CPU_BIND_INVAL_JOB_COUNT ; ++i) +xe_tlb_inval_job_put (update.ijobs[i]);return ERR_PTR (err);}
drivers/gpu/drm/xe/xe_pt.c +36
-23@@ -2913,52 +2907,65 @@ALLOW_ERROR_INJECTION (xe_pt_update_ops_run,ERRNO );/** * xe_pt_update_ops_fini() - Finish PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operations * * Finish PT update operations by committing to destroy page table memory */ -void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops) +void xe_pt_update_ops_fini (struct xe_device *xe,struct xe_vma_ops *vops) { -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; +struct xe_tile *tile; +int id; + +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; -xe_page_reclaim_entries_put (pt_update_ops->prl.entries); +if (!pt_update_ops->num_ops) +continue ; + +xe_page_reclaim_entries_put (pt_update_ops->prl.entries); + } }/** * xe_pt_update_ops_abort() - Abort PT update operations -* @tile: Tile of PT update operations +* @xe: xe device. * @vops: VMA operationa * * Abort PT update operations by unwinding internal PT state */ -void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops) +void xe_pt_update_ops_abort (struct xe_device *xe,struct xe_vma_ops *vops) { -struct xe_vm_pgtable_update_ops *pt_update_ops = - &vops->pt_update_ops[tile->id]; -int i; +struct xe_tile *tile; +int id;lockdep_assert_held (&vops->vm->lock);xe_vm_assert_held (vops->vm); -for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) { -struct xe_vm_pgtable_update_op *pt_op = -to_pt_op (pt_update_ops, i); +for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; +int i; -if (!pt_op->vma || i >=get_current_op (pt_update_ops)) -continue ; +for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) { +struct xe_vm_pgtable_update_op *pt_op = +to_pt_op (pt_update_ops, i); -if (pt_op->bind) -xe_pt_abort_bind (pt_op->vma, pt_op->entries, - pt_op->num_entries, - pt_op->rebind); -else -xe_pt_abort_unbind (pt_op->vma, pt_op->entries, - pt_op->num_entries); +if (!pt_op->vma || i >=get_current_op (pt_update_ops)) +continue ; + +if (pt_op->bind) +xe_pt_abort_bind (pt_op->vma, pt_op- >entries, + pt_op->num_entries, + pt_op->rebind); +else +xe_pt_abort_unbind (pt_op->vma, pt_op- >entries, + pt_op->num_entries); + } } -xe_pt_update_ops_fini (tile, vops); +xe_pt_update_ops_fini (xe, vops); }/** drivers/gpu/drm/xe/xe_pt.h +4
-4@@ -39,11 +39,11 @@ void xe_pt_destroy(struct xe_pt *pt, u32 flags, struct llist_head *deferred);void xe_pt_clear (struct xe_device *xe,struct xe_pt *pt); -int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops); -struct dma_fence *xe_pt_update_ops_run (struct xe_tile *tile, +int xe_pt_update_ops_prepare (struct xe_device *xe,struct xe_vma_ops *vops); +struct dma_fence *xe_pt_update_ops_run (struct xe_device *xe,struct xe_vma_ops *vops); -void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops); -void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops); +void xe_pt_update_ops_fini (struct xe_device *xe,struct xe_vma_ops *vops); +void xe_pt_update_ops_abort (struct xe_device *xe,struct xe_vma_ops *vops);bool
xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma); bool xe_pt_zap_ptes_range(structxe_tile*tile, structxe_vm*vm,drivers/gpu/drm/xe/xe_pt_types.h +0
-11@@ -120,8 +120,6 @@ struct xe_pt_job_ops {struct xe_vm_pgtable_update_ops {/** @pt_job_ops: PT update operations dynamic allocation*/ struct xe_pt_job_ops *pt_job_ops; -/** @q: exec queue for PT operations */ -struct xe_exec_queue *q;/** @prl: embedded page reclaim list */ struct xe_page_reclaim_list prl;/** @start: start address of ops */ @@ -134,18 +132,6 @@ struct xe_vm_pgtable_update_ops {bool needs_svm_lock;/** @needs_invalidation: Needs invalidation */ bool needs_invalidation; -/** -* @wait_vm_bookkeep: PT operations need to wait until VM is idle -* (bookkeep dma-resv slots are idle) and stage all future VM activity -* behind these operations (install PT operations into VM kernel -* dma-resv slot). -*/ -bool wait_vm_bookkeep; -/** -* @wait_vm_kernel: PT operations need to wait until VM kernel dma- resv
drivers/gpu/drm/xe/xe_pt_types.h +0
-3-* slots are idle. -*/ - bool wait_vm_kernel; };#endif drivers/gpu/drm/xe/xe_sched_job.c +0
-1@@ -73,8 +73,9 @@ static void job_free(struct xe_sched_job *job)struct xe_exec_queue *q = job->q;bool is_migration =xe_sched_job_is_migration (q); - kmem_cache_free(xe_exec_queue_is_parallel(job->q) ||
is_migration?drivers/gpu/drm/xe/xe_sched_job.c +1
-1- xe_sched_job_parallel_slab : xe_sched_job_slab, job); + kmem_cache_free(job->is_pt_job || xe_exec_queue_is_parallel(job-Quoted 1 line
q) ||
drivers/gpu/drm/xe/xe_sched_job.c +5
-1+ is_migration ? xe_sched_job_parallel_slab : + xe_sched_job_slab, job); }static struct xe_device *job_to_xe (struct xe_sched_job *job)@@ -127,10 +128,12 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q,xe_assert (xe, batch_addr || q->flags & (EXEC_QUEUE_FLAG_VM |EXEC_QUEUE_FLAG_MIGRATE )); - job =job_alloc (xe_exec_queue_is_parallel (q) || is_migration); + job =job_alloc (!batch_addr ||xe_exec_queue_is_parallel (q) || + is_migration);if (!job)return ERR_PTR (-ENOMEM ); + job->is_pt_job = !batch_addr; job->q = q;drivers/gpu/drm/xe/xe_sched_job.c +0
-1job->sample_timestamp =U64_MAX ;kref_init (&job->refcount);@@ -143,7 +146,6 @@ struct xe_sched_job *xe_sched_job_create(struct xe_exec_queue *q, if (!batch_addr) { job->fence =dma_fence_get_stub (); - job->is_pt_job = true; } else {for (i =0 ; i < q->width; ++i) {drivers/gpu/drm/xe/xe_sched_job.c +0
-0struct dma_fence *fence =xe_lrc_alloc_seqno_fence ();drivers/gpu/drm/xe/xe_sched_job_types.h +5
-6@@ -14,7 +14,7 @@ struct dma_fence;struct dma_fence_chain ;struct xe_exec_queue ; -struct xe_migrate_pt_update_ops ; +struct xe_cpu_bind_pt_update_ops ;struct xe_pt_job_ops ;struct xe_tile ;struct xe_vm ;@@ -25,12 +25,11 @@ struct xe_vm;struct xe_pt_update_args {/** @vm: VM which is being bound */ struct xe_vm *vm; -/** @tile: Tile which page tables belong to */ -struct xe_tile *tile; -/** @ops: Migrate PT update ops */ -const struct xe_migrate_pt_update_ops *ops; +/** @ops: CPU bind PT update ops */ +const struct xe_cpu_bind_pt_update_ops *ops; +#define XE_PT_UPDATE_JOB_OPS_COUNT 2/** @pt_job_ops: PT job ops state */ - struct xe_pt_job_ops *pt_job_ops; +struct xe_pt_job_ops *pt_job_ops[XE_PT_UPDATE_JOB_OPS_COUNT ]; };/**
drivers/gpu/drm/xe/xe_tlb_inval_job.c +1
-2@@ -4,6 +4,7 @@*/ #include "xe_assert.h" +#include "xe_cpu_bind.h" #include "xe_dep_job_types.h" #include "xe_dep_scheduler.h" #include "xe_exec_queue.h" @@ -12,7 +13,6 @@#include "xe_page_reclaim.h" #include "xe_tlb_inval.h" #include "xe_tlb_inval_job.h" -#include "xe_migrate.h" #include "xe_pm.h" #include "xe_vm.h" @@ -218,7 +218,6 @@ int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job)/** * xe_tlb_inval_job_push() - TLB invalidation job push * @job: TLB invalidation job to push -* @m: The migration object being used * @fence: Dependency for TLB invalidation job * * Pushes a TLB invalidation job for execution, using @fence as a dependency.
drivers/gpu/drm/xe/xe_tlb_inval_job.c +1
-1@@ -230,11 +229,11 @@ int xe_tlb_inval_job_alloc_dep(structxe_tlb_inval_job *job) * Return: Job's finished fence on success, cannot fail */ struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence) {struct xe_tlb_inval_fence *ifence =container_of (job->fence,typeof (*ifence), base); +struct xe_cpu_bind *cpu_bind =gt_to_xe (job->q->gt)->cpu_bind;if (!dma_fence_is_signaled (fence)) {drivers/gpu/drm/xe/xe_tlb_inval_job.c +0
-2void *ptr;@@ -258,11 +257,11 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, job->fence_armed = true;/* -* We need the migration lock to protect the job's seqno and the spsc -* queue, only taken on migration queue, user queues protected dma- resv
drivers/gpu/drm/xe/xe_tlb_inval_job.c +2
-0+* We need the cpu_bind lock to protect the job's seqno and the spsc +* queue, only taken on cpu_bind queue, user queues protected dma- resv
drivers/gpu/drm/xe/xe_tlb_inval_job.c +2
-2* VM lock. */ -xe_migrate_job_lock (m, job->q); +xe_cpu_bind_job_lock (cpu_bind, job->q);/* Creation ref pairs with put in xe_tlb_inval_job_destroy */ xe_tlb_inval_fence_init (job->tlb_inval, ifence, false);@@ -281,7 +280,7 @@ struct dma_fence *xe_tlb_inval_job_push(struct xe_tlb_inval_job *job, &job->dep.drm.s_fence->finished, job->idx); -xe_migrate_job_unlock (m, job->q); +xe_cpu_bind_job_unlock (cpu_bind, job->q);/* * Not using
job->fence, as it has its own dma-fence context, which
doesdrivers/gpu/drm/xe/xe_tlb_inval_job.h +0
-2@@ -11,7 +11,6 @@struct dma_fence ;struct xe_dep_scheduler ;struct xe_exec_queue ; -struct xe_migrate ;struct xe_page_reclaim_list ;struct xe_tlb_inval ;struct xe_tlb_inval_job ;@@ -28,7 +27,6 @@ void xe_tlb_inval_job_add_page_reclaim(structxe_tlb_inval_job *job, int xe_tlb_inval_job_alloc_dep(struct xe_tlb_inval_job *job);struct dma_fence *xe_tlb_inval_job_push (struct xe_tlb_inval_job *job, -struct xe_migrate *m,struct dma_fence *fence);void
xe_tlb_inval_job_get(struct xe_tlb_inval_job *job);drivers/gpu/drm/xe/xe_vm.c +1
-2@@ -24,6 +24,7 @@#include "regs/xe_gtt_defs.h" #include "xe_assert.h" #include "xe_bo.h" +#include "xe_cpu_bind.h" #include "xe_device.h" #include "xe_drm_client.h" #include "xe_exec_queue.h" @@ -795,8 +796,6 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)struct xe_vma *vma, *next;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;drivers/gpu/drm/xe/xe_vm.c +3
-6lockdep_assert_held (&vm->lock);@@ -804,12 +803,9 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker) list_empty(&vm->rebind_list))return 0 ; -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe- >cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ;drivers/gpu/drm/xe/xe_vm.c +4
-9xe_vm_assert_held (vm);list_for_each_entry (vma, &vm->rebind_list, combined_links.rebind) {@@ -854,21 +850,16 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_mastruct dma_fence *fence =NULL ;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;lockdep_assert_held (&vm->lock);xe_vm_assert_held (vm);xe_assert (vm->xe,xe_vm_in_fault_mode (vm)); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); - vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ; -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe- >cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT | +XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ;drivers/gpu/drm/xe/xe_vm.c +0
-2err =xe_vm_ops_add_rebind (&vops, vma, tile_mask);if (err)@@ -944,8 +935,6 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, struct dma_fence *fence = NULL;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;drivers/gpu/drm/xe/xe_vm.c +4
-7lockdep_assert_held (&range->lock);@@ -954,13 +943,10 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm,xe_assert (vm->xe,xe_vm_in_fault_mode (vm));xe_assert (vm->xe,xe_vma_is_cpu_addr_mirror (vma)); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); - vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ; -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe- >cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT | +XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ;drivers/gpu/drm/xe/xe_vm.c +0
-2err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask);if (err)@@ -1027,8 +1013,6 @@ struct dma_fence *xe_vm_range_unbind(structxe_vm *vm, struct dma_fence *fence = NULL;struct xe_vma_ops vops;struct xe_vma_op *op, *next_op; -struct xe_tile *tile; -u8 id;int err;drivers/gpu/drm/xe/xe_vm.c +3
-6lockdep_assert_held (&range->lock);@@ -1039,12 +1023,9 @@ struct dma_fence *xe_vm_range_unbind(struct xe_vm *vm, if (!range->tile_present)return dma_fence_get_stub (); -xe_vma_ops_init (&vops, vm,NULL ,NULL ,0 ); -for_each_tile (tile, vm->xe, id) { - vops.pt_update_ops[id].wait_vm_bookkeep = true; - vops.pt_update_ops[tile->id].q = -xe_migrate_bind_queue (tile->migrate); - } +xe_vma_ops_init (&vops, vm,xe_cpu_bind_queue (vm->xe- >cpu_bind), +NULL ,0 ); + vops.flags |=XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ;drivers/gpu/drm/xe/xe_vm.c +2
-5err =xe_vm_ops_add_range_unbind (&vops, range);if (err)@@ -1715,9 +1696,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) init_rwsem(&vm->exec_queues.lock);xe_vm_init_prove_locking (xe, vm); - -for_each_tile (tile, xe, id) -xe_range_fence_tree_init (&vm->rftree[id]); +xe_range_fence_tree_init (&vm->rftree); vm->pt_ops = &xelp_pt_ops;@@ -1859,8 +1838,7 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)xe_svm_fini (vm ); err_no_resv:mutex_destroy (&vm->snap_mutex); -for_each_tile (tile, xe, id) -xe_range_fence_tree_fini (&vm->rftree[id]); +xe_range_fence_tree_fini (&vm->rftree);ttm_lru_bulk_move_fini (&xe->ttm, &vm->lru_bulk_move);if (vm->xef)drivers/gpu/drm/xe/xe_vm.c +1
-4xe_file_put (vm->xef);@@ -1916,10 +1894,8 @@ void xe_vm_close_and_put(struct xe_vm *vm) {LIST_HEAD (contested);struct xe_device *xe = vm->xe; -struct xe_tile *tile;struct xe_vma *vma, *next_vma;struct drm_gpuva *gpuva, *next; -u8 id;xe_assert (xe, !vm->preempt.num_exec_queues);@@ -2011,8 +1987,7 @@ void xe_vm_close_and_put(struct xe_vm *vm)xe_vm_clear_fault_entries (vm); -for_each_tile (tile, xe, id) -xe_range_fence_tree_fini (&vm->rftree[id]); +xe_range_fence_tree_fini (&vm->rftree);xe_vm_put (vm); }
drivers/gpu/drm/xe/xe_vm.c static void trace_xe_vm_ops_execute(struct · line 3487
Quoted 4 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0@@ -3509,23 +3484,16 @@ static void trace_xe_vm_ops_execute(struct xe_vma_ops *vops)static int vm_ops_setup_tile_args (struct xe_vm *vm,struct xe_vma_ops *vops)
This func does not setup tile args anymore. Maybe just rename it to "vm_ops_setup"?
Rest LGTM. With upper comments fixed:
Quoted 195 lines
drivers/gpu/drm/xe/xe_vm.c +3
-10{ -struct xe_exec_queue *q = vops->q;struct xe_tile *tile;int number_tiles =0 ;u8 id; -for_each_tile (tile, vm->xe, id) { +for_each_tile (tile, vm->xe, id)if (vops->pt_update_ops[id].num_ops) ++number_tiles; -if (vops->pt_update_ops[id].q) -continue ; - -if (q) - vops->pt_update_ops[id].q = q; -else - vops->pt_update_ops[id].q = vm->q; - } +if (!vops->q) + vops->q = vm->q;drivers/gpu/drm/xe/xe_vm.c +12
-41return number_tiles; }@@ -3533,22 +3501,17 @@ static int vm_ops_setup_tile_args(struct xe_vm *vm, struct xe_vma_ops *vops)static struct dma_fence *ops_execute (struct xe_vm *vm,struct xe_vma_ops *vops) { -struct xe_tile *tile; +struct xe_device *xe = vm->xe;struct dma_fence *fence =NULL ;struct dma_fence **fences =NULL ;struct dma_fence_array *cf =NULL ; -int number_tiles =0 , current_fence =0 , n_fence =0 , err, i; -u8 id; +int current_fence =0 , n_fence =1 , err, i; - number_tiles =vm_ops_setup_tile_args (vm, vops); -if (number_tiles ==0 ) +if (!vm_ops_setup_tile_args (vm, vops))return ERR_PTR (-ENODATA ); -for_each_tile (tile, vm->xe, id) - ++n_fence; -if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) { -for_each_tlb_inval (vops->pt_update_ops[0 ].q, i) +for_each_tlb_inval (vops->q, i) ++n_fence; }@@ -3564,69 +3527,39 @@ static struct dma_fence *ops_execute(structxe_vm *vm, goto err_out; } -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - - err =xe_pt_update_ops_prepare (tile, vops); -if (err) { - fence =ERR_PTR (err); - goto err_out; - } + err =xe_pt_update_ops_prepare (xe, vops); +if (err) { + fence =ERR_PTR (err); + goto err_out; }trace_xe_vm_ops_execute (vops); -for_each_tile (tile, vm->xe, id) { -struct xe_exec_queue *q = vops->pt_update_ops[tile->id].q; - - fence =NULL ; -if (!vops->pt_update_ops[id].num_ops) - goto collect_fences; - - fence =xe_pt_update_ops_run (tile, vops); -if (IS_ERR (fence)) - goto err_out; - -collect_fences: - fences[current_fence++] = fence ?:dma_fence_get_stub (); -if (vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT ) -continue ; - -xe_migrate_job_lock (tile->migrate, q); -for_each_tlb_inval (q, i) { -if (i >= (tile->id +1 ) *XE_MAX_GT_PER_TILE || - i < tile->id *XE_MAX_GT_PER_TILE ) -continue ; + fence =xe_pt_update_ops_run (xe, vops); +if (IS_ERR (fence)) + goto err_out; + fences[current_fence++] = fence; - fences[current_fence++] = fence ? - xe_exec_queue_tlb_inval_last_fence_get(q,vm, i) :
drivers/gpu/drm/xe/xe_vm.c +5
-3-dma_fence_get_stub (); - } -xe_migrate_job_unlock (tile->migrate, q); +if (!(vops->flags &XE_VMA_OPS_FLAG_SKIP_TLB_WAIT )) { +xe_cpu_bind_job_lock (xe->cpu_bind, vops->q); +for_each_tlb_inval (vops->q, i) + fences[current_fence++] = +xe_exec_queue_tlb_inval_last_fence_get(vops->q,drivers/gpu/drm/xe/xe_vm.c +3
-1+ vm, i); +xe_cpu_bind_job_unlock (xe->cpu_bind, vops->q); } -xe_assert (vm->xe, current_fence == n_fence); +xe_assert (xe, current_fence == n_fence); dma_fence_array_init(cf, n_fence, fences,
dma_fence_context_alloc(1),drivers/gpu/drm/xe/xe_vm.c +4
-121); fence = &cf->base; -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - -xe_pt_update_ops_fini (tile, vops); - } +xe_pt_update_ops_fini (xe, vops);return fence; err_out: -for_each_tile (tile, vm->xe, id) { -if (!vops->pt_update_ops[id].num_ops) -continue ; - -xe_pt_update_ops_abort (tile, vops); - } +xe_pt_update_ops_abort (xe, vops);while (current_fence)dma_fence_put (fences[--current_fence]);kfree (fences);@@ -3938,6 +3871,8 @@ static void xe_vma_ops_init(struct xe_vma_ops *vops, struct xe_vm *vm, vops->syncs = syncs; vops->num_syncs = num_syncs; vops->flags =0 ; + vops->start = ~0x0ull ; + vops->last =0x0ull ; }static int xe_vm_bind_ioctl_validate_bo(struct
xe_device*xe, structxe_bo
*bo,drivers/gpu/drm/xe/xe_vm_types.h +0
-0@@ -317,7 +317,7 @@ struct xe_vm {* @rftree: range fence tree to track updates to page table structure. * Used to implement conflict tracking between independent bind engines.
drivers/gpu/drm/xe/xe_vm_types.h +5
-1*/ -struct xe_range_fence_tree rftree[XE_MAX_TILES_PER_DEVICE ]; +struct xe_range_fence_tree rftree;const struct xe_pt_ops *pt_ops;@@ -557,6 +557,10 @@ struct xe_vma_ops {u32 num_syncs;/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ]; +/** @start: start address of ops */ +u64 start; +/** @last: last address of ops */ +u64 last;/** @flag: signify the properties within xe_vma_ops*/ #define XE_VMA_OPS_FLAG_HAS_SVM_PREFETCH BIT(0)#define
XE_VMA_OPS_FLAG_MADVISEBIT(1)drivers/gpu/drm/xe/xe_vm_types.h +4
-1@@ -566,6 +570,10 @@ struct xe_vma_ops {#define XE_VMA_OPS_FLAG_MODIFIES_GPUVA BIT(5)#define XE_VMA_OPS_FLAG_DOWNGRADE_LOCK BIT(6)#define XE_VMA_OPS_FLAG_HAS_SVM_VALID_RANGE BIT(7) +#define XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP BIT(8) +#define XE_VMA_OPS_FLAG_WAIT_VM_KERNEL BIT(9) +#define XE_VMA_OPS_FLAG_NEEDS_INVALIDATION BIT(10) +#define XE_VMA_OPS_FLAG_NEEDS_SVM_LOCK BIT(11)u32 flags;#ifdef TEST_VM_OPS_ERROR /** @inject_error: inject error to test error handling */ --2.34.1
Matthew Brost
maintainerPATCH v6 17/24 drm/xe: Add device flag to enable PT mirroring across tiles
In reply to Matthew BrostSome multi-tile devices may want to mirror page tables across tiles for memory-bandwidth reasons, while others may not. Add a device flag that allows enabling or disabling page-table mirroring across tiles.
Setting the flag to true (the existing behavior) on PVC, but both modes have been tested and are working on PVC.
--- drivers/gpu/drm/xe/xe_device_types.h | 2 ++ drivers/gpu/drm/xe/xe_migrate.c | 5 ++-- drivers/gpu/drm/xe/xe_pci.c | 2 ++ drivers/gpu/drm/xe/xe_pci_types.h | 3 ++- drivers/gpu/drm/xe/xe_pt.c | 36 +++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 37 +++++++++++++++++++++++++--- drivers/gpu/drm/xe/xe_vm.h | 3 +++ 7 files changed, 79 insertions(+), 9 deletions(-)
drivers/gpu/drm/xe/xe_device_types.h +2 -0
@@ -224,6 +224,8 @@ struct xe_device { u8 has_usm:1;/** @info.has_64bit_timestamp: Device supports 64-bit timestamps */ u8 has_64bit_timestamp:1; +/** @info.has_pt_mirror: Device has PT mirroring across tiles */ + u8 has_pt_mirror:1;/** @info.is_dgfx: is discrete device */ u8 is_dgfx:1;/** @info.needs_scratch: needs scratch page for oob prefetch to work */
drivers/gpu/drm/xe/xe_migrate.c +3 -2
@@ -246,7 +246,8 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,struct xe_device *xe =tile_to_xe (tile);u16 pat_index =xe_cache_pat_idx (xe,XE_CACHE_WB );u8 id = tile->id; -u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; +u32 num_entries =NUM_PT_SLOTS , num_level = +xe_vm_pt_root (vm, id)->level;#define VRAM_IDENTITY_MAP_PT_COUNT 4u32 num_setup = num_level +VRAM_IDENTITY_MAP_PT_COUNT ;#undef VRAM_IDENTITY_MAP_PT_COUNT
@@ -258,7 +259,7 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u64 l1_pt_ofs =xe_bo_size (bo) -5 *XE_PAGE_SIZE ; entry = vm->pt_ops->pde_encode_bo (bo, l1_pt_ofs); -xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); +xe_pt_write (xe, &xe_vm_pt_root (vm, id)->bo->vmap,0 , entry); map_ofs = (num_entries - num_setup) *XE_PAGE_SIZE ;
drivers/gpu/drm/xe/xe_pci.c +2 -0
@@ -371,6 +371,7 @@ static const __maybe_unused struct xe_device_desc pvc_desc = { .has_display = false, .has_drm_ras = true, .has_gsc_nvm = 1, + .has_pt_mirror = 1, .has_heci_gscfi = 1, .max_gt_per_tile = 1, .max_remote_tiles = 1,
@@ -808,6 +809,7 @@ static int xe_info_init_early(struct xe_device *xe, xe->info.has_mert = desc->has_mert; xe->info.has_page_reclaim_hw_assist = desc->has_page_reclaim_hw_assist; xe->info.has_pre_prod_wa = desc->has_pre_prod_wa; + xe->info.has_pt_mirror = desc->has_pt_mirror; xe->info.has_pxp = desc->has_pxp; xe->info.has_soc_remapper_sysctrl = desc->has_soc_remapper_sysctrl; xe->info.has_soc_remapper_telem = desc->has_soc_remapper_telem;
drivers/gpu/drm/xe/xe_pci_types.h +2 -1
@@ -52,8 +52,9 @@ struct xe_device_desc { u8 has_mbx_power_limits:1; u8 has_mbx_thermal_info:1; u8 has_mert:1; - u8 has_pre_prod_wa:1; u8 has_page_reclaim_hw_assist:1; + u8 has_pre_prod_wa:1; + u8 has_pt_mirror:1; u8 has_pxp:1; u8 has_soc_remapper_sysctrl:1; u8 has_soc_remapper_telem:1;
drivers/gpu/drm/xe/xe_pt.c +34 -2
@@ -812,7 +812,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, .wupd.entries = entries, .clear_pt = clear_pt, }; -struct xe_pt *pt = vm->pt_root[tile->id]; +struct xe_pt *pt =xe_vm_pt_root (vm, tile->id);int ret;bool is_purged = false;
@@ -1005,6 +1005,11 @@ static int xe_pt_zap_ptes_entry(struct xe_ptw *parent, pgoff_t offset,return 0 ; } +static bool pt_mirroring_disabled_for_tile (struct xe_vm *vm,u8 tile_id) +{ +return xe_vm_pt_root (vm, tile_id) != vm->pt_root[tile_id]; +} +static const struct xe_pt_walk_ops xe_pt_zap_ptes_ops = { .pt_entry = xe_pt_zap_ptes_entry, };
@@ -1046,6 +1051,9 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma)if (!(pt_mask &BIT (tile->id)))return false; +if (pt_mirroring_disabled_for_tile (xe_vma_vm (vma), tile->id)) +return true; + (void )xe_pt_walk_shared (&pt->base, pt->level,xe_vma_start (vma),xe_vma_end (vma), &xe_walk.base);
@@ -1098,6 +1106,9 @@ bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm,if (!(pt_mask &BIT (tile->id)))return false; +if (pt_mirroring_disabled_for_tile (vm, tile->id)) +return true; + (void )xe_pt_walk_shared (&pt->base, pt->level,xe_svm_range_start (range),xe_svm_range_end (range), &xe_walk.base);
@@ -1988,7 +1999,7 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, .wupd.entries = entries, .prl = pt_update_op->prl, }; -struct xe_pt *pt = vm->pt_root[tile->id]; +struct xe_pt *pt =xe_vm_pt_root (vm, tile->id); (void )xe_pt_walk_shared (&pt->base, pt->level, start, end, &xe_walk.base);
@@ -2541,9 +2552,21 @@ int xe_pt_update_ops_prepare(struct xe_device *xe, struct xe_vma_ops *vops)int id, err;for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; +if (!vops->pt_update_ops[id].num_ops)continue ; +if (pt_mirroring_disabled_for_tile (vops->vm, id)) { +struct xe_page_reclaim_list *prl = &pt_update_ops->prl; + +/* Transfer root PT update ops PRL to current */ + *prl = vops->pt_update_ops[0 ].prl; +xe_page_reclaim_entries_get (prl->entries); +continue ; + } + err =__xe_pt_update_ops_prepare (tile, vops);if (err)return err;
@@ -2835,6 +2858,12 @@ xe_pt_update_ops_run(struct xe_device *xe, struct xe_vma_ops *vops)struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[j]; +if (pt_mirroring_disabled_for_tile (vm, j)) { +xe_tile_assert (tile, !get_current_op (pt_update_ops)); + tile_mask |=BIT (tile->id); +continue ; + } +for (i =0 ; i <get_current_op (pt_update_ops); ++i) {struct xe_vm_pgtable_update_op *pt_op =to_pt_op (pt_update_ops, i);
@@ -2948,6 +2977,9 @@ void xe_pt_update_ops_abort(struct xe_device *xe, struct xe_vma_ops *vops) &vops->pt_update_ops[id];int i; +if (pt_mirroring_disabled_for_tile (vops->vm, id)) +continue ; +for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) {struct xe_vm_pgtable_update_op *pt_op =to_pt_op (pt_update_ops, i);
drivers/gpu/drm/xe/xe_vm.c +33 -4
@@ -845,6 +845,14 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)return err; } +static u8 adjust_rebind_tile_mask (struct xe_vm *vm,u8 tile_mask) +{ +if (vm->xe->info.has_pt_mirror) +return tile_mask; + +return (0x1 << vm->xe->info.tile_count) -1 ; +} +struct dma_fence *xe_vma_rebind (struct xe_vm *vm,struct xe_vma *vma,u8 tile_mask) {struct dma_fence *fence =NULL ;
@@ -861,7 +869,8 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_ma vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT |XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; - err =xe_vm_ops_add_rebind (&vops, vma, tile_mask); + err =xe_vm_ops_add_rebind (&vops, vma, +adjust_rebind_tile_mask (vm, tile_mask));if (err)return ERR_PTR (err);
@@ -948,7 +957,8 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT |XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; - err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask); + err =xe_vm_ops_add_range_rebind (&vops, vma, range, +adjust_rebind_tile_mask (vm, tile_mask));if (err)return ERR_PTR (err);
@@ -1738,7 +1748,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)for_each_tile (tile, xe, id) {if (flags &XE_VM_FLAG_MIGRATION && - tile->id !=XE_VM_FLAG_TILE_ID (flags)) + tile->id !=XE_VM_FLAG_TILE_ID (flags) && + (vm->xe->info.has_pt_mirror || id))continue ; vm->pt_root[id] = xe_pt_create(vm, tile, xe->info.vm_max_level,
@@ -2047,7 +2058,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) { -return vm->pt_ops->pde_encode_bo (vm->pt_root[tile->id]->bo,0 ); +return vm->pt_ops->pde_encode_bo (xe_vm_pt_root (vm, tile->id)->bo,0 ); } static struct xe_exec_queue *
@@ -5072,3 +5083,21 @@ void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q) }up_write (&vm->exec_queues.lock); } + +/** +* xe_vm_pt_root() - Retrieve VM page-table root +* @vm: The VM. +* @tile_id: Tile ID +* +* Retrieve VM page-table root for a tile ID, used to abstract if PT mirroring is +* enabled across tiles. +* +* Return: VM page-table root for a tile ID +*/ +struct xe_pt *xe_vm_pt_root (struct xe_vm *vm,u8 tile_id) +{ +if (vm->xe->info.has_pt_mirror) +return vm->pt_root[tile_id]; + +return vm->pt_root[0 ]; +}
drivers/gpu/drm/xe/xe_vm.h +3 -0
@@ -436,4 +436,7 @@ static inline struct drm_exec *xe_vm_validation_exec(struct xe_vm *vm) ((READ_ONCE(tile_present) & ~READ_ONCE(tile_invalidated)) & BIT((tile)->id))void xe_vma_mem_attr_copy (struct xe_vma_mem_attr *to,struct xe_vma_mem_attr *from); + +struct xe_pt *xe_vm_pt_root (struct xe_vm *vm,u8 tile_id); +#endif
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 17/24] drm/xe: Add device flag to enable PT mirroring across tiles
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Unnecessary allocation of page table roots and scratch structures for secondary tiles when PT…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unnecessary allocation of page table roots and scratch structures for secondary tiles when PT mirroring is disabled. --
commit 34b539e899a7643dc15dedad6a3cd229e0c0f4d5
Author: Matthew Brost <[email protected]>
drm/xe: Add device flag to enable PT mirroring across tiles
This commit introduces a device flag to toggle page-table mirroring across tiles for memory bandwidth considerations. It updates the VM creation, rebind, and page table update paths to check this flag and fall back to the primary tile's root page table when mirroring is disabled.
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0
[ ... ]
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) · line 1755
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1@@ -1738,7 +1748,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)for_each_tile (tile, xe, id) {if (flags &XE_VM_FLAG_MIGRATION && - tile->id !=XE_VM_FLAG_TILE_ID (flags)) + tile->id !=XE_VM_FLAG_TILE_ID (flags) && + (vm->xe->info.has_pt_mirror || id))continue ; vm->pt_root[id] = xe_pt_create(vm, tile, xe->info.vm_max_level,
[Severity: Medium]
Does this code unnecessarily allocate page table roots and scratch structures for secondary tiles when PT mirroring is disabled?
For standard user VMs, XE_VM_FLAG_MIGRATION is unset, meaning this if condition evaluates to false and we proceed to allocate pt_root[id] for all tiles.
However, when vm->xe->info.has_pt_mirror is false, xe_vm_pt_root() unconditionally returns pt_root[0]. This means the allocated pt_root[id] (and associated scratch page tables up to vm_max_level) for id > 0 will never be used, permanently wasting VRAM for the lifetime of every VM.
Should there be an explicit check to skip secondary tile allocations when has_pt_mirror is false, regardless of whether it is a migration VM?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Re: [PATCH v6 17/24] drm/xe: Add device flag to enable PT mirroring across tiles
Quoted 2 lines · On 05-09-2026 03:10, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unnecessary allocation of page table roots and scratch structures for secondary tiles when PT mirroring is disabled.
Looks valid.
Quoted 47 lines
--
commit
34b539e899a7643dc15dedad6a3cd229e0c0f4d5
Author: Matthew Brost <[email protected]>drm/xe: Add device flag to enable PT mirroring across tiles
This commit introduces a device flag to toggle page-table mirroring across tiles for memory bandwidth considerations. It updates the VM creation, rebind, and page table update paths to check this flag and fall back to the primary tile's root page table when mirroring is disabled.
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0[ ... ]
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) · line 1755
Quoted 10 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1@@ -1738,7 +1748,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)for_each_tile (tile, xe, id) {if (flags &XE_VM_FLAG_MIGRATION && - tile->id !=XE_VM_FLAG_TILE_ID (flags)) + tile->id !=XE_VM_FLAG_TILE_ID (flags) && + (vm->xe->info.has_pt_mirror || id))continue ; vm->pt_root[id] = xe_pt_create(vm, tile, xe->info.vm_max_level,[Severity: Medium]
Does this code unnecessarily allocate page table roots and scratch structures for secondary tiles when PT mirroring is disabled?For standard user VMs,
XE_VM_FLAG_MIGRATIONis unset, meaning this if condition evaluates to false and we proceed to allocate pt_root[id] for all tiles.However, when
vm->xe->info.has_pt_mirroris false,xe_vm_pt_root()unconditionally returns pt_root[0]. This means the allocated pt_root[id] (and associated scratch page tables up tovm_max_level) for id > 0 will never be used, permanently wasting VRAM for the lifetime of every VM.Should there be an explicit check to skip secondary tile allocations when
has_pt_mirroris false, regardless of whether it is a migration VM?
Matthew Brost
maintainerRe: [PATCH v6 17/24] drm/xe: Add device flag to enable PT mirroring across tiles
Quoted 8 lines · Himal Prasad
Quoted 2 lines · On 05-09-2026 03:10, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Unnecessary allocation of page table roots and scratch structures for secondary tiles when PT mirroring is disabled.Looks valid.
I'd didn't really want to over engineer this one and this wastes 4k of memory.
Matt
Quoted 46 lines
Quoted 45 lines
--
commit
34b539e899a7643dc15dedad6a3cd229e0c0f4d5
Author: Matthew Brost <[email protected]>drm/xe: Add device flag to enable PT mirroring across tiles
This commit introduces a device flag to toggle page-table mirroring across tiles for memory bandwidth considerations. It updates the VM creation, rebind, and page table update paths to check this flag and fall back to the primary tile's root page table when mirroring is disabled.
Quoted 3 lines
drivers/gpu/drm/xe/xe_vm.c +0
-0[ ... ]
drivers/gpu/drm/xe/xe_vm.c struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef) · line 1753
Quoted 8 lines
drivers/gpu/drm/xe/xe_vm.c +2
-1@@ -1738,7 +1748,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)for_each_tile (tile, xe, id) {if (flags &XE_VM_FLAG_MIGRATION && - tile->id !=XE_VM_FLAG_TILE_ID (flags)) + tile->id !=XE_VM_FLAG_TILE_ID (flags) && + (vm->xe->info.has_pt_mirror || id))continue ; vm->pt_root[id] = xe_pt_create(vm, tile, xe->info.vm_max_level,[Severity: Medium]
Does this code unnecessarily allocate page table roots and scratch structures for secondary tiles when PT mirroring is disabled?For standard user VMs,
XE_VM_FLAG_MIGRATIONis unset, meaning this if condition evaluates to false and we proceed to allocate pt_root[id] for all tiles.However, when
vm->xe->info.has_pt_mirroris false,xe_vm_pt_root()unconditionally returns pt_root[0]. This means the allocated pt_root[id] (and associated scratch page tables up tovm_max_level) for id > 0 will never be used, permanently wasting VRAM for the lifetime of every VM.Should there be an explicit check to skip secondary tile allocations when
has_pt_mirroris false, regardless of whether it is a migration VM?
Re: [PATCH v6 17/24] drm/xe: Add device flag to enable PT mirroring across tiles
In reply to Matthew BrostQuoted 105 lines · Matthew Brost
Some multi-tile devices may want to mirror page tables across tiles for memory-bandwidth reasons, while others may not. Add a device flag that allows enabling or disabling page-table mirroring across tiles.
Setting the flag to true (the existing behavior) on PVC, but both modes have been tested and are working on PVC.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_device_types.h | 2 ++ drivers/gpu/drm/xe/xe_migrate.c | 5 ++-- drivers/gpu/drm/xe/xe_pci.c | 2 ++ drivers/gpu/drm/xe/xe_pci_types.h | 3 ++- drivers/gpu/drm/xe/xe_pt.c | 36 +++++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_vm.c | 37 +++++++++++++++++++++++++--- drivers/gpu/drm/xe/xe_vm.h | 3 +++ 7 files changed, 79 insertions(+), 9 deletions(-)drivers/gpu/drm/xe/xe_device_types.h +2
-0@@ -224,6 +224,8 @@ struct xe_device { u8 has_usm:1;/** @info.has_64bit_timestamp: Device supports 64-bit timestamps */ u8 has_64bit_timestamp:1; +/** @info.has_pt_mirror: Device has PT mirroring across tiles */ + u8 has_pt_mirror:1;/** @info.is_dgfx: is discrete device */ u8 is_dgfx:1;/** @info.needs_scratch: needs scratch page for oob prefetch to work */ drivers/gpu/drm/xe/xe_migrate.c +3
-2@@ -246,7 +246,8 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,struct xe_device *xe =tile_to_xe (tile);u16 pat_index =xe_cache_pat_idx (xe,XE_CACHE_WB );u8 id = tile->id; -u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; +u32 num_entries =NUM_PT_SLOTS , num_level = +xe_vm_pt_root (vm, id)->level;#define VRAM_IDENTITY_MAP_PT_COUNT 4u32 num_setup = num_level +VRAM_IDENTITY_MAP_PT_COUNT ;#undef VRAM_IDENTITY_MAP_PT_COUNT@@ -258,7 +259,7 @@ static void xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,u64 l1_pt_ofs =xe_bo_size (bo) -5 *XE_PAGE_SIZE ; entry = vm->pt_ops->pde_encode_bo (bo, l1_pt_ofs); -xe_pt_write (xe, &vm->pt_root[id]->bo->vmap,0 , entry); +xe_pt_write (xe, &xe_vm_pt_root (vm, id)->bo->vmap,0 , entry); map_ofs = (num_entries - num_setup) *XE_PAGE_SIZE ;drivers/gpu/drm/xe/xe_pci.c +2
-0@@ -371,6 +371,7 @@ static const __maybe_unused struct xe_device_desc pvc_desc = { .has_display = false, .has_drm_ras = true, .has_gsc_nvm = 1, + .has_pt_mirror = 1, .has_heci_gscfi = 1, .max_gt_per_tile = 1, .max_remote_tiles = 1,@@ -808,6 +809,7 @@ static int xe_info_init_early(struct xe_device *xe, xe->info.has_mert = desc->has_mert; xe->info.has_page_reclaim_hw_assist = desc->has_page_reclaim_hw_assist; xe->info.has_pre_prod_wa = desc->has_pre_prod_wa; + xe->info.has_pt_mirror = desc->has_pt_mirror; xe->info.has_pxp = desc->has_pxp; xe->info.has_soc_remapper_sysctrl = desc->has_soc_remapper_sysctrl; xe->info.has_soc_remapper_telem = desc->has_soc_remapper_telem;drivers/gpu/drm/xe/xe_pci_types.h +2
-1@@ -52,8 +52,9 @@ struct xe_device_desc { u8 has_mbx_power_limits:1; u8 has_mbx_thermal_info:1; u8 has_mert:1; - u8 has_pre_prod_wa:1; u8 has_page_reclaim_hw_assist:1; + u8 has_pre_prod_wa:1; + u8 has_pt_mirror:1; u8 has_pxp:1; u8 has_soc_remapper_sysctrl:1; u8 has_soc_remapper_telem:1;drivers/gpu/drm/xe/xe_pt.c +1
-1@@ -812,7 +812,7 @@ xe_pt_stage_bind(struct xe_tile *tile, struct xe_vma *vma, .wupd.entries = entries, .clear_pt = clear_pt, }; -struct xe_pt *pt = vm->pt_root[tile->id]; +struct xe_pt *pt =xe_vm_pt_root (vm, tile->id);int ret;bool is_purged = false;
drivers/gpu/drm/xe/xe_pt.c static int xe_pt_zap_ptes_entry(struct xe_ptw *parent, pgoff_t offset, · line 1010
Quoted 6 lines
drivers/gpu/drm/xe/xe_pt.c +3
-0@@ -1005,6 +1005,11 @@ static int xe_pt_zap_ptes_entry(struct xe_ptw *parent, pgoff_t offset,return 0 ; } +static bool pt_mirroring_disabled_for_tile (struct xe_vm *vm,u8 tile_id) +{ +return xe_vm_pt_root (vm, tile_id) != vm->pt_root[tile_id];
How about return !vm->xe->info.has_pt_mirror && tile_id ?
Quoted 171 lines
drivers/gpu/drm/xe/xe_pt.c +30
-1+} +static const struct xe_pt_walk_ops xe_pt_zap_ptes_ops = { .pt_entry = xe_pt_zap_ptes_entry, };@@ -1046,6 +1051,9 @@ bool xe_pt_zap_ptes(struct xe_tile *tile, struct xe_vma *vma)if (!(pt_mask &BIT (tile->id)))return false; +if (pt_mirroring_disabled_for_tile (xe_vma_vm (vma), tile->id)) +return true; + (void )xe_pt_walk_shared (&pt->base, pt->level,xe_vma_start (vma),xe_vma_end (vma), &xe_walk.base);@@ -1098,6 +1106,9 @@ bool xe_pt_zap_ptes_range(struct xe_tile *tile, struct xe_vm *vm,if (!(pt_mask &BIT (tile->id)))return false; +if (pt_mirroring_disabled_for_tile (vm, tile->id)) +return true; + (void )xe_pt_walk_shared (&pt->base, pt->level,xe_svm_range_start (range),xe_svm_range_end (range), &xe_walk.base);@@ -1988,7 +1999,7 @@ static unsigned int xe_pt_stage_unbind(struct xe_tile *tile, .wupd.entries = entries, .prl = pt_update_op->prl, }; -struct xe_pt *pt = vm->pt_root[tile->id]; +struct xe_pt *pt =xe_vm_pt_root (vm, tile->id); (void )xe_pt_walk_shared (&pt->base, pt->level, start, end, &xe_walk.base);@@ -2541,9 +2552,21 @@ int xe_pt_update_ops_prepare(struct xe_device *xe, struct xe_vma_ops *vops)int id, err;for_each_tile (tile, xe, id) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[id]; +if (!vops->pt_update_ops[id].num_ops)continue ; +if (pt_mirroring_disabled_for_tile (vops->vm, id)) { +struct xe_page_reclaim_list *prl = &pt_update_ops->prl; + +/* Transfer root PT update ops PRL to current */ + *prl = vops->pt_update_ops[0 ].prl; +xe_page_reclaim_entries_get (prl->entries); +continue ; + } + err =__xe_pt_update_ops_prepare (tile, vops);if (err)return err;@@ -2835,6 +2858,12 @@ xe_pt_update_ops_run(struct xe_device *xe, struct xe_vma_ops *vops)struct xe_vm_pgtable_update_ops *pt_update_ops = &vops->pt_update_ops[j]; +if (pt_mirroring_disabled_for_tile (vm, j)) { +xe_tile_assert (tile, !get_current_op (pt_update_ops)); + tile_mask |=BIT (tile->id); +continue ; + } +for (i =0 ; i <get_current_op (pt_update_ops); ++i) {struct xe_vm_pgtable_update_op *pt_op =to_pt_op (pt_update_ops, i);@@ -2948,6 +2977,9 @@ void xe_pt_update_ops_abort(struct xe_device *xe, struct xe_vma_ops *vops) &vops->pt_update_ops[id];int i; +if (pt_mirroring_disabled_for_tile (vops->vm, id)) +continue ; +for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) {struct xe_vm_pgtable_update_op *pt_op =to_pt_op (pt_update_ops, i);drivers/gpu/drm/xe/xe_vm.c +33
-4@@ -845,6 +845,14 @@ int xe_vm_rebind(struct xe_vm *vm, bool rebind_worker)return err; } +static u8 adjust_rebind_tile_mask (struct xe_vm *vm,u8 tile_mask) +{ +if (vm->xe->info.has_pt_mirror) +return tile_mask; + +return (0x1 << vm->xe->info.tile_count) -1 ; +} +struct dma_fence *xe_vma_rebind (struct xe_vm *vm,struct xe_vma *vma,u8 tile_mask) {struct dma_fence *fence =NULL ;@@ -861,7 +869,8 @@ struct dma_fence *xe_vma_rebind(struct xe_vm *vm, struct xe_vma *vma, u8 tile_ma vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT |XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; - err =xe_vm_ops_add_rebind (&vops, vma, tile_mask); + err =xe_vm_ops_add_rebind (&vops, vma, +adjust_rebind_tile_mask (vm, tile_mask));if (err)return ERR_PTR (err);@@ -948,7 +957,8 @@ struct dma_fence *xe_vm_range_rebind(struct xe_vm *vm, vops.flags |=XE_VMA_OPS_FLAG_SKIP_TLB_WAIT |XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP ; - err =xe_vm_ops_add_range_rebind (&vops, vma, range, tile_mask); + err =xe_vm_ops_add_range_rebind (&vops, vma, range, +adjust_rebind_tile_mask (vm, tile_mask));if (err)return ERR_PTR (err);@@ -1738,7 +1748,8 @@ struct xe_vm *xe_vm_create(struct xe_device *xe, u32 flags, struct xe_file *xef)for_each_tile (tile, xe, id) {if (flags &XE_VM_FLAG_MIGRATION && - tile->id !=XE_VM_FLAG_TILE_ID (flags)) + tile->id !=XE_VM_FLAG_TILE_ID (flags) && + (vm->xe->info.has_pt_mirror || id))continue ; vm->pt_root[id] = xe_pt_create(vm, tile, xe->info.vm_max_level,@@ -2047,7 +2058,7 @@ struct xe_vm *xe_vm_lookup(struct xe_file *xef, u32 id)u64 xe_vm_pdp4_descriptor (struct xe_vm *vm,struct xe_tile *tile) { -return vm->pt_ops->pde_encode_bo (vm->pt_root[tile->id]->bo,0 ); +return vm->pt_ops->pde_encode_bo (xe_vm_pt_root (vm, tile->id)->bo,0 ); } static struct xe_exec_queue *@@ -5072,3 +5083,21 @@ void xe_vm_remove_exec_queue(struct xe_vm *vm, struct xe_exec_queue *q) }up_write (&vm->exec_queues.lock); } + +/** +* xe_vm_pt_root() - Retrieve VM page-table root +* @vm: The VM. +* @tile_id: Tile ID +* +* Retrieve VM page-table root for a tile ID, used to abstract if PT mirroring is +* enabled across tiles. +* +* Return: VM page-table root for a tile ID +*/ +struct xe_pt *xe_vm_pt_root (struct xe_vm *vm,u8 tile_id) +{ +if (vm->xe->info.has_pt_mirror) +return vm->pt_root[tile_id]; + +return vm->pt_root[0 ]; +}drivers/gpu/drm/xe/xe_vm.h +3
-0@@ -436,4 +436,7 @@ static inline struct drm_exec *xe_vm_validation_exec(struct xe_vm *vm) ((READ_ONCE(tile_present) & ~READ_ONCE(tile_invalidated)) & BIT((tile)->id))void xe_vma_mem_attr_copy (struct xe_vma_mem_attr *to,struct xe_vma_mem_attr *from); + +struct xe_pt *xe_vm_pt_root (struct xe_vm *vm,u8 tile_id); +#endif
Matthew Brost
maintainerPATCH v6 18/24 drm/xe: Add ULLS support to LRC
In reply to Matthew BrostDefine memory layout for ULLS semaphores stored in LRC memory. Add support functions to return GGTT address and set semaphore based on a job's seqno.
--- drivers/gpu/drm/xe/xe_lrc.c | 73 +++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_lrc.h | 4 ++ drivers/gpu/drm/xe/xe_lrc_types.h | 4 ++ 3 files changed, 81 insertions(+)
drivers/gpu/drm/xe/xe_lrc.c +73 -0
@@ -706,6 +706,7 @@ u32 xe_lrc_pphwsp_offset(struct xe_lrc *lrc)#define LRC_CTX_JOB_TIMESTAMP_OFFSET 512#define LRC_ENGINE_ID_PPHWSP_OFFSET 1024#define LRC_PARALLEL_PPHWSP_OFFSET 2048 +#define LRC_ULLS_PPHWSP_OFFSET 2048/* Mutually exclusive with parallel */ #define LRC_SEQNO_OFFSET 0#define LRC_START_SEQNO_OFFSET (LRC_SEQNO_OFFSET + 8)
@@ -768,6 +769,12 @@ static inline u32 __xe_lrc_engine_id_offset(struct xe_lrc *lrc)return xe_lrc_pphwsp_offset (lrc) +LRC_ENGINE_ID_PPHWSP_OFFSET ; } +static u32 __xe_lrc_ulls_offset (struct xe_lrc *lrc) +{ +/* The ulls is stored in the driver-defined portion of PPHWSP */ +return xe_lrc_pphwsp_offset (lrc) +LRC_ULLS_PPHWSP_OFFSET ; +} +static u32 __xe_lrc_ctx_timestamp_offset (struct xe_lrc *lrc) {return __xe_lrc_regs_offset (lrc) +CTX_TIMESTAMP *sizeof (u32);
@@ -835,6 +842,7 @@ DECL_MAP_ADDR_HELPERS(ctx_job_timestamp, lrc->bo) DECL_MAP_ADDR_HELPERS(ctx_timestamp, lrc->bo) DECL_MAP_ADDR_HELPERS(ctx_timestamp_udw, lrc->bo) DECL_MAP_ADDR_HELPERS(parallel, lrc->bo) +DECL_MAP_ADDR_HELPERS(ulls, lrc->bo) DECL_MAP_ADDR_HELPERS(indirect_ring, lrc->bo) DECL_MAP_ADDR_HELPERS(engine_id, lrc->bo)DECL_MAP_ADDR_HELPERS (queue_timestamp, lrc->bo)
@@ -1805,6 +1813,26 @@ void xe_lrc_set_ring_tail(struct xe_lrc *lrc, u32 tail)xe_lrc_write_ctx_reg (lrc,CTX_RING_TAIL , tail); } +/** +* xe_lrc_ring_tail_ggtt_addr() - Saved ring tail GGTT address +* @lrc: Pointer to the lrc. +* +* GGTT address of the ring tail as saved for this context - in the indirect +* ring state on platforms which have it, otherwise in the context image. +* This is what a context restore loads the tail register from, so a ring +* which advances the tail register itself must keep this in sync. +* +* Returns: saved ring tail GGTT address +*/ +u32 xe_lrc_ring_tail_ggtt_addr (struct xe_lrc *lrc) +{ +if (xe_lrc_has_indirect_ring_state (lrc)) +return __xe_lrc_indirect_ring_ggtt_addr (lrc) + +INDIRECT_CTX_RING_TAIL *sizeof (u32); + +return __xe_lrc_regs_ggtt_addr (lrc) +CTX_RING_TAIL *sizeof (u32); +} +u32 xe_lrc_ring_tail (struct xe_lrc *lrc) {if (xe_lrc_has_indirect_ring_state (lrc))
@@ -1984,6 +2012,51 @@ static u32 xe_lrc_engine_id(struct xe_lrc *lrc)return xe_map_read32 (xe, &map); } +#define semaphore_offset (seqno) \ + (sizeof(u32) * ((seqno) % LRC_MIGRATION_ULLS_SEMAPHORE_COUNT)) + +/** +* xe_lrc_ulls_semaphore_ggtt_addr() - ULLS semaphore GGTT address +* @lrc: Pointer to the lrc. +* @seqno: seqno of current job. +* +* Calculate ULLS semaphore GGTT address based on input seqno +* +* Returns: ULLS semaphore GGTT address +*/ +u32 xe_lrc_ulls_semaphore_ggtt_addr (struct xe_lrc *lrc,u32 seqno) +{ +xe_assert (lrc_to_xe (lrc),semaphore_offset (seqno) < +LRC_PPHWSP_SIZE -LRC_ULLS_PPHWSP_OFFSET ); + +return __xe_lrc_ulls_ggtt_addr (lrc) +semaphore_offset (seqno); +} + +/** +* xe_lrc_set_ulls_semaphore() - Set ULLS semaphore +* @lrc: Pointer to the lrc. +* @seqno: seqno of current job. +* +* Set ULLS semaphore based on input seqno +*/ +void xe_lrc_set_ulls_semaphore (struct xe_lrc *lrc,u32 seqno) +{ +struct xe_device *xe =lrc_to_xe (lrc); +struct iosys_map map =__xe_lrc_ulls_map (lrc); + +xe_assert (xe,semaphore_offset (seqno) < +LRC_PPHWSP_SIZE -LRC_ULLS_PPHWSP_OFFSET ); + +/* +* The ring contents this semaphore releases are ordered by the +* xe_device_wmb() at the end of xe_lrc_write_ring(). +*/ +iosys_map_incr (&map,semaphore_offset (seqno)); +xe_map_write32 (xe, &map,LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ); + +xe_device_wmb (xe);/* Flush write to hardware */ +} +static int instr_dw (u32 cmd_header) {/* GFXPIPE "SINGLE_DW" opcodes are a single dword */
drivers/gpu/drm/xe/xe_lrc.h +4 -0
@@ -112,6 +112,7 @@ u32 xe_lrc_regs_offset(struct xe_lrc *lrc);void xe_lrc_set_ring_tail (struct xe_lrc *lrc,u32 tail);u32 xe_lrc_ring_tail (struct xe_lrc *lrc); +u32 xe_lrc_ring_tail_ggtt_addr (struct xe_lrc *lrc);void xe_lrc_set_ring_head (struct xe_lrc *lrc,u32 head);u32 xe_lrc_ring_head (struct xe_lrc *lrc);u32 xe_lrc_ring_space (struct xe_lrc *lrc);
@@ -127,6 +128,9 @@ void xe_default_lrc_update_memirq_regs_with_address(struct xe_hw_engine *hwe);void xe_lrc_update_memirq_regs_with_address (struct xe_lrc *lrc,struct xe_hw_engine *hwe,u32 *regs); +u32 xe_lrc_ulls_semaphore_ggtt_addr (struct xe_lrc *lrc,u32 seqno); +void xe_lrc_set_ulls_semaphore (struct xe_lrc *lrc,u32 seqno); +u32 xe_lrc_read_ctx_reg (struct xe_lrc *lrc,int reg_nr);void xe_lrc_write_ctx_reg (struct xe_lrc *lrc,int reg_nr,u32 val);
drivers/gpu/drm/xe/xe_lrc_types.h +4 -0
@@ -12,6 +12,10 @@struct xe_bo ; +#define LRC_MIGRATION_ULLS_SEMAPHORE_COUNT 64/* Must be pow2 */ +#define LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR 0 +#define LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL 1 +/** * struct xe_lrc - Logical ring context (LRC) and submission ring object */
Signature · 1 line
2.34.1
Re: [PATCH v6 18/24] drm/xe: Add ULLS support to LRC
Quoted 152 lines · Matthew Brost
Define memory layout for ULLS semaphores stored in LRC memory. Add support functions to return GGTT address and set semaphore based on a job's seqno.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_lrc.c | 73 +++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_lrc.h | 4 ++ drivers/gpu/drm/xe/xe_lrc_types.h | 4 ++ 3 files changed, 81 insertions(+)drivers/gpu/drm/xe/xe_lrc.c +73
-0@@ -706,6 +706,7 @@ u32 xe_lrc_pphwsp_offset(struct xe_lrc *lrc)#define LRC_CTX_JOB_TIMESTAMP_OFFSET 512#define LRC_ENGINE_ID_PPHWSP_OFFSET 1024#define LRC_PARALLEL_PPHWSP_OFFSET 2048 +#define LRC_ULLS_PPHWSP_OFFSET 2048/* Mutually exclusive with parallel */ #define LRC_SEQNO_OFFSET 0#define LRC_START_SEQNO_OFFSET (LRC_SEQNO_OFFSET + 8)@@ -768,6 +769,12 @@ static inline u32 __xe_lrc_engine_id_offset(struct xe_lrc *lrc)return xe_lrc_pphwsp_offset (lrc) +LRC_ENGINE_ID_PPHWSP_OFFSET ; } +static u32 __xe_lrc_ulls_offset (struct xe_lrc *lrc) +{ +/* The ulls is stored in the driver-defined portion of PPHWSP */ +return xe_lrc_pphwsp_offset (lrc) +LRC_ULLS_PPHWSP_OFFSET ; +} +static u32 __xe_lrc_ctx_timestamp_offset (struct xe_lrc *lrc) {return __xe_lrc_regs_offset (lrc) +CTX_TIMESTAMP *sizeof (u32);@@ -835,6 +842,7 @@ DECL_MAP_ADDR_HELPERS(ctx_job_timestamp, lrc->bo) DECL_MAP_ADDR_HELPERS(ctx_timestamp, lrc->bo) DECL_MAP_ADDR_HELPERS(ctx_timestamp_udw, lrc->bo) DECL_MAP_ADDR_HELPERS(parallel, lrc->bo) +DECL_MAP_ADDR_HELPERS(ulls, lrc->bo) DECL_MAP_ADDR_HELPERS(indirect_ring, lrc->bo) DECL_MAP_ADDR_HELPERS(engine_id, lrc->bo)DECL_MAP_ADDR_HELPERS (queue_timestamp, lrc->bo)@@ -1805,6 +1813,26 @@ void xe_lrc_set_ring_tail(struct xe_lrc *lrc, u32 tail)xe_lrc_write_ctx_reg (lrc,CTX_RING_TAIL , tail); } +/** +* xe_lrc_ring_tail_ggtt_addr() - Saved ring tail GGTT address +* @lrc: Pointer to the lrc. +* +* GGTT address of the ring tail as saved for this context - in the indirect +* ring state on platforms which have it, otherwise in the context image. +* This is what a context restore loads the tail register from, so a ring +* which advances the tail register itself must keep this in sync. +* +* Returns: saved ring tail GGTT address +*/ +u32 xe_lrc_ring_tail_ggtt_addr (struct xe_lrc *lrc) +{ +if (xe_lrc_has_indirect_ring_state (lrc)) +return __xe_lrc_indirect_ring_ggtt_addr (lrc) + +INDIRECT_CTX_RING_TAIL *sizeof (u32); + +return __xe_lrc_regs_ggtt_addr (lrc) +CTX_RING_TAIL *sizeof (u32); +} +u32 xe_lrc_ring_tail (struct xe_lrc *lrc) {if (xe_lrc_has_indirect_ring_state (lrc))@@ -1984,6 +2012,51 @@ static u32 xe_lrc_engine_id(struct xe_lrc *lrc)return xe_map_read32 (xe, &map); } +#define semaphore_offset (seqno) \ + (sizeof(u32) * ((seqno) % LRC_MIGRATION_ULLS_SEMAPHORE_COUNT)) + +/** +* xe_lrc_ulls_semaphore_ggtt_addr() - ULLS semaphore GGTT address +* @lrc: Pointer to the lrc. +* @seqno: seqno of current job. +* +* Calculate ULLS semaphore GGTT address based on input seqno +* +* Returns: ULLS semaphore GGTT address +*/ +u32 xe_lrc_ulls_semaphore_ggtt_addr (struct xe_lrc *lrc,u32 seqno) +{ +xe_assert (lrc_to_xe (lrc),semaphore_offset (seqno) < +LRC_PPHWSP_SIZE -LRC_ULLS_PPHWSP_OFFSET ); + +return __xe_lrc_ulls_ggtt_addr (lrc) +semaphore_offset (seqno); +} + +/** +* xe_lrc_set_ulls_semaphore() - Set ULLS semaphore +* @lrc: Pointer to the lrc. +* @seqno: seqno of current job. +* +* Set ULLS semaphore based on input seqno +*/ +void xe_lrc_set_ulls_semaphore (struct xe_lrc *lrc,u32 seqno) +{ +struct xe_device *xe =lrc_to_xe (lrc); +struct iosys_map map =__xe_lrc_ulls_map (lrc); + +xe_assert (xe,semaphore_offset (seqno) < +LRC_PPHWSP_SIZE -LRC_ULLS_PPHWSP_OFFSET ); + +/* +* The ring contents this semaphore releases are ordered by the +* xe_device_wmb() at the end of xe_lrc_write_ring(). +*/ +iosys_map_incr (&map,semaphore_offset (seqno)); +xe_map_write32 (xe, &map,LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ); + +xe_device_wmb (xe);/* Flush write to hardware */ +} +static int instr_dw (u32 cmd_header) {/* GFXPIPE "SINGLE_DW" opcodes are a single dword */ drivers/gpu/drm/xe/xe_lrc.h +4
-0@@ -112,6 +112,7 @@ u32 xe_lrc_regs_offset(struct xe_lrc *lrc);void xe_lrc_set_ring_tail (struct xe_lrc *lrc,u32 tail);u32 xe_lrc_ring_tail (struct xe_lrc *lrc); +u32 xe_lrc_ring_tail_ggtt_addr (struct xe_lrc *lrc);void xe_lrc_set_ring_head (struct xe_lrc *lrc,u32 head);u32 xe_lrc_ring_head (struct xe_lrc *lrc);u32 xe_lrc_ring_space (struct xe_lrc *lrc);@@ -127,6 +128,9 @@ void xe_default_lrc_update_memirq_regs_with_address(struct xe_hw_engine *hwe);void xe_lrc_update_memirq_regs_with_address (struct xe_lrc *lrc,struct xe_hw_engine *hwe,u32 *regs); +u32 xe_lrc_ulls_semaphore_ggtt_addr (struct xe_lrc *lrc,u32 seqno); +void xe_lrc_set_ulls_semaphore (struct xe_lrc *lrc,u32 seqno); +u32 xe_lrc_read_ctx_reg (struct xe_lrc *lrc,int reg_nr);void xe_lrc_write_ctx_reg (struct xe_lrc *lrc,int reg_nr,u32 val);drivers/gpu/drm/xe/xe_lrc_types.h +0
-0
drivers/gpu/drm/xe/xe_lrc_types.h line 18
Quoted 7 lines
drivers/gpu/drm/xe/xe_lrc_types.h +4
-0@@ -12,6 +12,10 @@struct xe_bo ; +#define LRC_MIGRATION_ULLS_SEMAPHORE_COUNT 64/* Must be pow2 */ +#define LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR 0 +#define LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL 1 +
LGTM
Quoted 3 lines
drivers/gpu/drm/xe/xe_lrc_types.h +0
-0/** * struct xe_lrc - Logical ring context (LRC) and submission ring object */
Matthew Brost
maintainerPATCH v6 19/24 drm/xe: Add ULLS migration job support to migration layer
In reply to Matthew BrostAdd function to enter ULLS mode for migration job and delayed worker to exit (power saving). ULLS mode expected to entered upon page fault or SVM prefetch. ULLS mode exit delay is currently set to 5ms.
ULLS mode only support on DGFX and USM platforms where a hardware engine is reserved for migrations jobs. When in ULLS mode, set several flags on migration jobs so submission backend / ring ops can properly submit in ULLS mode.
Upon ULLS mode enter, send a job trigger waiting a semphore pipling initial GuC / HW conetxt switch.
Upon ULLS mode exit, send a job to trigger that current ULLS semaphore so the ring can be taken off the hardware.
--- drivers/gpu/drm/xe/xe_exec_queue.c | 5 +- drivers/gpu/drm/xe/xe_exec_queue.h | 2 +- drivers/gpu/drm/xe/xe_migrate.c | 198 ++++++++++++++++++++++-- drivers/gpu/drm/xe/xe_migrate.h | 2 + drivers/gpu/drm/xe/xe_pt.c | 2 +- drivers/gpu/drm/xe/xe_sched_job.h | 56 +++++++ drivers/gpu/drm/xe/xe_sched_job_types.h | 19 +++ drivers/gpu/drm/xe/xe_vm.c | 2 +- 8 files changed, 265 insertions(+), 21 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +3 -2
@@ -1546,6 +1546,7 @@ bool xe_exec_queue_is_lr(struct xe_exec_queue *q)/** * xe_exec_queue_is_idle() - Whether an exec_queue is idle. * @q: The exec_queue +* @extra_jobs: Extra jobs on the queue * * FIXME: Need to determine what to use as the short-lived * timeline lock for the exec_queues, so that the return value
@@ -1557,9 +1558,9 @@ bool xe_exec_queue_is_lr(struct xe_exec_queue *q)* * Return: True if the exec_queue is idle, false otherwise. */ -bool xe_exec_queue_is_idle (struct xe_exec_queue *q) +bool xe_exec_queue_is_idle (struct xe_exec_queue *q,int extra_jobs) { -return !atomic_read (&q->job_cnt); +return !(atomic_read (&q->job_cnt) - extra_jobs); }/**
drivers/gpu/drm/xe/xe_exec_queue.h +1 -1
@@ -116,7 +116,7 @@ static inline struct xe_exec_queue *xe_exec_queue_multi_queue_primary(struct xe_bool xe_exec_queue_is_lr (struct xe_exec_queue *q); -bool xe_exec_queue_is_idle (struct xe_exec_queue *q); +bool xe_exec_queue_is_idle (struct xe_exec_queue *q,int extra_jobs);void xe_exec_queue_kill (struct xe_exec_queue *q);
drivers/gpu/drm/xe/xe_migrate.c +182 -16
273 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.h +2 -0
@@ -97,4 +97,6 @@ int xe_migrate_debug_ccs_overlap(struct xe_migrate *m, bool write_to_ccs);#endif +void xe_migrate_ulls_enter (struct xe_migrate *m); +#endif
drivers/gpu/drm/xe/xe_pt.c +1 -1
@@ -1427,7 +1427,7 @@ static int xe_pt_vm_dependencies(struct xe_sched_job *job,if (!job && !no_in_syncs (vops->syncs, vops->num_syncs))return -ETIME ; -if (!job && !xe_exec_queue_is_idle (vops->q)) +if (!job && !xe_exec_queue_is_idle (vops->q,0 ))return -ETIME ; if (vops->flags & (XE_VMA_OPS_FLAG_WAIT_VM_BOOKKEEP |
drivers/gpu/drm/xe/xe_sched_job.h +56 -0
62 lines not on this page · raw message
drivers/gpu/drm/xe/xe_sched_job_types.h +19 -0
@@ -49,6 +49,23 @@ struct xe_job_ptrs {u32 head; }; +/** +* enum xe_ulls_state - ULLS state of a migration job +* +* Describes where a job sits in a ULLS (Ultra Low Latency Submission) +* sequence. See the ULLS documentation in xe_migrate.c. +*/ +enum xe_ulls_state { +/** @ULLS_NONE: Not a ULLS job */ +ULLS_NONE =0 , +/** @ULLS_ENTER: Job which enters ULLS mode */ +ULLS_ENTER , +/** @ULLS_ACTIVE: Job submitted while in ULLS mode */ +ULLS_ACTIVE , +/** @ULLS_EXIT: Job which exits ULLS mode */ +ULLS_EXIT , +}; +/** * struct xe_sched_job - Xe schedule job (batch buffer tracking) */
@@ -79,6 +96,8 @@ struct xe_sched_job {u32 migrate_flush_flags;/** @sample_timestamp: Sampling of job timestamp in TDR */ u64 sample_timestamp; +/** @ulls: ULLS state of this job */ +enum xe_ulls_state ulls;/** @ring_ops_flush_tlb: The ring ops need to flush TLB before payload. */ bool ring_ops_flush_tlb;/** @ring_ops_force_reset: The ring ops need to trigger a reset before payload. */
drivers/gpu/drm/xe/xe_vm.c +1 -1
@@ -148,7 +148,7 @@ static bool xe_vm_is_idle(struct xe_vm *vm)xe_vm_assert_held (vm);list_for_each_entry (q, &vm->preempt.exec_queues, lr.link) { -if (!xe_exec_queue_is_idle (q)) +if (!xe_exec_queue_is_idle (q,0 ))return false; }
Signature · 1 line
2.34.1
[email protected]
AI reviewRe: [PATCH v6 19/24] drm/xe: Add ULLS migration job support to migration layer
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Holding job_mutex while waiting on a dma_fence in xe_migrate_ulls_exit violates the dma_fence…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Holding job_mutex while waiting on a dma_fence in xe_migrate_ulls_exit violates the dma_fence contract and risks deadlocks during GPU reset. --
commit bbc9d2795fb53c65f8a7edb5c701c0f994595d7a
Author: Matthew Brost <[email protected]>
drm/xe: Add ULLS migration job support to migration layer
This commit adds a function to enter ULLS mode for migration jobs and a delayed worker to exit it for power savings. It sets flags on migration jobs so submission backends can submit in ULLS mode, which is only supported on DGFX and USM platforms.
Quoted 4 lines
drivers/gpu/drm/xe/xe_migrate.c +0
-0
[ ... ]
Quoted 1 line
drivers/gpu/drm/xe/xe_migrate.c +0
-0@@ -448,6 +468,161 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *m
[ ... ]
Quoted 2 lines
drivers/gpu/drm/xe/xe_migrate.c +2
-0+static void xe_migrate_ulls_exit (struct work_struct *work) +{
[ ... ]
drivers/gpu/drm/xe/xe_migrate.c static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *m · line 604
Quoted 10 lines
drivers/gpu/drm/xe/xe_migrate.c +10
-0+mutex_lock (&m->job_mutex); + +if (!xe_exec_queue_is_idle (m->q,1 )) + goto unlock_exit; + +xe_sched_job_get (job); + fence =__xe_migrate_job_push (m, job,ULLS_EXIT ); + +/* Serialize the PM put against the ring being taken off the hardware */ +dma_fence_wait (fence, false);
[Severity: High]
Can this lead to a deadlock during GPU reset?
In xe_migrate_ulls_exit(), the delayed worker acquires m->job_mutex, pushes the ULLS_EXIT job, and blocks on its completion fence via dma_fence_wait() before releasing the mutex.
Since m->job_mutex is explicitly tainted with fs_reclaim during initialization, marking it as used in memory reclaim paths, blocking on a dma_fence while holding this lock violates dma_fence locking rules. Fence signaling can depend on memory allocations, creating a circular dependency (job_mutex -> dma_fence -> fs_reclaim -> job_mutex).
If the GPU hangs while processing the ULLS_EXIT job, the thread blocks indefinitely. Will the DRM scheduler's GPU reset recovery path permanently deadlock here if it needs to evict memory using xe_migrate_copy() or xe_migrate_clear(), which also attempt to acquire m->job_mutex?
Quoted 6 lines
drivers/gpu/drm/xe/xe_migrate.c +6
-0+dma_fence_put (fence); + + m->ulls.enabled = false; +unlock_exit: +if (job) +xe_sched_job_put (job);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
Matthew Brost
maintainerPATCH v6 20/24 drm/xe: Add ULLS migration job support to ring ops
In reply to Matthew BrostAdd preamble and postamble for ULLS migrations jobs. Preamble clears current semaphore for reuse. Postamble waits on next semaphore which is set upon next job submission, then advances the ring tail over that job with an LRI to RING_TAIL, so submitting it costs the CPU nothing beyond signalling the semaphore.
A job updates the tail on behalf of a successor which has not been emitted yet, so it cannot know how much ring that successor will occupy. Pad every ULLS job out to a fixed ULLS_JOB_SIZE_BYTES, which makes the next tail derivable from where the current job starts. The pad also supplies the NOPs which must follow an in-ring tail update.
The last ULLS migration job skips BB submission, the postamble and the tail update (clear current semaphore, write seqno, exit ULLS), padding the difference so that it still fills a job slot.
--- drivers/gpu/drm/xe/xe_ring_ops.c | 86 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_ring_ops_types.h | 24 +++++++ 2 files changed, 110 insertions(+)
drivers/gpu/drm/xe/xe_ring_ops.c +86 -0
@@ -494,6 +494,79 @@ static void __emit_job_gen12_render_compute(struct xe_sched_job *job,xe_lrc_write_ring (lrc, dw, i *sizeof (*dw)); } +static int emit_ulls_preamble (struct xe_lrc *lrc,u32 *dw,int i,u32 seqno) +{ +u32 addr =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno); + +return emit_store_imm_ggtt (addr,LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR , + dw, i); +} + +/* +* Advance the ring tail from within the ring, so submitting the next ULLS job +* needs nothing from the CPU beyond signalling the semaphore. All ULLS jobs +* occupy exactly ULLS_JOB_SIZE_BYTES, so the tail the next job ends at is two +* job slots on from where this job started, even though that job has not been +* emitted yet. +* +* The saved tail is written before the tail register so that a context save +* racing the pair can only restore a tail behind the register, never ahead of +* it. The register write is deliberately not MI_LRI_FORCE_POSTED: posted, the +* new tail can land after the command streamer has already drained this job, +* at which point it sees an empty ring, parks, and can be taken off the +* hardware with no H2G left to bring it back. +* +* The tail is published before the semaphore wait rather than after it, so +* that the non-posted write drains while the engine is parked anyway instead +* of adding a register round trip after the semaphore is signalled. +*/ +static int emit_ulls_ring_tail (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 head) +{ +u32 next_tail = (head +2 *ULLS_JOB_SIZE_BYTES ) & (lrc->ring.size -1 ); + +xe_gt_assert (gt,IS_ALIGNED (next_tail,8 )); + + i =emit_store_imm_ggtt (xe_lrc_ring_tail_ggtt_addr (lrc), next_tail, + dw, i); + + dw[i++] =MI_LOAD_REGISTER_IMM |MI_LRI_NUM_REGS (1 ) | +MI_LRI_LRM_CS_MMIO ; + dw[i++] =RING_TAIL (0 ).addr; + dw[i++] = next_tail; + +return i; +} + +/* Publish the next job's tail, then park the engine on its semaphore */ +static int emit_ulls_postamble (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 seqno,u32 head) +{ + i =emit_ulls_ring_tail (gt, lrc, dw, i, head); + + dw[i++] =MI_SEMAPHORE_WAIT | +MI_SEMW_GGTT | +MI_SEMW_POLL | +MI_SEMW_COMPARE (SAD_EQ_SDD ); + dw[i++] =LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ; + dw[i++] =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno +1 ); + dw[i++] =0 ; + dw[i++] =0 ; + +return i; +} + +/* Pad out to the fixed ULLS job size */ +static int emit_ulls_pad (struct xe_gt *gt,u32 *dw,int i) +{ +xe_gt_assert (gt, i <=ULLS_JOB_SIZE_DW ); + +while (i <ULLS_JOB_SIZE_DW ) + dw[i++] =MI_NOOP ; + +return i; +} +static void emit_migration_job_gen12 (struct xe_sched_job *job,struct xe_lrc *lrc,u32 *head,u32 seqno)
@@ -507,10 +580,16 @@ static void emit_migration_job_gen12(struct xe_sched_job *job,xe_gt_assert (gt, !job->ring_ops_force_reset); +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_preamble (lrc, dw, i, seqno); + i =emit_copy_timestamp (xe, lrc, dw, i); i =emit_store_imm_ggtt (saddr, seqno, dw, i); +if (!xe_sched_job_ulls_has_batch (job)) + goto seqno_write; + dw[i++] =MI_ARB_ON_OFF |MI_ARB_DISABLE ;/* Enabled again below */ i =emit_bb_start (job->ptrs[0 ].batch_addr,BIT (8 ), dw, i);
@@ -521,12 +600,19 @@ static void emit_migration_job_gen12(struct xe_sched_job *job, i =emit_bb_start (job->ptrs[1 ].batch_addr,BIT (8 ), dw, i); +seqno_write: i =emit_flush_imm_ggtt (xe_lrc_seqno_ggtt_addr (lrc), seqno, job->migrate_flush_flags, dw, i); i =emit_user_interrupt (dw, i); +if (xe_sched_job_ulls_parks (job)) + i =emit_ulls_postamble (gt, lrc, dw, i, seqno, *head); + +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_pad (gt, dw, i); +xe_gt_assert (job->q->gt, i <=MAX_JOB_SIZE_DW );xe_lrc_write_ring (lrc, dw, i *sizeof (*dw));
drivers/gpu/drm/xe/xe_ring_ops_types.h +24 -0
@@ -6,6 +6,7 @@#ifndef _XE_RING_OPS_TYPES_H_#define _XE_RING_OPS_TYPES_H_ +#include <linux/build_bug.h> #include <linux/types.h> struct xe_gt ;
@@ -14,6 +15,29 @@ struct xe_sched_job;#define MAX_JOB_SIZE_DW 74#define MAX_JOB_SIZE_BYTES (MAX_JOB_SIZE_DW * 4) +/* +* ULLS migration jobs advance the ring tail from within the ring itself, so a +* job has to know where its successor will end before that successor has been +* emitted. Every ULLS job is therefore padded to a fixed size, letting the +* next tail be derived arithmetically. +* +* Sized for the largest such job, emitted by emit_migration_job_gen12(): +* preamble (4), copy timestamp (8, its size on an SRIOV VF), start seqno +* store (4), arbitration off (1), batch buffer starts (2 * 3), pre-parser +* bracketed flush invalidate (6), seqno flush (4), user interrupt (3) and +* postamble (7 + 5). +*/ +#define ULLS_JOB_SIZE_DW 48 +#define ULLS_JOB_SIZE_BYTES (ULLS_JOB_SIZE_DW * 4) + +/* +* RING_TAIL only encodes a qword aligned offset, and xe_lrc_write_ring() +* appends a NOP to anything shorter, either of which would desynchronise the +* ring from the tail a job predicts for its successor. +*/ +static_assert (ULLS_JOB_SIZE_BYTES %8 ==0 ); +static_assert (ULLS_JOB_SIZE_DW <=MAX_JOB_SIZE_DW ); +/** * struct xe_ring_ops - Ring operations */
Signature · 1 line
2.34.1
RE: [PATCH v6 20/24] drm/xe: Add ULLS migration job support to ring ops
Quoted 82 lines · Matthew Brost
Add preamble and postamble for ULLS migrations jobs. Preamble clears current semaphore for reuse. Postamble waits on next semaphore which is set upon next job submission, then advances the ring tail over that job with an LRI to
RING_TAIL, so submitting it costs the CPU nothing beyond signalling the semaphore.A job updates the tail on behalf of a successor which has not been emitted yet, so it cannot know how much ring that successor will occupy.
Pad every ULLS job out to a fixedULLS_JOB_SIZE_BYTES, which makes the next tail derivable from where the current job starts. The pad also supplies the NOPs which must follow an in-ring tail update.The last ULLS migration job skips BB submission, the postamble and the tail update (clear current semaphore, write seqno, exit ULLS), padding the difference so that it still fills a job slot.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_ring_ops.c | 86 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_ring_ops_types.h | 24 +++++++ 2 files changed, 110 insertions(+)drivers/gpu/drm/xe/xe_ring_ops.c +47
-0@@ -494,6 +494,79 @@ static void __emit_job_gen12_render_compute(struct xe_sched_job *job, xe_lrc_write_ring(lrc, dw, i * sizeof(*dw)); } +static int emit_ulls_preamble (struct xe_lrc *lrc,u32 *dw,int i,u32 +seqno) { +u32 addr =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno); + +return emit_store_imm_ggtt (addr,LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR , + dw, i); +} + +/* +* Advance the ring tail from within the ring, so submitting the next +ULLS job +* needs nothing from the CPU beyond signalling the semaphore. All ULLS +jobs +* occupy exactly ULLS_JOB_SIZE_BYTES, so the tail the next job ends at +is two +* job slots on from where this job started, even though that job has +not been +* emitted yet. +* +* The saved tail is written before the tail register so that a context +save +* racing the pair can only restore a tail behind the register, never +ahead of +* it. The register write is deliberately not MI_LRI_FORCE_POSTED: +posted, the +* new tail can land after the command streamer has already drained +this job, +* at which point it sees an empty ring, parks, and can be taken off +the +* hardware with no H2G left to bring it back. +* +* The tail is published before the semaphore wait rather than after +it, so +* that the non-posted write drains while the engine is parked anyway +instead +* of adding a register round trip after the semaphore is signalled. +*/ +static int emit_ulls_ring_tail (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 head) +{ +u32 next_tail = (head +2 *ULLS_JOB_SIZE_BYTES ) & (lrc->ring.size - +1 ); + +xe_gt_assert (gt,IS_ALIGNED (next_tail,8 )); + + i =emit_store_imm_ggtt (xe_lrc_ring_tail_ggtt_addr (lrc), next_tail, + dw, i);
drivers/gpu/drm/xe/xe_ring_ops.c static void · line 554
Quoted 10 lines
drivers/gpu/drm/xe/xe_ring_ops.c +10
-0+ + dw[i++] =MI_LOAD_REGISTER_IMM |MI_LRI_NUM_REGS (1 ) | +MI_LRI_LRM_CS_MMIO ; + dw[i++] =RING_TAIL (0 ).addr; + dw[i++] = next_tail; + +return i; +} + +/* Publish the next job's tail, then park the engine on its semaphore
One question, while tail is updated and the next job is not filled yet, Will HW prefetch the stale ring buffer?
I assume it should not be an issue, just want to confirm it. Thanks.
Rest LGTM. With the question be cleared:
Quoted 122 lines
drivers/gpu/drm/xe/xe_ring_ops.c +33
-0+*/ static int emit_ulls_postamble (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 seqno,u32 head) +{ + i =emit_ulls_ring_tail (gt, lrc, dw, i, head); + + dw[i++] =MI_SEMAPHORE_WAIT | +MI_SEMW_GGTT | +MI_SEMW_POLL | +MI_SEMW_COMPARE (SAD_EQ_SDD ); + dw[i++] =LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ; + dw[i++] =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno +1 ); + dw[i++] =0 ; + dw[i++] =0 ; + +return i; +} + +/* Pad out to the fixed ULLS job size */ static int +emit_ulls_pad (struct xe_gt *gt,u32 *dw,int i) { +xe_gt_assert (gt, i <=ULLS_JOB_SIZE_DW ); + +while (i <ULLS_JOB_SIZE_DW ) + dw[i++] =MI_NOOP ; + +return i; +} +static void emit_migration_job_gen12 (struct xe_sched_job *job,struct xe_lrc *lrc,u32 *head,u32 seqno)@@ -507,10 +580,16 @@ static void emit_migration_job_gen12(struct xe_sched_job *job,xe_gt_assert (gt, !job->ring_ops_force_reset); +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_preamble (lrc, dw, i, seqno); + i =emit_copy_timestamp (xe, lrc, dw, i); i =emit_store_imm_ggtt (saddr, seqno, dw, i); +if (!xe_sched_job_ulls_has_batch (job)) + goto seqno_write; + dw[i++] =MI_ARB_ON_OFF |MI_ARB_DISABLE ;/* Enabled again below */
drivers/gpu/drm/xe/xe_ring_ops.c +8
-0i = emit_bb_start(job->ptrs[0].batch_addr, BIT(8), dw, i); @@ -521,12 +600,19 @@ static void emit_migration_job_gen12(struct xe_sched_job *job, i = emit_bb_start(job->ptrs[1].batch_addr, BIT(8), dw, i); +seqno_write: i =emit_flush_imm_ggtt (xe_lrc_seqno_ggtt_addr (lrc), seqno, job->migrate_flush_flags, dw, i); i =emit_user_interrupt (dw, i); +if (xe_sched_job_ulls_parks (job)) + i =emit_ulls_postamble (gt, lrc, dw, i, seqno, *head); + +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_pad (gt, dw, i); +xe_gt_assert (job->q->gt, i <=MAX_JOB_SIZE_DW ); xe_lrc_write_ring(lrc, dw, i * sizeof(*dw)); diff --git
a/drivers/gpu/drm/xe/xe_ring_ops_types.hb/drivers/gpu/drm/xe/xe_ring_ops_types.h
index52ff96bc4100..ea4af321dd7c100644drivers/gpu/drm/xe/xe_ring_ops_types.h +32
-1@@ -6,6 +6,7 @@#ifndef _XE_RING_OPS_TYPES_H_#define _XE_RING_OPS_TYPES_H_ +#include <linux/build_bug.h> #include <linux/types.h> struct xe_gt ;@@ -14,6 +15,29 @@ struct xe_sched_job;#define MAX_JOB_SIZE_DW 74#define MAX_JOB_SIZE_BYTES (MAX_JOB_SIZE_DW * 4) +/* +* ULLS migration jobs advance the ring tail from within the ring +itself, so a +* job has to know where its successor will end before that successor +has been +* emitted. Every ULLS job is therefore padded to a fixed size, letting +the +* next tail be derived arithmetically. +* +* Sized for the largest such job, emitted by emit_migration_job_gen12(): +* preamble (4), copy timestamp (8, its size on an SRIOV VF), start +seqno +* store (4), arbitration off (1), batch buffer starts (2 * 3), +pre-parser +* bracketed flush invalidate (6), seqno flush (4), user interrupt (3) +and +* postamble (7 + 5). +*/ +#define ULLS_JOB_SIZE_DW 48 +#define ULLS_JOB_SIZE_BYTES (ULLS_JOB_SIZE_DW * 4) + +/* +* RING_TAIL only encodes a qword aligned offset, and +xe_lrc_write_ring() +* appends a NOP to anything shorter, either of which would +desynchronise the +* ring from the tail a job predicts for its successor. +*/ +static_assert (ULLS_JOB_SIZE_BYTES %8 ==0 ); +static_assert (ULLS_JOB_SIZE_DW <=MAX_JOB_SIZE_DW ); +/** * struct xe_ring_ops - Ring operations */ --2.34.1
Matthew Brost
maintainerRe: [PATCH v6 20/24] drm/xe: Add ULLS migration job support to ring ops
Quoted 89 lines · Shuicheng
Quoted 88 lines · Matthew Brost
Add preamble and postamble for ULLS migrations jobs. Preamble clears current semaphore for reuse. Postamble waits on next semaphore which is set upon next job submission, then advances the ring tail over that job with an LRI to
RING_TAIL, so submitting it costs the CPU nothing beyond signalling the semaphore.A job updates the tail on behalf of a successor which has not been emitted yet, so it cannot know how much ring that successor will occupy.
Pad every ULLS job out to a fixedULLS_JOB_SIZE_BYTES, which makes the next tail derivable from where the current job starts. The pad also supplies the NOPs which must follow an in-ring tail update.The last ULLS migration job skips BB submission, the postamble and the tail update (clear current semaphore, write seqno, exit ULLS), padding the difference so that it still fills a job slot.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_ring_ops.c | 86 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_ring_ops_types.h | 24 +++++++ 2 files changed, 110 insertions(+)drivers/gpu/drm/xe/xe_ring_ops.c +53
-0@@ -494,6 +494,79 @@ static void __emit_job_gen12_render_compute(struct xe_sched_job *job, xe_lrc_write_ring(lrc, dw, i * sizeof(*dw)); } +static int emit_ulls_preamble (struct xe_lrc *lrc,u32 *dw,int i,u32 +seqno) { +u32 addr =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno); + +return emit_store_imm_ggtt (addr,LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR , + dw, i); +} + +/* +* Advance the ring tail from within the ring, so submitting the next +ULLS job +* needs nothing from the CPU beyond signalling the semaphore. All ULLS +jobs +* occupy exactly ULLS_JOB_SIZE_BYTES, so the tail the next job ends at +is two +* job slots on from where this job started, even though that job has +not been +* emitted yet. +* +* The saved tail is written before the tail register so that a context +save +* racing the pair can only restore a tail behind the register, never +ahead of +* it. The register write is deliberately not MI_LRI_FORCE_POSTED: +posted, the +* new tail can land after the command streamer has already drained +this job, +* at which point it sees an empty ring, parks, and can be taken off +the +* hardware with no H2G left to bring it back. +* +* The tail is published before the semaphore wait rather than after +it, so +* that the non-posted write drains while the engine is parked anyway +instead +* of adding a register round trip after the semaphore is signalled. +*/ +static int emit_ulls_ring_tail (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 head) +{ +u32 next_tail = (head +2 *ULLS_JOB_SIZE_BYTES ) & (lrc->ring.size - +1 ); + +xe_gt_assert (gt,IS_ALIGNED (next_tail,8 )); + + i =emit_store_imm_ggtt (xe_lrc_ring_tail_ggtt_addr (lrc), next_tail, + dw, i); + + dw[i++] =MI_LOAD_REGISTER_IMM |MI_LRI_NUM_REGS (1 ) | +MI_LRI_LRM_CS_MMIO ; + dw[i++] =RING_TAIL (0 ).addr; + dw[i++] = next_tail; +
Quoted 9 lines
drivers/gpu/drm/xe/xe_ring_ops.c static void __emit_job_gen12_render_compute(struct xe_sched_job *job,
Quoted 4 lines
drivers/gpu/drm/xe/xe_ring_ops.c +4
-0+return i; +} + +/* Publish the next job's tail, then park the engine on its semaphore One question, while tail is updated and the next job is not filled yet, Will HW prefetch the stale ring buffer?
I assume it should not be an issue, just want to confirm it. Thanks.
When I changed from MMIO ring tail move to a ring instruction move, at one point I disabled the pre-parser when I was chasing bugs here. It doesn't appear to be needed - I believe the semaphore after this acts as barrier in the ring prefetch for what I can tell.
Matt
Quoted 127 lines
Rest LGTM. With the question be cleared:
Reviewed-by Shuicheng Lin <[email protected]>Quoted 122 lines
drivers/gpu/drm/xe/xe_ring_ops.c +33
-0+*/ static int emit_ulls_postamble (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 seqno,u32 head) +{ + i =emit_ulls_ring_tail (gt, lrc, dw, i, head); + + dw[i++] =MI_SEMAPHORE_WAIT | +MI_SEMW_GGTT | +MI_SEMW_POLL | +MI_SEMW_COMPARE (SAD_EQ_SDD ); + dw[i++] =LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ; + dw[i++] =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno +1 ); + dw[i++] =0 ; + dw[i++] =0 ; + +return i; +} + +/* Pad out to the fixed ULLS job size */ static int +emit_ulls_pad (struct xe_gt *gt,u32 *dw,int i) { +xe_gt_assert (gt, i <=ULLS_JOB_SIZE_DW ); + +while (i <ULLS_JOB_SIZE_DW ) + dw[i++] =MI_NOOP ; + +return i; +} +static void emit_migration_job_gen12 (struct xe_sched_job *job,struct xe_lrc *lrc,u32 *head,u32 seqno)@@ -507,10 +580,16 @@ static void emit_migration_job_gen12(struct xe_sched_job *job,xe_gt_assert (gt, !job->ring_ops_force_reset); +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_preamble (lrc, dw, i, seqno); + i =emit_copy_timestamp (xe, lrc, dw, i); i =emit_store_imm_ggtt (saddr, seqno, dw, i); +if (!xe_sched_job_ulls_has_batch (job)) + goto seqno_write; + dw[i++] =MI_ARB_ON_OFF |MI_ARB_DISABLE ;/* Enabled again below */
drivers/gpu/drm/xe/xe_ring_ops.c +8
-0i = emit_bb_start(job->ptrs[0].batch_addr, BIT(8), dw, i); @@ -521,12 +600,19 @@ static void emit_migration_job_gen12(struct xe_sched_job *job, i = emit_bb_start(job->ptrs[1].batch_addr, BIT(8), dw, i); +seqno_write: i =emit_flush_imm_ggtt (xe_lrc_seqno_ggtt_addr (lrc), seqno, job->migrate_flush_flags, dw, i); i =emit_user_interrupt (dw, i); +if (xe_sched_job_ulls_parks (job)) + i =emit_ulls_postamble (gt, lrc, dw, i, seqno, *head); + +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_pad (gt, dw, i); +xe_gt_assert (job->q->gt, i <=MAX_JOB_SIZE_DW ); xe_lrc_write_ring(lrc, dw, i * sizeof(*dw)); diff --git
a/drivers/gpu/drm/xe/xe_ring_ops_types.hb/drivers/gpu/drm/xe/xe_ring_ops_types.h
index52ff96bc4100..ea4af321dd7c100644drivers/gpu/drm/xe/xe_ring_ops_types.h +32
-1@@ -6,6 +6,7 @@#ifndef _XE_RING_OPS_TYPES_H_#define _XE_RING_OPS_TYPES_H_ +#include <linux/build_bug.h> #include <linux/types.h> struct xe_gt ;@@ -14,6 +15,29 @@ struct xe_sched_job;#define MAX_JOB_SIZE_DW 74#define MAX_JOB_SIZE_BYTES (MAX_JOB_SIZE_DW * 4) +/* +* ULLS migration jobs advance the ring tail from within the ring +itself, so a +* job has to know where its successor will end before that successor +has been +* emitted. Every ULLS job is therefore padded to a fixed size, letting +the +* next tail be derived arithmetically. +* +* Sized for the largest such job, emitted by emit_migration_job_gen12(): +* preamble (4), copy timestamp (8, its size on an SRIOV VF), start +seqno +* store (4), arbitration off (1), batch buffer starts (2 * 3), +pre-parser +* bracketed flush invalidate (6), seqno flush (4), user interrupt (3) +and +* postamble (7 + 5). +*/ +#define ULLS_JOB_SIZE_DW 48 +#define ULLS_JOB_SIZE_BYTES (ULLS_JOB_SIZE_DW * 4) + +/* +* RING_TAIL only encodes a qword aligned offset, and +xe_lrc_write_ring() +* appends a NOP to anything shorter, either of which would +desynchronise the +* ring from the tail a job predicts for its successor. +*/ +static_assert (ULLS_JOB_SIZE_BYTES %8 ==0 ); +static_assert (ULLS_JOB_SIZE_DW <=MAX_JOB_SIZE_DW ); +/** * struct xe_ring_ops - Ring operations */ --2.34.1
Re: [PATCH v6 20/24] drm/xe: Add ULLS migration job support to ring ops
In reply to Matthew BrostQuoted 174 lines · Matthew Brost
Add preamble and postamble for ULLS migrations jobs. Preamble clears current semaphore for reuse. Postamble waits on next semaphore which is set upon next job submission, then advances the ring tail over that job with an LRI to
RING_TAIL, so submitting it costs the CPU nothing beyond signalling the semaphore.A job updates the tail on behalf of a successor which has not been emitted yet, so it cannot know how much ring that successor will occupy. Pad every ULLS job out to a fixed
ULLS_JOB_SIZE_BYTES, which makes the next tail derivable from where the current job starts. The pad also supplies the NOPs which must follow an in-ring tail update.The last ULLS migration job skips BB submission, the postamble and the tail update (clear current semaphore, write seqno, exit ULLS), padding the difference so that it still fills a job slot.
Signed-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_ring_ops.c | 86 ++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_ring_ops_types.h | 24 +++++++ 2 files changed, 110 insertions(+)drivers/gpu/drm/xe/xe_ring_ops.c +86
-0@@ -494,6 +494,79 @@ static void __emit_job_gen12_render_compute(struct xe_sched_job *job,xe_lrc_write_ring (lrc, dw, i *sizeof (*dw)); } +static int emit_ulls_preamble (struct xe_lrc *lrc,u32 *dw,int i,u32 seqno) +{ +u32 addr =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno); + +return emit_store_imm_ggtt (addr,LRC_MIGRATION_ULLS_SEMAPHORE_CLEAR , + dw, i); +} + +/* +* Advance the ring tail from within the ring, so submitting the next ULLS job +* needs nothing from the CPU beyond signalling the semaphore. All ULLS jobs +* occupy exactly ULLS_JOB_SIZE_BYTES, so the tail the next job ends at is two +* job slots on from where this job started, even though that job has not been +* emitted yet. +* +* The saved tail is written before the tail register so that a context save +* racing the pair can only restore a tail behind the register, never ahead of +* it. The register write is deliberately not MI_LRI_FORCE_POSTED: posted, the +* new tail can land after the command streamer has already drained this job, +* at which point it sees an empty ring, parks, and can be taken off the +* hardware with no H2G left to bring it back. +* +* The tail is published before the semaphore wait rather than after it, so +* that the non-posted write drains while the engine is parked anyway instead +* of adding a register round trip after the semaphore is signalled. +*/ +static int emit_ulls_ring_tail (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 head) +{ +u32 next_tail = (head +2 *ULLS_JOB_SIZE_BYTES ) & (lrc->ring.size -1 ); + +xe_gt_assert (gt,IS_ALIGNED (next_tail,8 )); + + i =emit_store_imm_ggtt (xe_lrc_ring_tail_ggtt_addr (lrc), next_tail, + dw, i); + + dw[i++] =MI_LOAD_REGISTER_IMM |MI_LRI_NUM_REGS (1 ) | +MI_LRI_LRM_CS_MMIO ; + dw[i++] =RING_TAIL (0 ).addr; + dw[i++] = next_tail; + +return i; +} + +/* Publish the next job's tail, then park the engine on its semaphore */ +static int emit_ulls_postamble (struct xe_gt *gt,struct xe_lrc *lrc,u32 *dw, +int i,u32 seqno,u32 head) +{ + i =emit_ulls_ring_tail (gt, lrc, dw, i, head); + + dw[i++] =MI_SEMAPHORE_WAIT | +MI_SEMW_GGTT | +MI_SEMW_POLL | +MI_SEMW_COMPARE (SAD_EQ_SDD ); + dw[i++] =LRC_MIGRATION_ULLS_SEMAPHORE_SIGNAL ; + dw[i++] =xe_lrc_ulls_semaphore_ggtt_addr (lrc, seqno +1 ); + dw[i++] =0 ; + dw[i++] =0 ; + +return i; +} + +/* Pad out to the fixed ULLS job size */ +static int emit_ulls_pad (struct xe_gt *gt,u32 *dw,int i) +{ +xe_gt_assert (gt, i <=ULLS_JOB_SIZE_DW ); + +while (i <ULLS_JOB_SIZE_DW ) + dw[i++] =MI_NOOP ; + +return i; +} +static void emit_migration_job_gen12 (struct xe_sched_job *job,struct xe_lrc *lrc,u32 *head,u32 seqno)@@ -507,10 +580,16 @@ static void emit_migration_job_gen12(struct xe_sched_job *job,xe_gt_assert (gt, !job->ring_ops_force_reset); +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_preamble (lrc, dw, i, seqno); + i =emit_copy_timestamp (xe, lrc, dw, i); i =emit_store_imm_ggtt (saddr, seqno, dw, i); +if (!xe_sched_job_ulls_has_batch (job)) + goto seqno_write; + dw[i++] =MI_ARB_ON_OFF |MI_ARB_DISABLE ;/* Enabled again below */ i =emit_bb_start (job->ptrs[0 ].batch_addr,BIT (8 ), dw, i);@@ -521,12 +600,19 @@ static void emit_migration_job_gen12(struct xe_sched_job *job, i =emit_bb_start (job->ptrs[1 ].batch_addr,BIT (8 ), dw, i); +seqno_write: i =emit_flush_imm_ggtt (xe_lrc_seqno_ggtt_addr (lrc), seqno, job->migrate_flush_flags, dw, i); i =emit_user_interrupt (dw, i); +if (xe_sched_job_ulls_parks (job)) + i =emit_ulls_postamble (gt, lrc, dw, i, seqno, *head); + +if (xe_sched_job_is_ulls (job)) + i =emit_ulls_pad (gt, dw, i); +xe_gt_assert (job->q->gt, i <=MAX_JOB_SIZE_DW );xe_lrc_write_ring (lrc, dw, i *sizeof (*dw));drivers/gpu/drm/xe/xe_ring_ops_types.h +14
-0@@ -6,6 +6,7 @@#ifndef _XE_RING_OPS_TYPES_H_#define _XE_RING_OPS_TYPES_H_ +#include <linux/build_bug.h> #include <linux/types.h> struct xe_gt ;@@ -14,6 +15,29 @@ struct xe_sched_job;#define MAX_JOB_SIZE_DW 74#define MAX_JOB_SIZE_BYTES (MAX_JOB_SIZE_DW * 4) +/* +* ULLS migration jobs advance the ring tail from within the ring itself, so a +* job has to know where its successor will end before that successor has been +* emitted. Every ULLS job is therefore padded to a fixed size, letting the +* next tail be derived arithmetically. +* +* Sized for the largest such job, emitted by emit_migration_job_gen12(): +* preamble (4), copy timestamp (8, its size on an SRIOV VF), start seqno +* store (4), arbitration off (1), batch buffer starts (2 * 3), pre-parser +* bracketed flush invalidate (6), seqno flush (4), user interrupt (3) and +* postamble (7 + 5). +*/ +#define ULLS_JOB_SIZE_DW 48
drivers/gpu/drm/xe/xe_ring_ops_types.h struct xe_sched_job; · line 40
Quoted 10 lines
drivers/gpu/drm/xe/xe_ring_ops_types.h +10
-0+#define ULLS_JOB_SIZE_BYTES (ULLS_JOB_SIZE_DW * 4) + +/* +* RING_TAIL only encodes a qword aligned offset, and xe_lrc_write_ring() +* appends a NOP to anything shorter, either of which would desynchronise the +* ring from the tail a job predicts for its successor. +*/ +static_assert (ULLS_JOB_SIZE_BYTES %8 ==0 ); +static_assert (ULLS_JOB_SIZE_DW <=MAX_JOB_SIZE_DW ); +
LGTM
Quoted 3 lines
drivers/gpu/drm/xe/xe_ring_ops_types.h +0
-0/** * struct xe_ring_ops - Ring operations */
Matthew Brost
maintainerPATCH v6 21/24 drm/xe: Add ULLS migration job support to GuC submission
In reply to Matthew BrostAdd ULLS migration job support to GuC submission backend.
Changes required:
- On migration queue, reduce max jobs to the number of ULLS semaphores minus one
- Skip writing the saved ring tail for ULLS jobs except for the first ULLS job - the ring updates both the saved tail and the tail register itself
- Set ULLS sempahore for current job releasing last job except for first ULLS job
- Suppress submit H2G for ULLS except for first ULLS job
--- drivers/gpu/drm/xe/xe_guc_submit.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_guc_submit.c +13 -4
@@ -1189,9 +1189,10 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)xe_gt_assert (guc_to_gt (guc),exec_queue_registered (q));if (!job->restore_replay || job->last_replay) { +/* A ULLS job past the first publishes its own ring tail */ if (xe_exec_queue_is_parallel (q))wq_item_append (q); -else +else if (!xe_sched_job_ulls_is_chained (job))xe_lrc_set_ring_tail (lrc, lrc->ring.tail); job->last_replay = false; }
@@ -1208,6 +1209,9 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)if (exec_queue_suspended (q))return ; +if (xe_sched_job_ulls_is_chained (job)) +xe_lrc_set_ulls_semaphore (lrc,xe_sched_job_lrc_seqno (job)); +if (!exec_queue_enabled (q)) { action[len++] =XE_GUC_ACTION_SCHED_CONTEXT_MODE_SET ; action[len++] = q->guc->id;
@@ -1221,13 +1225,14 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)set_exec_queue_pending_enable (q);set_exec_queue_enabled (q);trace_xe_exec_queue_scheduling_enable (q); - }else { + }else if (!xe_sched_job_ulls_is_chained (job)) { action[len++] =XE_GUC_ACTION_SCHED_CONTEXT ; action[len++] = q->guc->id;trace_xe_exec_queue_submit (q); } -xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h); +if (!xe_sched_job_ulls_is_chained (job) || num_g2h) +xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h);if (extra_submit) { len =0 ;
@@ -2098,6 +2103,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)struct xe_guc_exec_queue *ge;long timeout;int err, i; +int max_jobs = (xe_lrc_ring_size () /MAX_JOB_SIZE_BYTES );xe_gt_assert (guc_to_gt (guc),xe_device_uc_enabled (guc_to_xe (guc)));
@@ -2137,8 +2143,11 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) submit_wq = primary->guc->sched.base.submit_wq; } +if (q->vm && q->vm->flags &XE_VM_FLAG_MIGRATION ) + max_jobs =min (max_jobs,LRC_MIGRATION_ULLS_SEMAPHORE_COUNT -1 ); + err =xe_sched_init (&ge->sched, &drm_sched_ops, &xe_sched_ops, - submit_wq,xe_lrc_ring_size () /MAX_JOB_SIZE_BYTES ,64 , + submit_wq, max_jobs,64 , timeout,guc_to_gt (guc)->ordered_wq,NULL , ge->name,gt_to_xe (q->gt)->drm.dev);if (err)
Signature · 1 line
2.34.1
RE: [PATCH v6 21/24] drm/xe: Add ULLS migration job support to GuC submission
Quoted 79 lines · Matthew Brost
Add ULLS migration job support to GuC submission backend.
Changes required:
- On migration queue, reduce max jobs to the number of ULLS semaphores minus one
- Skip writing the saved ring tail for ULLS jobs except for the first ULLS job - the ring updates both the saved tail and the tail register itself
- Set ULLS sempahore for current job releasing last job except for first ULLS job
- Suppress submit H2G for ULLS except for first ULLS jobSigned-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_guc_submit.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)drivers/gpu/drm/xe/xe_guc_submit.c +2
-1@@ -1189,9 +1189,10 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)xe_gt_assert (guc_to_gt (guc ),exec_queue_registered (q ));if (!job->restore_replay || job->last_replay) { +/* A ULLS job past the first publishes its own ring tail */ if (xe_exec_queue_is_parallel (q))wq_item_append (q); -else +else if (!xe_sched_job_ulls_is_chained (job))xe_lrc_set_ring_tail (lrc, lrc->ring.tail); job->last_replay = false;drivers/gpu/drm/xe/xe_guc_submit.c +3
-0}@@ -1208,6 +1209,9 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job) if (exec_queue_suspended(q)) return; +if (xe_sched_job_ulls_is_chained (job)) +xe_lrc_set_ulls_semaphore (lrc,xe_sched_job_lrc_seqno (job)); +if (!exec_queue_enabled (q)) { action[len++] =
XE_GUC_ACTION_SCHED_CONTEXT_MODE_SET;drivers/gpu/drm/xe/xe_guc_submit.c +3
-2action[len++] = q->guc->id;@@ -1221,13 +1225,14 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)set_exec_queue_pending_enable (q );set_exec_queue_enabled (q);trace_xe_exec_queue_scheduling_enable (q); - }else { + }else if (!xe_sched_job_ulls_is_chained (job)) { action[len++] =XE_GUC_ACTION_SCHED_CONTEXT ; action[len++] = q->guc->id;trace_xe_exec_queue_submit (q); } -xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h); +if (!xe_sched_job_ulls_is_chained (job) || num_g2h) +xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h);if (extra_submit) {drivers/gpu/drm/xe/xe_guc_submit.c +1
-0len =0 ;@@ -2098,6 +2103,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)struct xe_guc_exec_queue *ge;long timeout;int err, i; +int max_jobs = (xe_lrc_ring_size () /MAX_JOB_SIZE_BYTES ); xe_gt_assert(guc_to_gt(guc),xe_device_uc_enabled(guc_to_xe(guc)));
drivers/gpu/drm/xe/xe_guc_submit.c static int guc_exec_queue_init(struct · line 2149
Quoted 7 lines
drivers/gpu/drm/xe/xe_guc_submit.c +2
-0@@ -2137,8 +2143,11 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) submit_wq = primary->guc->sched.base.submit_wq; } +if (q->vm && q->vm->flags &XE_VM_FLAG_MIGRATION ) + max_jobs =min (max_jobs,LRC_MIGRATION_ULLS_SEMAPHORE_COUNT -1 );
If (xe_sched_job_is_migration(q)) ...
This check is broader than the queue that can enter ULLS.
It is OK now, as xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES equal to 55, while LRC_MIGRATION_ULLS_SEMAPHORE_COUNT is 64, so the max_jobs is kept at 55. If we reduce LRC_MIGRATION_ULLS_SEMAPHORE_COUNT later, we may need to restrict this min() only to ULLS queue.
LGTM.
Quoted 10 lines
drivers/gpu/drm/xe/xe_guc_submit.c +2
-1+ err = xe_sched_init(&ge->sched, &drm_sched_ops, &xe_sched_ops, - submit_wq, xe_lrc_ring_size() / MAX_JOB_SIZE_BYTES, 64, + submit_wq, max_jobs, 64,drivers/gpu/drm/xe/xe_guc_submit.c +0
-1timeout,guc_to_gt (guc)->ordered_wq,NULL , ge->name, gt_to_xe(q->gt)->drm.dev);if (err) --2.34.1
Re: [PATCH v6 21/24] drm/xe: Add ULLS migration job support to GuC submission
In reply to Matthew BrostQuoted 71 lines · Matthew Brost
Add ULLS migration job support to GuC submission backend.
Changes required:
- On migration queue, reduce max jobs to the number of ULLS semaphores minus one
- Skip writing the saved ring tail for ULLS jobs except for the first ULLS job - the ring updates both the saved tail and the tail register itself
- Set ULLS sempahore for current job releasing last job except for first ULLS job
- Suppress submit H2G for ULLS except for first ULLS jobSigned-off-by Matthew Brost <[email protected]>Signed-off-by Maarten Lankhorst <[email protected]>--- drivers/gpu/drm/xe/xe_guc_submit.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)drivers/gpu/drm/xe/xe_guc_submit.c +9
-3@@ -1189,9 +1189,10 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)xe_gt_assert (guc_to_gt (guc),exec_queue_registered (q));if (!job->restore_replay || job->last_replay) { +/* A ULLS job past the first publishes its own ring tail */ if (xe_exec_queue_is_parallel (q))wq_item_append (q); -else +else if (!xe_sched_job_ulls_is_chained (job))xe_lrc_set_ring_tail (lrc, lrc->ring.tail); job->last_replay = false; }@@ -1208,6 +1209,9 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)if (exec_queue_suspended (q))return ; +if (xe_sched_job_ulls_is_chained (job)) +xe_lrc_set_ulls_semaphore (lrc,xe_sched_job_lrc_seqno (job)); +if (!exec_queue_enabled (q)) { action[len++] =XE_GUC_ACTION_SCHED_CONTEXT_MODE_SET ; action[len++] = q->guc->id;@@ -1221,13 +1225,14 @@ static void submit_exec_queue(struct xe_exec_queue *q, struct xe_sched_job *job)set_exec_queue_pending_enable (q);set_exec_queue_enabled (q);trace_xe_exec_queue_scheduling_enable (q); - }else { + }else if (!xe_sched_job_ulls_is_chained (job)) { action[len++] =XE_GUC_ACTION_SCHED_CONTEXT ; action[len++] = q->guc->id;trace_xe_exec_queue_submit (q); } -xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h); +if (!xe_sched_job_ulls_is_chained (job) || num_g2h) +xe_guc_ct_send (&guc->ct, action, len, g2h_len, num_g2h);if (extra_submit) { len =0 ;@@ -2098,6 +2103,7 @@ static int guc_exec_queue_init(struct xe_exec_queue *q)struct xe_guc_exec_queue *ge;long timeout;int err, i; +int max_jobs = (xe_lrc_ring_size () /MAX_JOB_SIZE_BYTES );xe_gt_assert (guc_to_gt (guc),xe_device_uc_enabled (guc_to_xe (guc)));
drivers/gpu/drm/xe/xe_guc_submit.c static int guc_exec_queue_init(struct xe_exec_queue *q) · line 2148
Quoted 6 lines
drivers/gpu/drm/xe/xe_guc_submit.c +3
-0@@ -2137,8 +2143,11 @@ static int guc_exec_queue_init(struct xe_exec_queue *q) submit_wq = primary->guc->sched.base.submit_wq; } +if (q->vm && q->vm->flags &XE_VM_FLAG_MIGRATION ) + max_jobs =min (max_jobs,LRC_MIGRATION_ULLS_SEMAPHORE_COUNT -1 ); +
Quoted 6 lines
drivers/gpu/drm/xe/xe_guc_submit.c +1
-1err =xe_sched_init (&ge->sched, &drm_sched_ops, &xe_sched_ops, - submit_wq,xe_lrc_ring_size () /MAX_JOB_SIZE_BYTES ,64 , + submit_wq, max_jobs,64 , timeout,guc_to_gt (guc)->ordered_wq,NULL , ge->name,gt_to_xe (q->gt)->drm.dev);if (err)
Matthew Brost
maintainerPATCH v6 22/24 drm/xe: Enter ULLS for migration jobs upon page fault or SVM prefetch
In reply to Matthew BrostCall xe_migration_ulls_enter upon page fault or SVM prefetch in an effort speed up these critical paths.
--- drivers/gpu/drm/xe/xe_pagefault.c | 3 +++ drivers/gpu/drm/xe/xe_vm.c | 4 +++- 2 files changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_pagefault.c +3 -0
@@ -15,6 +15,7 @@#include "xe_gt_stats.h" #include "xe_hw_engine.h" #include "xe_log.h" +#include "xe_migrate.h" #include "xe_pagefault.h" #include "xe_pagefault_types.h" #include "xe_svm.h"
@@ -262,6 +263,8 @@ static int xe_pagefault_service(struct xe_pagefault *pf)if (IS_ERR (vm))return PTR_ERR (vm); +xe_migrate_ulls_enter (gt_to_tile (gt)->migrate); +down_read (&vm->lock);if (xe_vm_is_closed (vm)) {
drivers/gpu/drm/xe/xe_vm.c +3 -1
@@ -2526,8 +2526,10 @@ vm_bind_ioctl_ops_create(struct xe_vm *vm, struct xe_vma_ops *vops, ctx.devmem_possible =IS_DGFX (vm->xe) &&IS_ENABLED (CONFIG_DRM_XE_PAGEMAP ); -for_each_tile (tile, vm->xe, id) +for_each_tile (tile, vm->xe, id) { +xe_migrate_ulls_enter (tile->migrate); tile_mask |=0x1 << id; + }if (prefetch_region ==DRM_XE_CONSULT_MEM_ADVISE_PREF_LOC ) { dpagemap = xe_vma_resolve_pagemap(vma,
Signature · 1 line
2.34.1