Convert multiple bind ops to 1 job
Subsystems (2)
Matthew Brost
maintainerv7 cover Convert multiple bind ops to 1 job
Continuation of merging parts of [1]. Patch #2 in this series is quite large but unsure how to split the patch without breaking functionality.
Tested with [2].
v2:
- Rebase
- Add error injection patch
- Fix dma-fence reservation for binds
v3:
- Rebase
- Error injection patch omitted in this rev
v4:
- Rebase
- Fix tlb invalidation (Paulo)
- Add missing last fence dep / test to inspection (Inspection)
- Add error injection patch back in
v5:
- Address Matthew Auld's feedback
v6:
- Fix CI failure (Matthew Auld, CI)
- Fix checkpatch issues
- Return -ENOBUFS in Xe layer rather than DRM SA layer to avoid warns (Matthew Auld)
v7:
- Address Matthew Auld's feedback
- Fix mesa failure in which VM bind jobs timed on integrated parts if number of PTE updates > 510 (Paulo)
Matt
[1] https://patchwork.freedesktop.org/series/125608/
[2] https://patchwork.freedesktop.org/series/129606/
Matthew Brost (7):
drm/xe: s/xe_tile_migrate_engine/xe_tile_migrate_exec_queue drm/xe: Add xe_vm_pgtable_update_op to xe_vma_ops drm/xe: Add xe_exec_queue_last_fence_test_dep drm/xe: Convert multiple bind ops into single job drm/xe: Update VM trace events
drm/xe: Update PT layer with better error handling drm/xe: Add VM bind IOCTL error injection
drivers/gpu/drm/xe/xe_bo_types.h | 2 + drivers/gpu/drm/xe/xe_device_types.h | 12 + drivers/gpu/drm/xe/xe_exec_queue.c | 23 + drivers/gpu/drm/xe/xe_exec_queue.h | 2 + drivers/gpu/drm/xe/xe_migrate.c | 342 ++++--- drivers/gpu/drm/xe/xe_migrate.h | 34 +- drivers/gpu/drm/xe/xe_pt.c | 1284 +++++++++++++++++--------- drivers/gpu/drm/xe/xe_pt.h | 14 +- drivers/gpu/drm/xe/xe_pt_types.h | 48 + drivers/gpu/drm/xe/xe_sa.c | 7 + drivers/gpu/drm/xe/xe_trace_bo.h | 10 +- drivers/gpu/drm/xe/xe_vm.c | 647 +++++-------- drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 55 +- 14 files changed, 1413 insertions(+), 1069 deletions(-)
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 1/7 drm/xe: s/xe_tile_migrate_engine/xe_tile_migrate_exec_queue
Engine is old nomenclature, replace with exec queue.
--- drivers/gpu/drm/xe/xe_migrate.c | 9 ++++----- drivers/gpu/drm/xe/xe_migrate.h | 2 +- 2 files changed, 5 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_migrate.c +4 -5
@@ -84,15 +84,14 @@ struct xe_migrate {#define MAX_PTE_PER_SDI 0x1FE/** -* xe_tile_migrate_engine() - Get this tile's migrate engine. +* xe_tile_migrate_exec_queue() - Get this tile's migrate exec queue. * @tile: The tile. * -* Returns the default migrate engine of this tile. -* TODO: Perhaps this function is slightly misplaced, and even unneeded? +* Returns the default migrate exec queue of this tile. * -* Return: The default migrate engine +* Return: The default migrate exec queue */ -struct xe_exec_queue *xe_tile_migrate_engine (struct xe_tile *tile) +struct xe_exec_queue *xe_tile_migrate_exec_queue (struct xe_tile *tile) {return tile->migrate->q; }
drivers/gpu/drm/xe/xe_migrate.h +1 -1
@@ -106,5 +106,5 @@ xe_migrate_update_pgtables(struct xe_migrate *m,void xe_migrate_wait (struct xe_migrate *m); -struct xe_exec_queue *xe_tile_migrate_engine (struct xe_tile *tile); +struct xe_exec_queue *xe_tile_migrate_exec_queue (struct xe_tile *tile);#endif
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 2/7 drm/xe: Add xe_vm_pgtable_update_op to xe_vma_ops
In reply to Matthew BrostEach xe_vma_op resolves to 0-3 pt_ops. Add storage for the pt_ops to xe_vma_ops which is dynamically allocated based the number and types of xe_vma_op in the xe_vma_ops list. Allocation only implemented in this patch.
This will help with converting xe_vma_ops (multiple xe_vma_op) in a atomic update unit.
--- drivers/gpu/drm/xe/xe_pt_types.h | 12 ++++++ drivers/gpu/drm/xe/xe_vm.c | 66 +++++++++++++++++++++++++++++++- drivers/gpu/drm/xe/xe_vm_types.h | 8 ++++ 3 files changed, 84 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_pt_types.h +12 -0
@@ -74,4 +74,16 @@ struct xe_vm_pgtable_update { u32 flags; }; +/** struct xe_vm_pgtable_update_op - Page table update operation */ +struct xe_vm_pgtable_update_op { +/** @entries: entries to update for this operation */ +struct xe_vm_pgtable_update entries[XE_VM_MAX_LEVEL *2 +1 ]; +/** @num_entries: number of entries for this update operation */ +u32 num_entries; +/** @bind: is a bind */ +bool bind; +/** @rebind: is a rebind */ +bool rebind; +}; +#endif
drivers/gpu/drm/xe/xe_vm.c +64 -2
143 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm_types.h +8 -0
@@ -21,6 +21,7 @@ struct xe_bo;struct xe_sync_entry ;struct xe_user_fence ;struct xe_vm ; +struct xe_vm_pgtable_update_op ;#define XE_VMA_READ_ONLY DRM_GPUVA_USERBITS#define XE_VMA_DESTROYED (DRM_GPUVA_USERBITS << 1)
@@ -368,6 +369,13 @@ struct xe_vma_ops {struct xe_sync_entry *syncs;/** @num_syncs: number of syncs */ u32 num_syncs; +/** @pt_update_ops: page table update operations */ +struct { +/** @ops: operations */ +struct xe_vm_pgtable_update_op *ops; +/** @num_ops: number of operations */ +u32 num_ops; + } pt_update_ops[XE_MAX_TILES_PER_DEVICE]; };#endif
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 3/7 drm/xe: Add xe_exec_queue_last_fence_test_dep
In reply to Matthew BrostHelpful to determine if a bind can immediately use CPU or needs to be deferred a drm scheduler job.
v7:
- Better wording in kernel doc (Matthew Auld)
--- drivers/gpu/drm/xe/xe_exec_queue.c | 23 +++++++++++++++++++++++ drivers/gpu/drm/xe/xe_exec_queue.h | 2 ++ 2 files changed, 25 insertions(+)
drivers/gpu/drm/xe/xe_exec_queue.c +23 -0
@@ -906,3 +906,26 @@ void xe_exec_queue_last_fence_set(struct xe_exec_queue *q, struct xe_vm *vm,xe_exec_queue_last_fence_put (q, vm); q->last_fence =dma_fence_get (fence); } + +/** +* xe_exec_queue_last_fence_test_dep - Test last fence dependency of queue +* @q: The exec queue +* @vm: The VM the engine does a bind or exec for +* +* Returns: +* -ETIME if there exists an unsignalled last fence dependency, zero otherwise. +*/ +int xe_exec_queue_last_fence_test_dep (struct xe_exec_queue *q,struct xe_vm *vm) +{ +struct dma_fence *fence; +int err =0 ; + + fence =xe_exec_queue_last_fence_get (q, vm); +if (fence) { + err =test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags) ? +0 : -ETIME ; +dma_fence_put (fence); + } + +return err; +}
drivers/gpu/drm/xe/xe_exec_queue.h +2 -0
@@ -75,6 +75,8 @@ struct dma_fence *xe_exec_queue_last_fence_get(struct xe_exec_queue *e, struct xe_vm *vm);void xe_exec_queue_last_fence_set (struct xe_exec_queue *e,struct xe_vm *vm,struct dma_fence *fence); +int xe_exec_queue_last_fence_test_dep (struct xe_exec_queue *q, +struct xe_vm *vm);void xe_exec_queue_update_run_ticks (struct xe_exec_queue *q);#endif
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 4/7 drm/xe: Convert multiple bind ops into single job
In reply to Matthew BrostThis aligns with the uAPI of an array of binds or single bind that results in multiple GPUVA ops to be considered a single atomic operations.
The design is roughly:
- xe_vma_ops is a list of xe_vma_op (GPUVA op)
- each xe_vma_op resolves to 0-3 PT ops
- xe_vma_ops creates a single job
- if at any point during binding a failure occurs, xe_vma_ops contains the information necessary unwind the PT and VMA (GPUVA) state
v2:
- add missing dma-resv slot reservation (CI, testing)
v4:
- Fix TLB invalidation (Paulo)
- Add missing xe_sched_job_last_fence_add/test_dep check (Inspection)
v5:
- Invert i, j usage (Matthew Auld)
- Add helper to test and add job dep (Matthew Auld)
- Return on anything but -ETIME for cpu bind (Matthew Auld)
- Return -ENOBUFS if suballoc of BB fails due to size (Matthew Auld)
- s/do/Do (Matthew Auld)
- Add missing comma (Matthew Auld)
- Do not assign return value to xe_range_fence_insert (Matthew Auld)
v6:
- s/0x1ff/MAX_PTE_PER_SDI (Matthew Auld, CI)
- Check to large of SA in Xe to avoid triggering WARN (Matthew Auld)
- Fix checkpatch issues
v7:
- Rebase
- Support more than 510 PTEs updates in a bind job (Paulo, mesa testing)
--- drivers/gpu/drm/xe/xe_bo_types.h | 2 + drivers/gpu/drm/xe/xe_migrate.c | 333 +++++---- drivers/gpu/drm/xe/xe_migrate.h | 32 +- drivers/gpu/drm/xe/xe_pt.c | 1096 +++++++++++++++++++----------- drivers/gpu/drm/xe/xe_pt.h | 14 +- drivers/gpu/drm/xe/xe_pt_types.h | 36 + drivers/gpu/drm/xe/xe_sa.c | 7 + drivers/gpu/drm/xe/xe_vm.c | 519 +++----------- drivers/gpu/drm/xe/xe_vm.h | 2 + drivers/gpu/drm/xe/xe_vm_types.h | 45 +- 10 files changed, 1062 insertions(+), 1024 deletions(-)
drivers/gpu/drm/xe/xe_bo_types.h +2 -0
@@ -58,6 +58,8 @@ struct xe_bo {#endif /** @freed: List node for delayed put. */ struct llist_node freed; +/** @update_index: Update index if PT BO */ +int update_index;/** @created: Whether the bo has passed initial creation */ bool created;
drivers/gpu/drm/xe/xe_migrate.c +166 -167
442 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.h +20 -12
@@ -47,6 +47,24 @@ struct xe_migrate_pt_update_ops { struct xe_tile *tile, struct iosys_map *map, void *pos, u32 ofs, u32 num_qwords, const struct xe_vm_pgtable_update *update); +/** +* @clear: Clear a command buffer or page-table with ptes. +* @pt_update: Embeddable callback argument. +* @tile: The tile for the current operation. +* @map: struct iosys_map into the memory to be populated. +* @pos: If @map is NULL, map into the memory to be populated. +* @ofs: qword offset into @map, unused if @map is NULL. +* @num_qwords: Number of qwords to write. +* @update: Information about the PTEs to be inserted. +* +* This interface is intended to be used as a callback into the +* page-table system to populate command buffers or shared +* page-tables with PTEs. +*/ +void (*clear)(struct xe_migrate_pt_update *pt_update, +struct xe_tile *tile,struct iosys_map *map, +void *pos,u32 ofs,u32 num_qwords, +const struct xe_vm_pgtable_update *update);/** * @pre_commit: Callback to be called just before arming the
@@ -67,14 +85,10 @@ struct xe_migrate_pt_update_ops {struct xe_migrate_pt_update {/** @ops: Pointer to the struct xe_migrate_pt_update_ops callbacks */ const struct xe_migrate_pt_update_ops *ops; -/** @vma: The vma we're updating the pagetable for. */ -struct xe_vma *vma; +/** @vops: VMA operations */ +struct xe_vma_ops *vops;/** @job: The job if a GPU page-table update. NULL otherwise */ struct xe_sched_job *job; -/** @start: Start of update for the range fence */ -u64 start; -/** @last: Last of update for the range fence */ -u64 last;/** @tile_id: Tile ID of the update */ u8 tile_id; };
@@ -96,12 +110,6 @@ struct xe_vm *xe_migrate_get_vm(struct xe_migrate *m);struct dma_fence *xe_migrate_update_pgtables (struct xe_migrate *m, -struct xe_vm *vm, -struct xe_bo *bo, -struct xe_exec_queue *q, -const struct xe_vm_pgtable_update *updates, -u32 num_updates, -struct xe_sync_entry *syncs,u32 num_syncs,struct xe_migrate_pt_update *pt_update);void xe_migrate_wait (struct xe_migrate *m);
drivers/gpu/drm/xe/xe_pt.c +696 -400
1,250 lines not on this page · raw message
drivers/gpu/drm/xe/xe_pt.h +6 -8
@@ -17,6 +17,7 @@ struct xe_sync_entry;struct xe_tile ;struct xe_vm ;struct xe_vma ; +struct xe_vma_ops ;/* Largest huge pte is currently 1GiB. May become device dependent. */ #define MAX_HUGEPTE_LEVEL 2
@@ -34,14 +35,11 @@ void xe_pt_populate_empty(struct xe_tile *tile, struct xe_vm *vm,void xe_pt_destroy (struct xe_pt *pt,u32 flags,struct llist_head *deferred); -struct dma_fence * -__xe_pt_bind_vma (struct xe_tile *tile,struct xe_vma *vma,struct xe_exec_queue *q, -struct xe_sync_entry *syncs,u32 num_syncs, -bool rebind); - -struct dma_fence * -__xe_pt_unbind_vma (struct xe_tile *tile,struct xe_vma *vma,struct xe_exec_queue *q, -struct xe_sync_entry *syncs,u32 num_syncs); +int xe_pt_update_ops_prepare (struct xe_tile *tile,struct xe_vma_ops *vops); +struct dma_fence *xe_pt_update_ops_run (struct xe_tile *tile, +struct xe_vma_ops *vops); +void xe_pt_update_ops_fini (struct xe_tile *tile,struct xe_vma_ops *vops); +void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops);bool xe_pt_zap_ptes (struct xe_tile *tile,struct xe_vma *vma);
drivers/gpu/drm/xe/xe_pt_types.h +36 -0
@@ -78,6 +78,8 @@ struct xe_vm_pgtable_update {struct xe_vm_pgtable_update_op {/** @entries: entries to update for this operation */ struct xe_vm_pgtable_update entries[XE_VM_MAX_LEVEL *2 +1 ]; +/** @vma: VMA for operation, operation not valid if NULL */ +struct xe_vma *vma;/** @num_entries: number of entries for this update operation */ u32 num_entries;/** @bind: is a bind */
@@ -86,4 +88,38 @@ struct xe_vm_pgtable_update_op { bool rebind; }; +/** struct xe_vm_pgtable_update_ops: page table update operations */ +struct xe_vm_pgtable_update_ops { +/** @ops: operations */ +struct xe_vm_pgtable_update_op *ops; +/** @deferred: deferred list to destroy PT entries */ +struct llist_head deferred; +/** @q: exec queue for PT operations */ +struct xe_exec_queue *q; +/** @start: start address of ops */ +u64 start; +/** @last: last address of ops */ +u64 last; +/** @num_ops: number of operations */ +u32 num_ops; +/** @current_op: current operations */ +u32 current_op; +/** @needs_userptr_lock: Needs userptr lock */ +bool needs_userptr_lock; +/** @needs_invalidation: Needs invalidation */ +bool needs_invalidation; +/** +* @wait_vm_bookkeep: PT operations need to wait until VM is idle +* (bookkeep dma-resv slots are idle) and stage all future VM activity +* behind these operations (install PT operations into VM kernel +* dma-resv slot). +*/ +bool wait_vm_bookkeep; +/** +* @wait_vm_kernel: PT operations need to wait until VM kernel dma-resv +* slots are idle. +*/ +bool wait_vm_kernel; +}; +#endif
drivers/gpu/drm/xe/xe_sa.c +7 -0
@@ -84,6 +84,13 @@ struct xe_sa_manager *xe_sa_bo_manager_init(struct xe_tile *tile, u32 size, u32struct drm_suballoc *xe_sa_bo_new (struct xe_sa_manager *sa_manager,unsigned int size) { +/* +* BB to large, return -ENOBUFS indicating user should split +* array of binds into smaller chunks. +*/ +if (size > sa_manager->base.size) +return ERR_PTR (-ENOBUFS ); +return drm_suballoc_new (&sa_manager->base, size,GFP_KERNEL , true,0 ); }
drivers/gpu/drm/xe/xe_vm.c +113 -406
633 lines not on this page · raw message
drivers/gpu/drm/xe/xe_vm.h +2 -0
@@ -259,6 +259,8 @@ static inline struct dma_resv *xe_vm_resv(struct xe_vm *vm)return drm_gpuvm_resv (&vm->gpuvm); } +void xe_vm_kill (struct xe_vm *vm,bool unlocked); +/** * xe_vm_assert_held(vm) - Assert that the vm's reservation object is held. * @vm: The vm
drivers/gpu/drm/xe/xe_vm_types.h +14 -31
84 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 5/7 drm/xe: Update VM trace events
In reply to Matthew BrostThe trace events have changed moving to a single job per VM bind IOCTL, update the trace events align with old behavior as much as possible.
--- drivers/gpu/drm/xe/xe_trace_bo.h | 10 ++++---- drivers/gpu/drm/xe/xe_vm.c | 42 ++++++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 7 deletions(-)
drivers/gpu/drm/xe/xe_trace_bo.h +5 -5
@@ -117,11 +117,6 @@ DEFINE_EVENT(xe_vma, xe_vma_acc,TP_ARGS (vma) ); -DEFINE_EVENT (xe_vma, xe_vma_fail, - TP_PROTO(struct xe_vma *vma), -TP_ARGS (vma) -); -DEFINE_EVENT (xe_vma, xe_vma_bind, TP_PROTO(struct xe_vma *vma), TP_ARGS(vma)
@@ -237,6 +232,11 @@ DEFINE_EVENT(xe_vm, xe_vm_rebind_worker_exit,TP_ARGS (vm) ); +DEFINE_EVENT (xe_vm, xe_vm_ops_fail, + TP_PROTO(struct xe_vm *vm), +TP_ARGS (vm) +); +#endif /* This part must be outside protection */
drivers/gpu/drm/xe/xe_vm.c +40 -2
66 lines not on this page · raw message
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 6/7 drm/xe: Update PT layer with better error handling
In reply to Matthew BrostUpdate PT layer so if a memory allocation for a PTE fails the error can be propagated to the user without requiring the VM to be killed.
v5:
- change return value invalidation_fence_init to void (Matthew Auld)
v7:
- Invert i,j usage in two places (Matthew Auld)
- s/0/NULL (Matthew Auld)
- Don't ignore return value of xe_pt_new_shared (Matthew Auld)
- Don't check for NULL in xe_pt_entry (Matthew Auld)
--- drivers/gpu/drm/xe/xe_pt.c | 232 ++++++++++++++++++++++++++----------- 1 file changed, 167 insertions(+), 65 deletions(-)
drivers/gpu/drm/xe/xe_pt.c +167 -65
@@ -846,19 +846,27 @@ xe_vm_populate_pgtable(struct xe_migrate_pt_update *pt_update, struct xe_tile *t } } -static void xe_pt_abort_bind (struct xe_vma *vma, -struct xe_vm_pgtable_update *entries, -u32 num_entries) +static void xe_pt_cancel_bind (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries) {u32 i, j;for (i =0 ; i < num_entries; i++) { -if (!entries[i].pt_entries) +struct xe_pt *pt = entries[i].pt; + +if (!pt)continue ; -for (j =0 ; j < entries[i].qwords; j++) -xe_pt_destroy (entries[i].pt_entries[j].pt,xe_vma_vm (vma)->flags,NULL ); +if (pt->level) { +for (j =0 ; j < entries[i].qwords; j++) +xe_pt_destroy (entries[i].pt_entries[j].pt, +xe_vma_vm (vma)->flags,NULL ); + } +kfree (entries[i].pt_entries); + entries[i].pt_entries =NULL ; + entries[i].qwords =0 ; } }
@@ -874,10 +882,61 @@ static void xe_pt_commit_locks_assert(struct xe_vma *vma)xe_vm_assert_held (vm); } -static void xe_pt_commit_bind (struct xe_vma *vma, -struct xe_vm_pgtable_update *entries, -u32 num_entries,bool rebind, -struct llist_head *deferred) +static void xe_pt_commit (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries,struct llist_head *deferred) +{ +u32 i, j; + +xe_pt_commit_locks_assert (vma); + +for (i =0 ; i < num_entries; i++) { +struct xe_pt *pt = entries[i].pt; + +if (!pt->level) +continue ; + +for (j =0 ; j < entries[i].qwords; j++) { +struct xe_pt *oldpte = entries[i].pt_entries[j].pt; + +xe_pt_destroy (oldpte,xe_vma_vm (vma)->flags, deferred); + } + } +} + +static void xe_pt_abort_bind (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries,bool rebind) +{ +int i, j; + +xe_pt_commit_locks_assert (vma); + +for (i = num_entries -1 ; i >=0 ; --i) { +struct xe_pt *pt = entries[i].pt; +struct xe_pt_dir *pt_dir; + +if (!rebind) + pt->num_live -= entries[i].qwords; + +if (!pt->level) +continue ; + + pt_dir =as_xe_pt_dir (pt); +for (j =0 ; j < entries[i].qwords; j++) { +u32 j_ = j + entries[i].ofs; +struct xe_pt *newpte =xe_pt_entry (pt_dir, j_); +struct xe_pt *oldpte = entries[i].pt_entries[j].pt; + + pt_dir->children[j_] = oldpte ? &oldpte->base :0 ; +xe_pt_destroy (newpte,xe_vma_vm (vma)->flags,NULL ); + } + } +} + +static void xe_pt_commit_prepare_bind (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries,bool rebind) {u32 i, j;
@@ -897,12 +956,13 @@ static void xe_pt_commit_bind(struct xe_vma *vma,for (j =0 ; j < entries[i].qwords; j++) {u32 j_ = j + entries[i].ofs;struct xe_pt *newpte = entries[i].pt_entries[j].pt; +struct xe_pt *oldpte =NULL ;if (xe_pt_entry (pt_dir, j_)) -xe_pt_destroy (xe_pt_entry (pt_dir, j_), -xe_vma_vm (vma)->flags, deferred); + oldpte =xe_pt_entry (pt_dir, j_); pt_dir->children[j_] = &newpte->base; + entries[i].pt_entries[j].pt = oldpte; } } }
@@ -926,8 +986,6 @@ xe_pt_prepare_bind(struct xe_tile *tile, struct xe_vma *vma, err =xe_pt_stage_bind (tile, vma, entries, num_entries);if (!err)xe_tile_assert (tile, *num_entries); -else /* abort! */ -xe_pt_abort_bind (vma, entries, *num_entries);return err; }
@@ -1305,10 +1363,10 @@ static void invalidation_fence_work_func(struct work_struct *w) ifence->end, ifence->asid); } -static int invalidation_fence_init (struct xe_gt *gt, -struct invalidation_fence *ifence, -struct dma_fence *fence, -u64 start,u64 end,u32 asid) +static void invalidation_fence_init (struct xe_gt *gt, +struct invalidation_fence *ifence, +struct dma_fence *fence, +u64 start,u64 end,u32 asid) {int ret;
@@ -1340,8 +1398,6 @@ static int invalidation_fence_init(struct xe_gt *gt, }xe_gt_assert (gt, !ret || ret == -ENOENT ); - -return ret && ret != -ENOENT ? ret :0 ; }struct xe_pt_stage_unbind_walk {
@@ -1426,6 +1482,7 @@ xe_pt_stage_unbind_post_descend(struct xe_ptw *parent, pgoff_t offset,struct xe_pt *xe_child =container_of (*child,typeof (*xe_child), base);pgoff_t end_offset;u64 size =1ull << walk->shifts[--level]; +int err;if (!IS_ALIGNED (addr, size)) addr = xe_walk->modified_start;
@@ -1441,7 +1498,10 @@ xe_pt_stage_unbind_post_descend(struct xe_ptw *parent, pgoff_t offset, &end_offset))return 0 ; - (void )xe_pt_new_shared (&xe_walk->wupd, xe_child, offset, false); + err =xe_pt_new_shared (&xe_walk->wupd, xe_child, offset, true); +if (err) +return err; + xe_walk->wupd.updates[level].update->qwords = end_offset - offset;return 0 ;
@@ -1509,32 +1569,54 @@ xe_migrate_clear_pgtable_callback(struct xe_migrate_pt_update *pt_update,memset64 (ptr, empty, num_qwords); } +static void xe_pt_abort_unbind (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries) +{ +int i, j; + +xe_pt_commit_locks_assert (vma); + +for (i = num_entries -1 ; i >=0 ; --i) { +struct xe_vm_pgtable_update *entry = &entries[i]; +struct xe_pt *pt = entry->pt; +struct xe_pt_dir *pt_dir =as_xe_pt_dir (pt); + + pt->num_live += entry->qwords; + +if (!pt->level) +continue ; + +for (j = entry->ofs; j < entry->ofs + entry->qwords; j++) + pt_dir->children[i] = + entries[i].pt_entries[j - entry->ofs].pt ? + &entries[i].pt_entries[j - entry->ofs].pt->base :NULL ; + } +} +static void -xe_pt_commit_unbind (struct xe_vma *vma, -struct xe_vm_pgtable_update *entries,u32 num_entries, -struct llist_head *deferred) +xe_pt_commit_prepare_unbind (struct xe_vma *vma, +struct xe_vm_pgtable_update *entries, +u32 num_entries) { -u32 j; +int i, j;xe_pt_commit_locks_assert (vma); -for (j =0 ; j < num_entries; ++j) { -struct xe_vm_pgtable_update *entry = &entries[j]; +for (i =0 ; i < num_entries; ++i) { +struct xe_vm_pgtable_update *entry = &entries[i];struct xe_pt *pt = entry->pt; +struct xe_pt_dir *pt_dir; pt->num_live -= entry->qwords; -if (pt->level) { -struct xe_pt_dir *pt_dir =as_xe_pt_dir (pt); -u32 i; - -for (i = entry->ofs; i < entry->ofs + entry->qwords; - i++) { -if (xe_pt_entry (pt_dir, i)) -xe_pt_destroy (xe_pt_entry (pt_dir, i), -xe_vma_vm (vma)->flags, deferred); +if (!pt->level) +continue ; - pt_dir->children[i] =NULL ; - } + pt_dir =as_xe_pt_dir (pt); +for (j = entry->ofs; j < entry->ofs + entry->qwords; j++) { + entry->pt_entries[j - entry->ofs].pt = +xe_pt_entry (pt_dir, j); + pt_dir->children[j] =NULL ; } } }
@@ -1580,7 +1662,6 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile, {u32 current_op = pt_update_ops->current_op;struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; -struct llist_head *deferred = &pt_update_ops->deferred;int err;xe_bo_assert_held (xe_vma_bo (vma));
@@ -1628,11 +1709,12 @@ static int bind_op_prepare(struct xe_vm *vm, struct xe_tile *tile,/* We bump also if batch_invalidate_tlb is true */ vm->tlb_flush_seqno++; -/* FIXME: Don't commit right away */ vma->tile_staged |=BIT (tile->id); pt_op->vma = vma; -xe_pt_commit_bind (vma, pt_op->entries, pt_op->num_entries, - pt_op->rebind, deferred); +xe_pt_commit_prepare_bind (vma, pt_op->entries, + pt_op->num_entries, pt_op->rebind); + }else { +xe_pt_cancel_bind (vma, pt_op->entries, pt_op->num_entries); }return err;
@@ -1644,7 +1726,6 @@ static int unbind_op_prepare(struct xe_tile *tile, {u32 current_op = pt_update_ops->current_op;struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[current_op]; -struct llist_head *deferred = &pt_update_ops->deferred;int err;if (!((vma->tile_present | vma->tile_staged) &BIT (tile->id)))
@@ -1680,9 +1761,7 @@ static int unbind_op_prepare(struct xe_tile *tile, pt_update_ops->needs_userptr_lock |=xe_vma_is_userptr (vma); pt_update_ops->needs_invalidation = true; -/* FIXME: Don't commit right away */ -xe_pt_commit_unbind (vma, pt_op->entries, pt_op->num_entries, - deferred); +xe_pt_commit_prepare_unbind (vma, pt_op->entries, pt_op->num_entries);return 0 ; }
@@ -1903,7 +1982,7 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)struct invalidation_fence *ifence =NULL ;struct xe_range_fence *rfence;struct xe_vma_op *op; -int err =0 ; +int err =0 , i;struct xe_migrate_pt_update update = { .ops = pt_update_ops->needs_userptr_lock ? &userptr_migrate_ops :
@@ -1923,8 +2002,10 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)if (pt_update_ops->needs_invalidation) { ifence =kzalloc (sizeof (*ifence),GFP_KERNEL ); -if (!ifence) -return ERR_PTR (-ENOMEM ); +if (!ifence) { + err = -ENOMEM ; + goto kill_vm_tile1; + } } rfence =kzalloc (sizeof (*rfence),GFP_KERNEL );
@@ -1939,6 +2020,15 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops) goto free_rfence; } +/* Point of no return - VM killed if failure after this */ +for (i =0 ; i < pt_update_ops->current_op; ++i) { +struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; + +xe_pt_commit (pt_op->vma, pt_op->entries, + pt_op->num_entries, &pt_update_ops->deferred); + pt_op->vma =NULL ;/* skip in xe_pt_update_ops_abort */ + } + if (xe_range_fence_insert(&vm->rftree[tile->id], rfence, &xe_range_fence_kfree_ops, pt_update_ops->start,
@@ -1947,12 +2037,9 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)/* tlb invalidation must be done before signaling rebind */ if (ifence) { - err =invalidation_fence_init (tile->primary_gt, ifence, fence, - pt_update_ops->start, - pt_update_ops->last, - vm->usm.asid); -if (err) - goto put_fence; +invalidation_fence_init (tile->primary_gt, ifence, fence, + pt_update_ops->start, + pt_update_ops->last, vm->usm.asid); fence = &ifence->base.base; }
@@ -1969,14 +2056,13 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)return fence; -put_fence: -if (pt_update_ops->needs_userptr_lock) -up_read (&vm->userptr.notifier_lock); -dma_fence_put (fence); free_rfence:kfree (rfence); free_ifence:kfree (ifence); +kill_vm_tile1: +if (err != -EAGAIN && tile->id) +xe_vm_kill (vops->vm, false);return ERR_PTR (err); }
@@ -1997,12 +2083,10 @@ void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops)lockdep_assert_held (&vops->vm->lock);xe_vm_assert_held (vops->vm); -/* FIXME: Not 100% correct */ -for (i =0 ; i < pt_update_ops->num_ops; ++i) { +for (i =0 ; i < pt_update_ops->current_op; ++i) {struct xe_vm_pgtable_update_op *pt_op = &pt_update_ops->ops[i]; -if (pt_op->bind) -xe_pt_free_bind (pt_op->entries, pt_op->num_entries); +xe_pt_free_bind (pt_op->entries, pt_op->num_entries); }xe_bo_put_commit (&vops->pt_update_ops[tile->id].deferred); }
@@ -2016,10 +2100,28 @@ void xe_pt_update_ops_fini(struct xe_tile *tile, struct xe_vma_ops *vops)*/ void xe_pt_update_ops_abort (struct xe_tile *tile,struct xe_vma_ops *vops) { +struct xe_vm_pgtable_update_ops *pt_update_ops = + &vops->pt_update_ops[tile->id]; +int i; +lockdep_assert_held (&vops->vm->lock);xe_vm_assert_held (vops->vm); -/* FIXME: Just kill VM for now + cleanup PTs */ +for (i = pt_update_ops->num_ops -1 ; i >=0 ; --i) { +struct xe_vm_pgtable_update_op *pt_op = + &pt_update_ops->ops[i]; + +if (!pt_op->vma || i >= pt_update_ops->current_op) +continue ; + +if (pt_op->bind) +xe_pt_abort_bind (pt_op->vma, pt_op->entries, + pt_op->num_entries, + pt_op->rebind); +else +xe_pt_abort_unbind (pt_op->vma, pt_op->entries, + pt_op->num_entries); + } +xe_bo_put_commit (&vops->pt_update_ops[tile->id].deferred); -xe_vm_kill (vops->vm, false); }
Signature · 1 line
2.34.1
Matthew Brost
maintainerPATCH v7 7/7 drm/xe: Add VM bind IOCTL error injection
In reply to Matthew BrostAdd VM bind IOCTL error injection which steals MSB of the bind flags field which if set injects errors at various points in the VM bind IOCTL. Intended to validate error paths. Enabled by CONFIG_DRM_XE_DEBUG.
v4:
- Change define layout (Jonathan)
--- drivers/gpu/drm/xe/xe_device_types.h | 12 ++++++++++++ drivers/gpu/drm/xe/xe_pt.c | 12 ++++++++++++ drivers/gpu/drm/xe/xe_vm.c | 24 +++++++++++++++++++++++- drivers/gpu/drm/xe/xe_vm_types.h | 14 ++++++++++++++ 4 files changed, 61 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_device_types.h +12 -0
@@ -23,6 +23,10 @@#include "xe_sriov_types.h" #include "xe_step_types.h" +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG ) +#define TEST_VM_OPS_ERROR +#endif +#if IS_ENABLED (CONFIG_DRM_XE_DISPLAY )#include "soc/intel_pch.h" #include "intel_display_core.h"
@@ -477,6 +481,14 @@ struct xe_device {int mode; } wedged; +#ifdef TEST_VM_OPS_ERROR +/** +* @vm_inject_error_position: inject errors at different places in VM +* bind IOCTL based on this value +*/ +u8 vm_inject_error_position; +#endif +/* private: */ #if IS_ENABLED (CONFIG_DRM_XE_DISPLAY )
drivers/gpu/drm/xe/xe_pt.c +12 -0
@@ -1860,6 +1860,12 @@ int xe_pt_update_ops_prepare(struct xe_tile *tile, struct xe_vma_ops *vops)xe_tile_assert (tile, pt_update_ops->current_op <= pt_update_ops->num_ops); +#ifdef TEST_VM_OPS_ERROR +if (vops->inject_error && + vops->vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_PREPARE ) +return -ENOSPC ; +#endif +return 0 ; }
@@ -2000,6 +2006,12 @@ xe_pt_update_ops_run(struct xe_tile *tile, struct xe_vma_ops *vops)return dma_fence_get_stub (); } +#ifdef TEST_VM_OPS_ERROR +if (vops->inject_error && + vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_RUN ) +return ERR_PTR (-ENOSPC ); +#endif +if (pt_update_ops->needs_invalidation) { ifence =kzalloc (sizeof (*ifence),GFP_KERNEL );if (!ifence) {
drivers/gpu/drm/xe/xe_vm.c +23 -1
@@ -2468,6 +2468,12 @@ static int vm_bind_ioctl_ops_lock_and_prep(struct drm_exec *exec,return err; } +#ifdef TEST_VM_OPS_ERROR +if (vops->inject_error && + vm->xe->vm_inject_error_position ==FORCE_OP_ERROR_LOCK ) +return -ENOSPC ; +#endif +return 0 ; }
@@ -2704,11 +2710,18 @@ static int vm_bind_ioctl_ops_execute(struct xe_vm *vm,return err; } -#define SUPPORTED_FLAGS \ +#define SUPPORTED_FLAGS_STUB \ (DRM_XE_VM_BIND_FLAG_READONLY | \ DRM_XE_VM_BIND_FLAG_IMMEDIATE | \ DRM_XE_VM_BIND_FLAG_NULL | \ DRM_XE_VM_BIND_FLAG_DUMPABLE) + +#ifdef TEST_VM_OPS_ERROR +#define SUPPORTED_FLAGS (SUPPORTED_FLAGS_STUB | FORCE_OP_ERROR) +#else +#define SUPPORTED_FLAGS SUPPORTED_FLAGS_STUB +#endif +#define XE_64K_PAGE_MASK 0xffffull#define ALL_DRM_XE_SYNCS_FLAGS (DRM_XE_SYNCS_FLAG_WAIT_FOR_OP)
@@ -3056,6 +3069,15 @@ int xe_vm_bind_ioctl(struct drm_device *dev, void *data, struct drm_file *file) err =vm_bind_ioctl_ops_parse (vm, ops[i], &vops);if (err) goto unwind_ops; + +#ifdef TEST_VM_OPS_ERROR +if (flags &FORCE_OP_ERROR ) { + vops.inject_error = true; + vm->xe->vm_inject_error_position = + (vm->xe->vm_inject_error_position +1 ) % +FORCE_OP_ERROR_COUNT ; + } +#endif }/* Nothing to do */
drivers/gpu/drm/xe/xe_vm_types.h +14 -0
@@ -23,6 +23,16 @@ struct xe_user_fence;struct xe_vm ;struct xe_vm_pgtable_update_op ; +#if IS_ENABLED (CONFIG_DRM_XE_DEBUG ) +#define TEST_VM_OPS_ERROR +#define FORCE_OP_ERROR BIT(31) + +#define FORCE_OP_ERROR_LOCK 0 +#define FORCE_OP_ERROR_PREPARE 1 +#define FORCE_OP_ERROR_RUN 2 +#define FORCE_OP_ERROR_COUNT 3 +#endif +#define XE_VMA_READ_ONLY DRM_GPUVA_USERBITS#define XE_VMA_DESTROYED (DRM_GPUVA_USERBITS << 1)#define XE_VMA_ATOMIC_PTE_BIT (DRM_GPUVA_USERBITS << 2)
@@ -359,6 +369,10 @@ struct xe_vma_ops {u32 num_syncs;/** @pt_update_ops: page table update operations */ struct xe_vm_pgtable_update_ops pt_update_ops[XE_MAX_TILES_PER_DEVICE ]; +#ifdef TEST_VM_OPS_ERROR +/** @inject_error: inject error to test error handling */ +bool inject_error; +#endif };#endif
Signature · 1 line
2.34.1
Patchwork
CI✗ CI.Patch_applied: failure for Convert multiple bind ops to 1 job (rev7)
In reply to Matthew Brost== Series Details ==
== Series Details ==
Series: Convert multiple bind ops to 1 job (rev7)
URL : https://patchwork.freedesktop.org/series/133034/
State : failure
== Summary ==
=== Applying kernel patches on branch 'drm-tip' with base: ===
Base commit: 73af7ac6d6ae drm-tip: 2024y-07m-03d-23h-50m-41s UTC integration manifest === git am output follows ===
.git/rebase-apply/patch:248: trailing whitespace.
batch_size += 6 * (num_updates / MAX_PTE_PER_SDI + 1)
error: patch failed: drivers/gpu/drm/xe/xe_vm.c:311
error: drivers/gpu/drm/xe/xe_vm.c: patch does not apply
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Applying: drm/xe: s/xe_tile_migrate_engine/xe_tile_migrate_exec_queue
Applying: drm/xe: Add xe_vm_pgtable_update_op to xe_vma_ops
Applying: drm/xe: Add xe_exec_queue_last_fence_test_dep
Applying: drm/xe: Convert multiple bind ops into single job Patch failed at 0004 drm/xe: Convert multiple bind ops into single job When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".