drm/xe/vf: New approach for post-migration LRC fixups
Subsystems (2)
v4 cover drm/xe/vf: New approach for post-migration LRC fixups
The current approach to LRC recovery requires caution from developers when modifying any LRC-related code. More flexible solution may allow to avoid future troubles.
The current code also includes LRC creation repeat which looks a bit suspicious - it was developed without detailed knowledge of which part of LRC creation requires a repeat. Now after extensive debug we know that the repeat is only required for BO mapping, and actually all BOs should be protected, not only the ones storing LRCs.
This fixes a very rare bug where BO would be incorrectly created during migration resulting in CAT error generated by GPU. Chances of triggering this error were well below 0.1%.
v2: A rebase after 6 months, with adjustments to current source.
v3: Re-adding a check carelessly removed in v2, fixed handling
GGTT start/end, fixed issue with signals.
v4: Several fixes to error paths (mutex creation, and prolonged
migration recovery while BO mapping). Altered BO insertion to not repeat placement in GGTT upon migration, only repeat the mapping.
Tomasz Lis (2):
drm/xe/vf: Late fixups of LRCs after VF migration drm/xe: After VF migration, repeat BO mapping in progress
drivers/gpu/drm/xe/xe_exec_queue.c | 63 +++++++++++++---------- drivers/gpu/drm/xe/xe_exec_queue.h | 2 +- drivers/gpu/drm/xe/xe_ggtt.c | 54 ++++++++++++++++--- drivers/gpu/drm/xe/xe_gt_sriov_vf.c | 18 ++++--- drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h | 2 + drivers/gpu/drm/xe/xe_guc_submit.c | 35 +++---------- drivers/gpu/drm/xe/xe_guc_submit.h | 2 - drivers/gpu/drm/xe/xe_memirq.c | 10 +++- 8 files changed, 109 insertions(+), 77 deletions(-)
Signature · 1 line
2.43.0
PATCH v4 1/2 drm/xe/vf: Late fixups of LRCs after VF migration
For contexts not registered to GuC, do not fix LRCs during VF post-migration recovery. Instead, perform fixups before every exec queue registration.
This introduces a minor extra work during context registration, but also decreases the amount of work required to be finished during post-migration recovery.
To mitigate issues with concurrent registrations, the scratch buffer required for LRC fixups is now protected by a mutex.
v2: Just a rebase - getting back to the subject after 6 months.
v3: Use drm-managed mutex lifecycle.
v4: Protect from mutex creation fail; make more precise description.
--- drivers/gpu/drm/xe/xe_exec_queue.c | 28 +++++++++++++++--- drivers/gpu/drm/xe/xe_exec_queue.h | 2 +- drivers/gpu/drm/xe/xe_gt_sriov_vf.c | 18 +++++++----- drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h | 2 ++ drivers/gpu/drm/xe/xe_guc_submit.c | 35 ++++------------------- drivers/gpu/drm/xe/xe_guc_submit.h | 2 -- 6 files changed, 44 insertions(+), 43 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +24 -4
@@ -1904,15 +1904,14 @@ void xe_exec_queue_tlb_inval_last_fence_set(struct xe_exec_queue *q, q->tlb_inval[idx].last_fence =dma_fence_get (fence); } -/** -* xe_exec_queue_contexts_hwsp_rebase - Re-compute GGTT references -* within all LRCs of a queue. +/* +* Re-compute GGTT references within all LRCs of a queue. * @q: the &xe_exec_queue struct instance containing target LRCs * @scratch: scratch buffer to be used as temporary storage * * Returns: zero on success, negative error code on failure */ -int xe_exec_queue_contexts_hwsp_rebase (struct xe_exec_queue *q,void *scratch) +static int xe_exec_queue_lrcs_hwsp_rebase_with_scratch (struct xe_exec_queue *q,void *scratch) {int i;int err =0 ;
@@ -1934,3 +1933,24 @@ int xe_exec_queue_contexts_hwsp_rebase(struct xe_exec_queue *q, void *scratch)return err; } + +/** +* xe_exec_queue_lrcs_hwsp_rebase - Re-compute GGTT references +* within all LRCs of a queue, using migration scratch. +* @q: the &xe_exec_queue struct instance containing target LRCs +* +* Returns: zero on success, negative error code on failure +*/ +int xe_exec_queue_lrcs_hwsp_rebase (struct xe_exec_queue *q) +{ +struct xe_gt *gt = q->gt; +int err; + +xe_gt_assert (gt,IS_SRIOV_VF (gt_to_xe (gt))); + +mutex_lock (>->sriov.vf.migration.scratch_lock); + err =xe_exec_queue_lrcs_hwsp_rebase_with_scratch (q, gt->sriov.vf.migration.scratch); +mutex_unlock (>->sriov.vf.migration.scratch_lock); + +return err; +}
drivers/gpu/drm/xe/xe_exec_queue.h +1 -1
@@ -158,7 +158,7 @@ void xe_exec_queue_tlb_inval_last_fence_set(struct xe_exec_queue *q,void xe_exec_queue_update_run_ticks (struct xe_exec_queue *q); -int xe_exec_queue_contexts_hwsp_rebase (struct xe_exec_queue *q,void *scratch); +int xe_exec_queue_lrcs_hwsp_rebase (struct xe_exec_queue *q);struct xe_lrc *xe_exec_queue_lrc (struct xe_exec_queue *q);struct xe_lrc *xe_exec_queue_get_lrc (struct xe_exec_queue *q,u16 idx);
drivers/gpu/drm/xe/xe_gt_sriov_vf.c +11 -7
@@ -1319,7 +1319,6 @@ static size_t post_migration_scratch_size(struct xe_device *xe)static int vf_post_migration_fixups (struct xe_gt *gt) { -void *buf = gt->sriov.vf.migration.scratch;int err;VF_MIGRATION_INJECT_WAIT (gt,FIXUPS );
@@ -1333,9 +1332,6 @@ static int vf_post_migration_fixups(struct xe_gt *gt)xe_sriov_vf_ccs_rebase (gt_to_xe (gt));xe_gt_sriov_vf_default_lrcs_hwsp_rebase (gt); - err =xe_guc_contexts_hwsp_rebase (>->uc.guc, buf); -if (err) -return err;atomic_inc (>->sriov.vf.migration.fixups_complete_count);
@@ -1498,17 +1494,25 @@ static void vf_migration_fini(void *arg)*/ int xe_gt_sriov_vf_init_early (struct xe_gt *gt) { +struct xe_device *xe =gt_to_xe (gt);void *buf; +int err; -if (!xe_sriov_vf_migration_supported (gt_to_xe (gt))) +if (!xe_sriov_vf_migration_supported (xe))return 0 ; - buf =drmm_kmalloc (>_to_xe (gt)->drm, -post_migration_scratch_size (gt_to_xe (gt)), + buf =drmm_kmalloc (&xe->drm, +post_migration_scratch_size (xe),GFP_KERNEL );if (!buf)return -ENOMEM ; + err =drmm_mutex_init (&xe->drm, >->sriov.vf.migration.scratch_lock); +if (err) { +drmm_kfree (&xe->drm, buf); +return err; + } + gt->sriov.vf.migration.scratch = buf;spin_lock_init (>->sriov.vf.migration.lock);INIT_WORK (>->sriov.vf.migration.worker, migration_worker_func);
drivers/gpu/drm/xe/xe_gt_sriov_vf_types.h +2 -0
@@ -56,6 +56,8 @@ struct xe_gt_sriov_vf_migration {spinlock_t lock;/** @wq: wait queue for migration fixes */ wait_queue_head_t wq; +/** @scratch_lock: Mutex lock to serilize scratch buffer use */ +struct mutex scratch_lock;/** @scratch: Scratch memory for VF recovery */ void *scratch;/** @fixups_complete_count: Counts completed fixups stages */
drivers/gpu/drm/xe/xe_guc_submit.c +6 -29
@@ -47,6 +47,7 @@#include "xe_ring_ops_types.h" #include "xe_sched_job.h" #include "xe_sleep.h" +#include "xe_sriov_vf.h" #include "xe_tlb_inval.h" #include "xe_trace.h" #include "xe_uc_fw.h"
@@ -1035,6 +1036,9 @@ static void register_exec_queue(struct xe_exec_queue *q, int ctx_type)xe_gt_assert (guc_to_gt (guc), !exec_queue_registered (q));xe_gt_assert (guc_to_gt (guc), ctx_type <GUC_CONTEXT_COUNT ); +if (IS_SRIOV_VF (xe) &&xe_sriov_vf_migration_supported (xe)) +xe_exec_queue_lrcs_hwsp_rebase (q); +memset (&info,0 ,sizeof (info)); info.context_idx = q->guc->id; info.engine_class =xe_hwe_to_guc_class (q->hwe);
@@ -3152,6 +3156,8 @@ static void guc_exec_queue_unpause_prepare(struct xe_guc *guc,struct drm_sched_job *s_job;bool restore_replay = false; +if (exec_queue_registered (q)) +xe_exec_queue_lrcs_hwsp_rebase (q);drm_sched_for_each_pending_job (s_job, &sched->base,NULL ) { job =to_xe_sched_job (s_job); restore_replay |= job->restore_replay;
@@ -4099,32 +4105,3 @@ u32 xe_guc_submit_active_multi_queue_lrca(struct xe_guc *guc,return lrca; } - -/** -* xe_guc_contexts_hwsp_rebase - Re-compute GGTT references within all -* exec queues registered to given GuC. -* @guc: the &xe_guc struct instance -* @scratch: scratch buffer to be used as temporary storage -* -* Returns: zero on success, negative error code on failure. -*/ -int xe_guc_contexts_hwsp_rebase (struct xe_guc *guc,void *scratch) -{ -struct xe_exec_queue *q; -unsigned long index; -int err =0 ; - -mutex_lock (&guc->submission_state.lock); -xa_for_each (&guc->submission_state.exec_queue_lookup, index, q) { -/* Prevent redundant attempts to stop parallel queues */ -if (q->guc->id != index) -continue ; - - err =xe_exec_queue_contexts_hwsp_rebase (q, scratch); -if (err) -break ; - } -mutex_unlock (&guc->submission_state.lock); - -return err; -}
drivers/gpu/drm/xe/xe_guc_submit.h +0 -2
@@ -60,6 +60,4 @@ u32 xe_guc_submit_active_multi_queue_lrca(struct xe_guc *guc,struct xe_hw_engine *hwe, u32 cur_lrca, u32 active_id); -int xe_guc_contexts_hwsp_rebase (struct xe_guc *guc,void *scratch); -#endif
Signature · 1 line
2.43.0
PATCH v4 2/2 drm/xe: After VF migration, repeat BO mapping in progress
In reply to Tomasz LisIf VF migration happens during BO mapping, it is possible that some PTE values were not written at the correct place. This can happen because GGTT range, and therefore range of assigned PTEs, have changed during the migration. Restoring VF state on PF side will ensure that any PTEs already set within the previous VM are moved to correct positions, but if VM was paused and migrated while setting PTEs, it will continue the loop at the old position. The GGTT base address is used to select which PTEs to write, so setting them again needs to wait until the post-migration recovery procedure updates the GGTT base.
Previously, LRC creation was repeated. This means only BOs that were part of exec queues were properly protected from VF migration. With this change, as the redo is moved to BO creation, all BOs are protected.
v2: Removed memirq safety check - looking at current memirq init,
no extra check should be necessary.
v3: Re-added memirq check. Fixed multiple subtractions of
GGTT range bounds. Added clear of GGTT entries before retry. Handled issue with signals while inserting GGTT entries.
v4: Do not re-reserve a node after VF migration was detected,
just set all PTEs again. Comment on why the percieved infinite loop is impossible.
--- drivers/gpu/drm/xe/xe_exec_queue.c | 35 ++++++------------- drivers/gpu/drm/xe/xe_ggtt.c | 54 +++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_memirq.c | 10 ++++-- 3 files changed, 65 insertions(+), 34 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +11 -24
@@ -432,34 +432,21 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)* post-migration recovery, as the fixups for the LRC GGTT addresses * depend on the queue being present in the backend tracking structure. * -* In addition to above, we must wait on inflight GGTT changes to avoid -* writing out stale values here. Such wait provides a solid solution -* (without a race) only if the function can detect migration instantly -* from the moment vCPU resumes execution. +* In addition to above, we must make sure inflight GGTT changes are +* properly handled (with wait and/or repeat) to avoid writing out stale +* values here; but that is handled deeper, within BO pinning code. */ for (i =0 ; i < q->width; ++i) { -struct xe_lrc *__lrc =NULL ; -int marker; - -do { -struct xe_lrc *lrc; - - marker =xe_gt_sriov_vf_wait_valid_ggtt (q->gt); - - lrc =xe_lrc_create (q->hwe, q->vm, q->replay_state,xe_lrc_ring_size (), - q->msix_vec,xe_lrc_init_flags (q, exec_queue_flags)); -if (IS_ERR (lrc)) { - err =PTR_ERR (lrc); - goto err_lrc; - } - -xe_exec_queue_set_lrc (q, lrc, i); +struct xe_lrc *lrc; -if (__lrc) -xe_lrc_put (__lrc); - __lrc = lrc; + lrc =xe_lrc_create (q->hwe, q->vm, q->replay_state,xe_lrc_ring_size (), + q->msix_vec,xe_lrc_init_flags (q, exec_queue_flags)); +if (IS_ERR (lrc)) { + err =PTR_ERR (lrc); + goto err_lrc; + } - }while (marker !=xe_vf_migration_fixups_complete_count (q->gt)); +xe_exec_queue_set_lrc (q, lrc, i); }return 0 ;
drivers/gpu/drm/xe/xe_ggtt.c +46 -8
101 lines not on this page · raw message
drivers/gpu/drm/xe/xe_memirq.c +8 -2
@@ -524,8 +524,14 @@ bool xe_memirq_guc_sw_int_0_irq_pending(struct xe_memirq *memirq, struct xe_guc {struct xe_gt *gt =guc_to_gt (guc);u32 offset =xe_gt_is_media_type (gt) ?ilog2 (INTR_MGUC ) :ilog2 (INTR_GUC ); -struct iosys_map map =IOSYS_MAP_INIT_OFFSET (&memirq->bo->vmap, -XE_MEMIRQ_VECTOR_OFFSET (0 , offset)); +struct iosys_map map; + +/* protect for a call during driver probe */ +if (!iosys_map_is_set (&memirq->source)) +return false; + + map =IOSYS_MAP_INIT_OFFSET (&memirq->bo->vmap, +XE_MEMIRQ_VECTOR_OFFSET (0 , offset));return memirq_received_noclear (memirq, &map,ilog2 (GUC_INTR_SW_INT_0 ),guc_name (guc));
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v4 2/2] drm/xe: After VF migration, repeat BO mapping in progress
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] drm/xe: dead -ERESTARTSYS check and busy-spin in __xe_ggtt_insert_bo_at() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] drm/xe: dead -ERESTARTSYS check and busy-spin in __xe_ggtt_insert_bo_at()
--
commit b16142a275191b42c8c4d7d66be1c8cd542936f0
Author: Tomasz Lis <[email protected]>
drm/xe: After VF migration, repeat BO mapping in progress
This commit moves the VF post-migration retry loop from LRC creation in __xe_exec_queue_init() to GGTT BO insertion in __xe_ggtt_insert_bo_at() so that all GGTT BO mappings rewrite their PTEs if a VF migration occurs while mapping is in progress.
Quoted 4 lines
drivers/gpu/drm/xe/xe_ggtt.c +0
-0
[ ... ]
Quoted 25 lines
drivers/gpu/drm/xe/xe_ggtt.c +21
-0@@ -1080,6 +1086,38 @@ static int __xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo, }mutex_unlock (&ggtt->lock); +if (!err) { +/* +* If VF migration happens during BO mapping, some PTE value writes may +* be ignored by HW. This can happen in case GGTT range changed; already +* set PTEs are moved by the VF restore, but xe_ggtt_map_bo() within VM +* may still loop at old position. To be sure, redo all PTE writes. +* Make this a loop as another VF migration could start immediately. +*/ +while (marker !=xe_vf_migration_fixups_complete_count (ggtt->tile->primary_gt)) { +u16 cache_mode = bo->flags &XE_BO_FLAG_NEEDS_UC ? +XE_CACHE_NONE :XE_CACHE_WB ; +u16 pat_index =xe_cache_pat_idx (tile_to_xe (ggtt->tile), cache_mode); +u64 pte = ggtt->pt_ops->pte_encode_flags (bo, pat_index); + + marker =xe_gt_sriov_vf_wait_valid_ggtt (ggtt->tile->primary_gt); +/* +* Only abort fixing the PTEs if we're signalled or wedged; in +* case of timeout the migration recovery may be just repeated +* or prolonged, so do not treat that as bailout condition. +* If the VF migration was detected, we can be sure that recovery +* will start or already has started. The recovery worker assures
drivers/gpu/drm/xe/xe_ggtt.c static int __xe_ggtt_insert_bo_at(struct xe_ggtt *ggtt, struct xe_bo *bo, · line 1119
Quoted 10 lines
drivers/gpu/drm/xe/xe_ggtt.c +10
-0+* on finish that this loop will exit. +*/ +if (marker == -ERESTARTSYS ||xe_wedged (tile_to_xe (ggtt->tile))) +break ; + +mutex_lock (&ggtt->lock); +xe_ggtt_map_bo (ggtt, bo->ggtt_node[tile_id], bo, pte); +mutex_unlock (&ggtt->lock); + } + }
[Severity: High]
Can marker == -ERESTARTSYS ever be true here, or will a pending signal during VF migration cause this loop to busy-spin?
xe_gt_sriov_vf_wait_valid_ggtt() discards the return value from wait_event_interruptible_timeout() and always returns the non-negative fixups_complete_count:
drivers/gpu/drm/xe/xe_gt_sriov_vf.c:xe_gt_sriov_vf_wait_valid_ggtt() {
...
ret = wait_event_interruptible_timeout(gt->sriov.vf.migration.wq,
vf_valid_ggtt(gt),
HZ * 5);
xe_gt_WARN_ON(gt, !ret);
return atomic_read(>->sriov.vf.migration.fixups_complete_count);
}Meanwhile, xe_vf_migration_fixups_complete_count() returns -1 while !vf_valid_ggtt(gt). If a thread has a pending signal during VF migration recovery, wait_event_interruptible_timeout() returns -ERESTARTSYS immediately without sleeping, marker receives the non-negative fixups_complete_count, and the loop spins without sleeping while repeatedly acquiring ggtt->lock and rewriting all BO PTEs over MMIO. On a single-vCPU guest VM without involuntary preemption, this can starve the migration recovery worker and cause a soft lockup.
Also, when the loop breaks out (on xe_wedged(), or if -ERESTARTSYS were propagated), err is still 0 and bo->ggtt_node[tile_id] remains inserted, so __xe_ggtt_insert_bo_at() returns success with stale or unfixed GGTT PTEs.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2