Introduce Xe PCIe FLR
Subsystems (2)
v2 cover Introduce Xe PCIe FLR
Here's my humble attempt at introducing PCIe Function Level Reset (FLR) support in xe driver. This is ofcourse a half baked implementation and only limited to reloading GT. This needs to be extended for a lot of different components which I've skipped here for my lack of competence, so feel free to join in and support them.
PS: All xe_exec_basic tests and clpeak run smoothly after FLR. Give it a spin and let me know if any regressions.
Trigger it with:
$ echo 1 > /sys/bus/pci/devices/<BDF>/reset
v2: Re-initialize migrate context (Matthew Brost)
Add kernel doc (Matthew Brost)
Spell out Function Level Reset (Jani)
Raag Jadav (9):
drm/xe/uc_fw: Allow re-initializing firmware drm/xe/gt: Introduce FLR helpers drm/xe/irq: Introduce xe_irq_disable() drm/xe: Introduce xe_device_assert_lmem_ready() drm/xe/bo_evict: Introduce xe_bo_restore_map() drm/xe/lrc: Introduce xe_lrc_reinit() drm/xe/exec_queue: Introduce xe_exec_queue_reinit() drm/xe/migrate: Introduce xe_migrate_reinit() drm/xe/pci: Introduce PCIe FLR
drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_bo_evict.c | 51 ++++++++-- drivers/gpu/drm/xe/xe_bo_evict.h | 2 + drivers/gpu/drm/xe/xe_device.c | 10 +- drivers/gpu/drm/xe/xe_device.h | 1 + drivers/gpu/drm/xe/xe_exec_queue.c | 34 ++++++- drivers/gpu/drm/xe/xe_exec_queue.h | 1 + drivers/gpu/drm/xe/xe_gsc.c | 14 +++ drivers/gpu/drm/xe/xe_gsc.h | 1 + drivers/gpu/drm/xe/xe_gt.c | 32 ++++++ drivers/gpu/drm/xe/xe_gt.h | 2 + drivers/gpu/drm/xe/xe_guc.c | 14 +++ drivers/gpu/drm/xe/xe_guc.h | 1 + drivers/gpu/drm/xe/xe_huc.c | 14 +++ drivers/gpu/drm/xe/xe_huc.h | 1 + drivers/gpu/drm/xe/xe_irq.c | 13 ++- drivers/gpu/drm/xe/xe_irq.h | 1 + drivers/gpu/drm/xe/xe_lrc.c | 149 ++++++++++++++++------------ drivers/gpu/drm/xe/xe_lrc.h | 2 + drivers/gpu/drm/xe/xe_migrate.c | 65 ++++++++++--- drivers/gpu/drm/xe/xe_migrate.h | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 +++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_uc.c | 36 +++++++ drivers/gpu/drm/xe/xe_uc.h | 2 + drivers/gpu/drm/xe/xe_uc_fw.c | 39 ++++++-- drivers/gpu/drm/xe/xe_uc_fw.h | 1 + 28 files changed, 544 insertions(+), 97 deletions(-) create mode 100644 drivers/gpu/drm/xe/xe_pci_err.c
Signature · 1 line
2.43.0
PATCH v2 1/9 drm/xe/uc_fw: Allow re-initializing firmware
In preparation of usecases which require re-initializing firmware without reloading the driver, introduce xe_uc_fw_reinit(). The uC firmware bo already exists but since it's contents are on VRAM, they are lost on PCIe FLR. Copy the firmware back to it's bo and mark it as loadable as part of re-initialization.
---
v2: Add kernel doc (Matthew Brost)
--- drivers/gpu/drm/xe/xe_uc_fw.c | 39 +++++++++++++++++++++++++++++------ drivers/gpu/drm/xe/xe_uc_fw.h | 1 + 2 files changed, 34 insertions(+), 6 deletions(-)
drivers/gpu/drm/xe/xe_uc_fw.c +33 -6
@@ -704,13 +704,7 @@ static int uc_fw_request(struct xe_uc_fw *uc_fw, const struct firmware **firmwarconst struct firmware *fw =NULL ;int err; -/* -* we use FIRMWARE_UNINITIALIZED to detect checks against uc_fw->status -* before we're looked at the HW caps to see if we have uc support -*/ BUILD_BUG_ON (XE_UC_FIRMWARE_UNINITIALIZED ); -xe_gt_assert (gt, !uc_fw->status); -xe_gt_assert (gt, !uc_fw->path);uc_fw_auto_select (xe, uc_fw);
@@ -823,6 +817,14 @@ static int uc_fw_copy(struct xe_uc_fw *uc_fw, const void *data, size_t size, u32return err; } +static void uc_fw_reinit (struct xe_uc_fw *uc_fw,const void *data) +{ +struct xe_device *xe =uc_fw_to_xe (uc_fw); + +xe_map_memcpy_to (xe, &uc_fw->bo->vmap,0 , data, uc_fw->size); +xe_uc_fw_change_status (uc_fw,XE_UC_FIRMWARE_LOADABLE ); +} +int xe_uc_fw_init (struct xe_uc_fw *uc_fw) {const struct firmware *fw =NULL ;
@@ -846,6 +848,31 @@ int xe_uc_fw_init(struct xe_uc_fw *uc_fw) }ALLOW_ERROR_INJECTION (xe_uc_fw_init,ERRNO );/* See xe_pci_probe() */ +/** +* xe_uc_fw_reinit() - Re-initialize uC firmware into its bo +* @uc_fw: uC firmware +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_uc_fw_reinit (struct xe_uc_fw *uc_fw) +{ +const struct firmware *fw =NULL ; +int err; + + err =uc_fw_request (uc_fw, &fw); +if (err) +return err; + +/* no error and no firmware means nothing to copy */ +if (!fw) +return 0 ; + +uc_fw_reinit (uc_fw, fw->data); +uc_fw_release (fw); + +return err; +} +static u32 uc_fw_ggtt_offset (struct xe_uc_fw *uc_fw) {return xe_bo_ggtt_addr (uc_fw->bo);
drivers/gpu/drm/xe/xe_uc_fw.h +1 -0
@@ -15,6 +15,7 @@struct drm_printer ;int xe_uc_fw_init (struct xe_uc_fw *uc_fw); +int xe_uc_fw_reinit (struct xe_uc_fw *uc_fw);size_t xe_uc_fw_copy_rsa (struct xe_uc_fw *uc_fw,void *dst,u32 max_len);int xe_uc_fw_upload (struct xe_uc_fw *uc_fw,u32 offset,u32 dma_flags);int xe_uc_fw_check_version_requirements (struct xe_uc_fw *uc_fw);
Signature · 1 line
2.43.0
PATCH v2 2/9 drm/xe/gt: Introduce FLR helpers
In reply to Raag JadavIn preparation of usecases which require preparing/re-initializing GT before/after PCIe FLR, introduce flr_prepare/done() helpers. All GT uCs are already brought up as part of xe_gt_resume() sequence, but before that happens we still need to re-initialize their firmwares so that they can be loaded as part of xe_gt_resume() sequence.
---
v2: Add kernel doc (Matthew Brost)
--- drivers/gpu/drm/xe/xe_gsc.c | 14 ++++++++++++++ drivers/gpu/drm/xe/xe_gsc.h | 1 + drivers/gpu/drm/xe/xe_gt.c | 22 ++++++++++++++++++++++ drivers/gpu/drm/xe/xe_gt.h | 2 ++ drivers/gpu/drm/xe/xe_guc.c | 14 ++++++++++++++ drivers/gpu/drm/xe/xe_guc.h | 1 + drivers/gpu/drm/xe/xe_huc.c | 14 ++++++++++++++ drivers/gpu/drm/xe/xe_huc.h | 1 + drivers/gpu/drm/xe/xe_uc.c | 36 ++++++++++++++++++++++++++++++++++++ drivers/gpu/drm/xe/xe_uc.h | 2 ++ 10 files changed, 107 insertions(+)
drivers/gpu/drm/xe/xe_gsc.c +14 -0
@@ -552,6 +552,20 @@ void xe_gsc_wait_for_worker_completion(struct xe_gsc *gsc)flush_work (&gsc->work); } +/** +* xe_gsc_flr_done() - Re-initialize GSC firmware after FLR +* @gsc: The GSC object +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_gsc_flr_done (struct xe_gsc *gsc) +{ +if (!xe_uc_fw_is_loadable (&gsc->fw)) +return 0 ; + +return xe_uc_fw_reinit (&gsc->fw); +} +void xe_gsc_stop_prepare (struct xe_gsc *gsc) {struct xe_gt *gt =gsc_to_gt (gsc);
drivers/gpu/drm/xe/xe_gsc.h +1 -0
@@ -13,6 +13,7 @@ struct xe_gsc;struct xe_gt ;struct xe_hw_engine ; +int xe_gsc_flr_done (struct xe_gsc *gsc);int xe_gsc_init (struct xe_gsc *gsc);int xe_gsc_init_post_hwconfig (struct xe_gsc *gsc);void xe_gsc_wait_for_worker_completion (struct xe_gsc *gsc);
drivers/gpu/drm/xe/xe_gt.c +22 -0
@@ -947,6 +947,28 @@ void xe_gt_reset_async(struct xe_gt *gt)xe_pm_runtime_put (gt_to_xe (gt)); } +/** +* xe_gt_flr_prepare() - Prepare GT for FLR +* @gt: the GT object +* +* Prepare all GT uCs for FLR. +*/ +void xe_gt_flr_prepare (struct xe_gt *gt) +{ +xe_uc_flr_prepare (>->uc); +} + +/** +* xe_gt_flr_done() - Re-initialize GT after FLR +* @gt: the GT object +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_gt_flr_done (struct xe_gt *gt) +{ +return xe_uc_flr_done (>->uc); +} +void xe_gt_suspend_prepare (struct xe_gt *gt) {CLASS (xe_force_wake, fw_ref)(gt_to_fw (gt),XE_FORCEWAKE_ALL );
drivers/gpu/drm/xe/xe_gt.h +2 -0
@@ -63,6 +63,8 @@ int xe_gt_record_default_lrcs(struct xe_gt *gt);*/ void xe_gt_record_user_engines (struct xe_gt *gt); +int xe_gt_flr_done (struct xe_gt *gt); +void xe_gt_flr_prepare (struct xe_gt *gt);void xe_gt_suspend_prepare (struct xe_gt *gt);int xe_gt_suspend (struct xe_gt *gt);void xe_gt_shutdown (struct xe_gt *gt);
drivers/gpu/drm/xe/xe_guc.c +14 -0
@@ -1656,6 +1656,20 @@ void xe_guc_sanitize(struct xe_guc *guc)xe_guc_submit_disable (guc); } +/** +* xe_guc_flr_done() - Re-initialize GuC firmware after FLR +* @guc: The GuC object +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_guc_flr_done (struct xe_guc *guc) +{ +if (!xe_uc_fw_is_loadable (&guc->fw)) +return 0 ; + +return xe_uc_fw_reinit (&guc->fw); +} +int xe_guc_reset_prepare (struct xe_guc *guc) {return xe_guc_submit_reset_prepare (guc);
drivers/gpu/drm/xe/xe_guc.h +1 -0
@@ -32,6 +32,7 @@struct drm_printer ;void xe_guc_comm_init_early (struct xe_guc *guc); +int xe_guc_flr_done (struct xe_guc *guc);int xe_guc_init_noalloc (struct xe_guc *guc);int xe_guc_init (struct xe_guc *guc);int xe_guc_init_post_hwconfig (struct xe_guc *guc);
drivers/gpu/drm/xe/xe_huc.c +14 -0
@@ -296,6 +296,20 @@ void xe_huc_sanitize(struct xe_huc *huc)xe_uc_fw_sanitize (&huc->fw); } +/** +* xe_huc_flr_done() - Re-initialize HuC firmware after FLR +* @huc: The HuC object +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_huc_flr_done (struct xe_huc *huc) +{ +if (!xe_uc_fw_is_loadable (&huc->fw)) +return 0 ; + +return xe_uc_fw_reinit (&huc->fw); +} +void xe_huc_print_info (struct xe_huc *huc,struct drm_printer *p) {struct xe_gt *gt =huc_to_gt (huc);
drivers/gpu/drm/xe/xe_huc.h +1 -0
@@ -17,6 +17,7 @@ enum xe_huc_auth_types { XE_HUC_AUTH_TYPES_COUNT }; +int xe_huc_flr_done (struct xe_huc *huc);int xe_huc_init (struct xe_huc *huc);int xe_huc_init_post_hwconfig (struct xe_huc *huc);int xe_huc_upload (struct xe_huc *huc);
drivers/gpu/drm/xe/xe_uc.c +36 -0
@@ -17,6 +17,7 @@#include "xe_guc_engine_activity.h" #include "xe_huc.h" #include "xe_sriov.h" +#include "xe_uc_fw.h" #include "xe_wopcm.h" static struct xe_gt *
@@ -283,6 +284,41 @@ static void uc_reset_wait(struct xe_uc *uc) goto again; } +/** +* xe_uc_flr_prepare() - Prepare uCs for FLR +* @uc: The uC object +* +* Flush pending work and stop all uCs. +*/ +void xe_uc_flr_prepare (struct xe_uc *uc) +{ +xe_gsc_wait_for_worker_completion (&uc->gsc); +xe_uc_reset_prepare (uc); +xe_uc_stop (uc); +xe_uc_sanitize (uc); +} + +/** +* xe_uc_flr_done() - Re-initialize uCs after FLR +* @uc: The uC object +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_uc_flr_done (struct xe_uc *uc) +{ +int ret; + + ret =xe_guc_flr_done (&uc->guc); +if (ret) +return ret; + + ret =xe_huc_flr_done (&uc->huc); +if (ret) +return ret; + +return xe_gsc_flr_done (&uc->gsc); +} +void xe_uc_suspend_prepare (struct xe_uc *uc) {xe_gsc_wait_for_worker_completion (&uc->gsc);
drivers/gpu/drm/xe/xe_uc.h +2 -0
@@ -8,6 +8,8 @@struct xe_uc ; +int xe_uc_flr_done (struct xe_uc *uc); +void xe_uc_flr_prepare (struct xe_uc *uc);int xe_uc_init_noalloc (struct xe_uc *uc);int xe_uc_init (struct xe_uc *uc);int xe_uc_init_post_hwconfig (struct xe_uc *uc);
Signature · 1 line
2.43.0
PATCH v2 3/9 drm/xe/irq: Introduce xe_irq_disable()
In reply to Raag JadavIn preparation of usecases which require disabling interrupts before PCIe FLR, introduce xe_irq_disable() helper. With this we avoid any side-effects of unwarranted irqs.
---
v2: Add kernel doc (Matthew Brost)
--- drivers/gpu/drm/xe/xe_irq.c | 13 ++++++++++++- drivers/gpu/drm/xe/xe_irq.h | 1 + 2 files changed, 13 insertions(+), 1 deletion(-)
drivers/gpu/drm/xe/xe_irq.c +12 -1
@@ -843,7 +843,13 @@ static void xe_irq_msi_synchronize_irq(struct xe_device *xe)synchronize_irq (to_pci_dev (xe->drm.dev)->irq); } -void xe_irq_suspend (struct xe_device *xe) +/** +* xe_irq_disable() - Disable irqs +* @xe: xe device instance +* +* Synchronize pending irqs and disable them. +*/ +void xe_irq_disable (struct xe_device *xe) {atomic_set (&xe->irq.enabled,0 );/* no new irqs */
@@ -852,6 +858,11 @@ void xe_irq_suspend(struct xe_device *xe)xe_irq_msix_synchronize_irq (xe); elsexe_irq_msi_synchronize_irq (xe ); +} + +void xe_irq_suspend (struct xe_device *xe) +{ +xe_irq_disable (xe);xe_irq_reset (xe);/* turn irqs off */ }
drivers/gpu/drm/xe/xe_irq.h +1 -0
@@ -14,6 +14,7 @@ struct xe_device;struct xe_tile ;struct xe_gt ; +void xe_irq_disable (struct xe_device *xe);int xe_irq_init (struct xe_device *xe);int xe_irq_install (struct xe_device *xe);void xe_irq_suspend (struct xe_device *xe);
Signature · 1 line
2.43.0
PATCH v2 4/9 drm/xe: Introduce xe_device_assert_lmem_ready()
In reply to Raag JadavIn preparation of usecases which require verifying LMEM readiness after PCIe FLR, introduce xe_device_assert_lmem_ready() helper.
---
v2: Add kernel doc (Matthew Brost)
--- drivers/gpu/drm/xe/xe_device.c | 10 ++++++++-- drivers/gpu/drm/xe/xe_device.h | 1 + 2 files changed, 9 insertions(+), 2 deletions(-)
drivers/gpu/drm/xe/xe_device.c +8 -2
@@ -650,7 +650,13 @@ static int xe_set_dma_info(struct xe_device *xe)return err; } -static void assert_lmem_ready (struct xe_device *xe) +/** +* xe_device_assert_lmem_ready() - Verify LMEM readiness +* @xe: xe device instance +* +* Verify LMEM readiness and assert on failure. +*/ +void xe_device_assert_lmem_ready (struct xe_device *xe) {if (!IS_DGFX (xe) ||IS_SRIOV_VF (xe))return ;
@@ -738,7 +744,7 @@ int xe_device_probe_early(struct xe_device *xe)* is flagged after full initialization is complete. Assert if lmem is * not initialized. */ -assert_lmem_ready (xe); +xe_device_assert_lmem_ready (xe); xe->wedged.mode =xe_device_validate_wedged_mode (xe, xe_modparam.wedged_mode) ?XE_DEFAULT_WEDGED_MODE : xe_modparam.wedged_mode;
drivers/gpu/drm/xe/xe_device.h +1 -0
@@ -140,6 +140,7 @@ static inline struct xe_force_wake *gt_to_fw(struct xe_gt *gt)return >->pm.fw; } +void xe_device_assert_lmem_ready (struct xe_device *xe);void xe_device_assert_mem_access (struct xe_device *xe);static inline bool xe_device_has_flat_ccs (struct xe_device *xe)
Signature · 1 line
2.43.0
PATCH v2 5/9 drm/xe/bo_evict: Introduce xe_bo_restore_map()
In reply to Raag JadavPCIe FLR clears all hardware state along with GGTT mappings of all existing bos. Iterate over early and late kernel bo list and restore their GGTT mappings in hardware, so that kernel can make use of them during re-initialization after PCIe FLR.
---
v2: Add kernel doc (Matthew Brost)
--- drivers/gpu/drm/xe/xe_bo_evict.c | 51 +++++++++++++++++++++++++++----- drivers/gpu/drm/xe/xe_bo_evict.h | 2 ++ 2 files changed, 45 insertions(+), 8 deletions(-)
drivers/gpu/drm/xe/xe_bo_evict.c +43 -8
@@ -189,14 +189,8 @@ int xe_bo_evict_all(struct xe_device *xe) xe_bo_evict_pinned); } -static int xe_bo_restore_and_map_ggtt (struct xe_bo *bo) +static int xe_bo_map_ggtt (struct xe_bo *bo) { -int ret; - - ret =xe_bo_restore_pinned (bo); -if (ret) -return ret; -if (bo->flags &XE_BO_FLAG_GGTT ) {struct xe_tile *tile;u8 id;
@@ -212,6 +206,41 @@ static int xe_bo_restore_and_map_ggtt(struct xe_bo *bo)return 0 ; } +/** +* xe_bo_restore_map() - Restore GGTT mappings of kernel bos +* @xe: xe device +* +* PCIe FLR clears all hardware state along with GGTT mappings of all +* existing bos. Iterate over early and late kernel bo list and restore +* their GGTT mappings in hardware, so that kernel can make use of them +* during re-initialization after PCIe FLR. +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_bo_restore_map (struct xe_device *xe) +{ +int ret; + + ret =xe_bo_apply_to_pinned (xe, &xe->pinned.early.kernel_bo_present, + &xe->pinned.early.kernel_bo_present, xe_bo_map_ggtt); +if (!ret) + ret =xe_bo_apply_to_pinned (xe, &xe->pinned.late.kernel_bo_present, + &xe->pinned.late.kernel_bo_present, xe_bo_map_ggtt); + +return ret; +} + +static int xe_bo_restore_and_map_ggtt (struct xe_bo *bo) +{ +int ret; + + ret =xe_bo_restore_pinned (bo); +if (ret) +return ret; + +return xe_bo_map_ggtt (bo); +} +/** * xe_bo_restore_early - restore early phase kernel BOs to VRAM *
@@ -270,7 +299,13 @@ int xe_bo_restore_late(struct xe_device *xe)return ret; } -static void xe_bo_pci_dev_remove_pinned (struct xe_device *xe) +/** +* xe_bo_pci_dev_remove_pinned() - Unmap external bos +* @xe: xe device +* +* Drop dma mappings of all external pinned bos. +*/ +void xe_bo_pci_dev_remove_pinned (struct xe_device *xe) {struct xe_tile *tile;unsigned int id;
drivers/gpu/drm/xe/xe_bo_evict.h +2 -0
@@ -14,7 +14,9 @@ int xe_bo_notifier_prepare_all_pinned(struct xe_device *xe);void xe_bo_notifier_unprepare_all_pinned (struct xe_device *xe);int xe_bo_restore_early (struct xe_device *xe);int xe_bo_restore_late (struct xe_device *xe); +int xe_bo_restore_map (struct xe_device *xe); +void xe_bo_pci_dev_remove_pinned (struct xe_device *xe);void xe_bo_pci_dev_remove_all (struct xe_device *xe);int xe_bo_pinned_init (struct xe_device *xe);
Signature · 1 line
2.43.0
PATCH v2 6/9 drm/xe/lrc: Introduce xe_lrc_reinit()
In reply to Raag JadavIn preparation of usecases which require re-initializing LRC after PCIe FLR, introduce xe_lrc_reinit() helper. The LRC bo already exists but since it's contents are on VRAM, they are lost on PCIe FLR. Recreate ring context as part of re-initialization.
---
v2: Re-initialize migrate context (Matthew Brost)
--- drivers/gpu/drm/xe/xe_lrc.c | 149 +++++++++++++++++++++--------------- drivers/gpu/drm/xe/xe_lrc.h | 2 + 2 files changed, 90 insertions(+), 61 deletions(-)
drivers/gpu/drm/xe/xe_lrc.c +88 -61
@@ -1438,65 +1438,16 @@ void xe_lrc_set_multi_queue_priority(struct xe_lrc *lrc, enum xe_multi_queue_pri lrc->desc |=FIELD_PREP (LRC_PRIORITY ,xe_multi_queue_prio_to_lrc (lrc, priority)); } -static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe, -struct xe_vm *vm,void *replay_state,u32 ring_size, -u16 msix_vec, -u32 init_flags) +static int xe_lrc_init_ctx (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) {struct xe_gt *gt = hwe->gt; -const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); -u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ;struct xe_tile *tile =gt_to_tile (gt);struct xe_device *xe =gt_to_xe (gt); -struct xe_bo *seqno_bo;struct iosys_map map;u32 arb_enable; -u32 bo_flags;int err; -kref_init (&lrc->refcount); - lrc->gt = gt; - lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); - lrc->size = lrc_size; - lrc->flags =0 ; - lrc->ring.size = ring_size; - lrc->ring.tail =0 ; - -if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { - lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; - bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; - } - -if (xe_gt_has_indirect_ring_state (gt)) - lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; - - bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE ; - -if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; - - lrc->bo =xe_bo_create_pin_map_novm (xe, tile, - bo_size, - ttm_bo_type_kernel, - bo_flags, false); -if (IS_ERR (lrc->bo)) -return PTR_ERR (lrc->bo); - - seqno_bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , - ttm_bo_type_kernel, -XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE | -XE_BO_FLAG_SYSTEM , false); -if (IS_ERR (seqno_bo)) { - err =PTR_ERR (seqno_bo); - goto err_lrc_finish; - } - lrc->seqno_bo = seqno_bo; - -xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, - hwe->fence_irq, hwe->name); -/* * Init Per-Process of HW status Page, LRC / context state to known * values. If there's already a primed default_lrc, just copy it, otherwise
@@ -1508,7 +1459,7 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,xe_map_memset (xe, &map,0 ,0 ,LRC_PPHWSP_SIZE );/* PPHWSP */ xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , gt->default_lrc[hwe->class] +LRC_PPHWSP_SIZE , - lrc_size -LRC_PPHWSP_SIZE ); + lrc->size -LRC_PPHWSP_SIZE );if (replay_state)xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , replay_state, lrc->replay_size);
@@ -1516,21 +1467,16 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,void *init_data =empty_lrc_data (hwe);if (!init_data) { - err = -ENOMEM ; - goto err_lrc_finish; +return -ENOMEM ; } -xe_map_memcpy_to (xe, &map,0 , init_data, lrc_size); +xe_map_memcpy_to (xe, &map,0 , init_data, lrc->size);kfree (init_data); } -if (vm) { +if (vm)xe_lrc_set_ppgtt (lrc, vm); -if (vm->xef) -xe_drm_client_add_bo (vm->xef->client, lrc->bo); - } -if (xe_device_has_msix (xe)) {xe_lrc_write_ctx_reg (lrc,CTX_INT_STATUS_REPORT_PTR ,xe_memirq_status_ptr (&tile->memirq, hwe));
@@ -1602,12 +1548,93 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, err =setup_wa_bb (lrc, hwe);if (err) - goto err_lrc_finish; +return err; err =setup_indirect_ctx (lrc, hwe); + +return err; +} + +/** +* xe_lrc_reinit() - Re-initialize LRC +* @lrc: Pointer to the LRC +* @hwe: Hardware Engine +* @vm: The VM (address space) +* @replay_state: GPU hang replay state +* @msix_vec: MSI-X interrupt vector (for platforms that support it) +* @init_flags: LRC initialization flags +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) +{ +return xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags); +} + +static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u32 ring_size,u16 msix_vec,u32 init_flags) +{ +struct xe_gt *gt = hwe->gt; +const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); +u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ; +struct xe_tile *tile =gt_to_tile (gt); +struct xe_device *xe =gt_to_xe (gt); +struct xe_bo *bo; +u32 bo_flags; +int err; + +kref_init (&lrc->refcount); + lrc->gt = gt; + lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); + lrc->size = lrc_size; + lrc->flags =0 ; + lrc->ring.size = ring_size; + lrc->ring.tail =0 ; + +if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { + lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; + bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; + } + +if (xe_gt_has_indirect_ring_state (gt)) + lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; + + bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE ; + +if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + + bo =xe_bo_create_pin_map_novm (xe, tile, bo_size, + ttm_bo_type_kernel, + bo_flags, false); +if (IS_ERR (lrc->bo)) +return PTR_ERR (lrc->bo); + + lrc->bo = bo; + + bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , + ttm_bo_type_kernel, +XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE | +XE_BO_FLAG_SYSTEM , false); +if (IS_ERR (bo)) { + err =PTR_ERR (bo); + goto err_lrc_finish; + } + lrc->seqno_bo = bo; + +xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, + hwe->fence_irq, hwe->name); + + err =xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags);if (err) goto err_lrc_finish; +if (vm && vm->xef) +xe_drm_client_add_bo (vm->xef->client, lrc->bo); +return 0 ; err_lrc_finish:
drivers/gpu/drm/xe/xe_lrc.h +2 -0
@@ -52,6 +52,8 @@ struct xe_lrc_snapshot {struct xe_lrc *xe_lrc_create (struct xe_hw_engine *hwe,struct xe_vm *vm,void *replay_state,u32 ring_size,u16 msix_vec,u32 flags); +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags);void xe_lrc_destroy (struct kref *ref);/**
Signature · 1 line
2.43.0
Matthew Brost
maintainerRe: [PATCH v2 6/9] drm/xe/lrc: Introduce xe_lrc_reinit()
Quoted 135 lines · Raag Jadav
In preparation of usecases which require re-initializing LRC after PCIe FLR, introduce
xe_lrc_reinit()helper. The LRC bo already exists but since it's contents are on VRAM, they are lost on PCIe FLR. Recreate ring context as part of re-initialization.Signed-off-by Raag Jadav <[email protected]>---
v2: Re-initialize migrate context (Matthew Brost)--- drivers/gpu/drm/xe/xe_lrc.c | 149 +++++++++++++++++++++--------------- drivers/gpu/drm/xe/xe_lrc.h | 2 + 2 files changed, 90 insertions(+), 61 deletions(-)drivers/gpu/drm/xe/xe_lrc.c +15
-61@@ -1438,65 +1438,16 @@ void xe_lrc_set_multi_queue_priority(struct xe_lrc *lrc, enum xe_multi_queue_pri lrc->desc |=FIELD_PREP (LRC_PRIORITY ,xe_multi_queue_prio_to_lrc (lrc, priority)); } -static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe, -struct xe_vm *vm,void *replay_state,u32 ring_size, -u16 msix_vec, -u32 init_flags) +static int xe_lrc_init_ctx (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) {struct xe_gt *gt = hwe->gt; -const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); -u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ;struct xe_tile *tile =gt_to_tile (gt);struct xe_device *xe =gt_to_xe (gt); -struct xe_bo *seqno_bo;struct iosys_map map;u32 arb_enable; -u32 bo_flags;int err; -kref_init (&lrc->refcount); - lrc->gt = gt; - lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); - lrc->size = lrc_size; - lrc->flags =0 ; - lrc->ring.size = ring_size; - lrc->ring.tail =0 ; - -if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { - lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; - bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; - } - -if (xe_gt_has_indirect_ring_state (gt)) - lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; - - bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE ; - -if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; - - lrc->bo =xe_bo_create_pin_map_novm (xe, tile, - bo_size, - ttm_bo_type_kernel, - bo_flags, false); -if (IS_ERR (lrc->bo)) -return PTR_ERR (lrc->bo); - - seqno_bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , - ttm_bo_type_kernel, -XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE | -XE_BO_FLAG_SYSTEM , false); -if (IS_ERR (seqno_bo)) { - err =PTR_ERR (seqno_bo); - goto err_lrc_finish; - } - lrc->seqno_bo = seqno_bo; - -xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, - hwe->fence_irq, hwe->name); -/* * Init Per-Process of HW status Page, LRC / context state to known * values. If there's already a primed default_lrc, just copy it, otherwise @@ -1508,7 +1459,7 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,xe_map_memset (xe, &map,0 ,0 ,LRC_PPHWSP_SIZE );/* PPHWSP */ xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , gt->default_lrc[hwe->class] +LRC_PPHWSP_SIZE , - lrc_size -LRC_PPHWSP_SIZE ); + lrc->size -LRC_PPHWSP_SIZE );if (replay_state)xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , replay_state, lrc->replay_size);@@ -1516,21 +1467,16 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,void *init_data =empty_lrc_data (hwe);if (!init_data) { - err = -ENOMEM ; - goto err_lrc_finish; +return -ENOMEM ; } -xe_map_memcpy_to (xe, &map,0 , init_data, lrc_size); +xe_map_memcpy_to (xe, &map,0 , init_data, lrc->size);kfree (init_data); } -if (vm) { +if (vm)xe_lrc_set_ppgtt (lrc, vm); -if (vm->xef) -xe_drm_client_add_bo (vm->xef->client, lrc->bo); - } -if (xe_device_has_msix (xe)) {xe_lrc_write_ctx_reg (lrc,CTX_INT_STATUS_REPORT_PTR ,xe_memirq_status_ptr (&tile->memirq, hwe));@@ -1602,12 +1548,93 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, err =setup_wa_bb (lrc, hwe);if (err) - goto err_lrc_finish; +return err; err =setup_indirect_ctx (lrc, hwe); + +return err; +} + +/** +* xe_lrc_reinit() - Re-initialize LRC +* @lrc: Pointer to the LRC +* @hwe: Hardware Engine
drivers/gpu/drm/xe/xe_lrc.c static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, · line 1571
Quoted 10 lines
drivers/gpu/drm/xe/xe_lrc.c +10
-0+* @vm: The VM (address space) +* @replay_state: GPU hang replay state +* @msix_vec: MSI-X interrupt vector (for platforms that support it) +* @init_flags: LRC initialization flags +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) +{
I think you likely want to set lrc->ring.tail = 0 here (or in xe_lrc_init_ctx), right? Alternatively, you could set both INDIRECT_CTX_RING_HEAD and INDIRECT_CTX_RING_TAIL to lrc->ring.tail in xe_lrc_init_ctx.
Consider the case where a bunch of work has run on the migration queue and lrc->ring.tail ends up in the middle of the ring, then xe_lrc_reinit is called. The next submission on the LRC will execute the instructions between 0 (INDIRECT_CTX_RING_HEAD is set zero in xe_lrc_init_ctx) and the lrc->ring.tail value at the time xe_lrc_reinit was invoked, which will be stale or invalid if VRAM was clobbered.
I would have expected this to show up in testing if you ran something like:
xe_exec_basic;
echo 1 > /sys/bus/pci/devices/<BDF>/reset;xe_exec_basic;
Otherwise this is good cleanup adding xe_lrc_init_ctx regardless of Xe PCIe FLR so if we work out above, feel free to post this an independent which we can merge ahead of Xe PCIe FLR.
Matt
Quoted 85 lines
drivers/gpu/drm/xe/xe_lrc.c +63
-0+return xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags); +} + +static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u32 ring_size,u16 msix_vec,u32 init_flags) +{ +struct xe_gt *gt = hwe->gt; +const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); +u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ; +struct xe_tile *tile =gt_to_tile (gt); +struct xe_device *xe =gt_to_xe (gt); +struct xe_bo *bo; +u32 bo_flags; +int err; + +kref_init (&lrc->refcount); + lrc->gt = gt; + lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); + lrc->size = lrc_size; + lrc->flags =0 ; + lrc->ring.size = ring_size; + lrc->ring.tail =0 ; + +if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { + lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; + bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; + } + +if (xe_gt_has_indirect_ring_state (gt)) + lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; + + bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE ; + +if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + + bo =xe_bo_create_pin_map_novm (xe, tile, bo_size, + ttm_bo_type_kernel, + bo_flags, false); +if (IS_ERR (lrc->bo)) +return PTR_ERR (lrc->bo); + + lrc->bo = bo; + + bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , + ttm_bo_type_kernel, +XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE | +XE_BO_FLAG_SYSTEM , false); +if (IS_ERR (bo)) { + err =PTR_ERR (bo); + goto err_lrc_finish; + } + lrc->seqno_bo = bo; + +xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, + hwe->fence_irq, hwe->name); + + err =xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags);if (err) goto err_lrc_finish; +if (vm && vm->xef) +xe_drm_client_add_bo (vm->xef->client, lrc->bo); +return 0 ; err_lrc_finish:drivers/gpu/drm/xe/xe_lrc.h +2
-0@@ -52,6 +52,8 @@ struct xe_lrc_snapshot {struct xe_lrc *xe_lrc_create (struct xe_hw_engine *hwe,struct xe_vm *vm,void *replay_state,u32 ring_size,u16 msix_vec,u32 flags); +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags);void xe_lrc_destroy (struct kref *ref);/** Signature · 1 line
2.43.0
Re: [PATCH v2 6/9] drm/xe/lrc: Introduce xe_lrc_reinit()
Quoted 162 lines · Matthew Brost
Quoted 135 lines · Raag Jadav
In preparation of usecases which require re-initializing LRC after PCIe FLR, introduce
xe_lrc_reinit()helper. The LRC bo already exists but since it's contents are on VRAM, they are lost on PCIe FLR. Recreate ring context as part of re-initialization.Signed-off-by Raag Jadav <[email protected]>---
v2: Re-initialize migrate context (Matthew Brost)--- drivers/gpu/drm/xe/xe_lrc.c | 149 +++++++++++++++++++++--------------- drivers/gpu/drm/xe/xe_lrc.h | 2 + 2 files changed, 90 insertions(+), 61 deletions(-)drivers/gpu/drm/xe/xe_lrc.c +15
-61@@ -1438,65 +1438,16 @@ void xe_lrc_set_multi_queue_priority(struct xe_lrc *lrc, enum xe_multi_queue_pri lrc->desc |=FIELD_PREP (LRC_PRIORITY ,xe_multi_queue_prio_to_lrc (lrc, priority)); } -static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe, -struct xe_vm *vm,void *replay_state,u32 ring_size, -u16 msix_vec, -u32 init_flags) +static int xe_lrc_init_ctx (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) {struct xe_gt *gt = hwe->gt; -const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); -u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ;struct xe_tile *tile =gt_to_tile (gt);struct xe_device *xe =gt_to_xe (gt); -struct xe_bo *seqno_bo;struct iosys_map map;u32 arb_enable; -u32 bo_flags;int err; -kref_init (&lrc->refcount); - lrc->gt = gt; - lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); - lrc->size = lrc_size; - lrc->flags =0 ; - lrc->ring.size = ring_size; - lrc->ring.tail =0 ; - -if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { - lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; - bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; - } - -if (xe_gt_has_indirect_ring_state (gt)) - lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; - - bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE ; - -if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ - bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; - - lrc->bo =xe_bo_create_pin_map_novm (xe, tile, - bo_size, - ttm_bo_type_kernel, - bo_flags, false); -if (IS_ERR (lrc->bo)) -return PTR_ERR (lrc->bo); - - seqno_bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , - ttm_bo_type_kernel, -XE_BO_FLAG_GGTT | -XE_BO_FLAG_GGTT_INVALIDATE | -XE_BO_FLAG_SYSTEM , false); -if (IS_ERR (seqno_bo)) { - err =PTR_ERR (seqno_bo); - goto err_lrc_finish; - } - lrc->seqno_bo = seqno_bo; - -xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, - hwe->fence_irq, hwe->name); -/* * Init Per-Process of HW status Page, LRC / context state to known * values. If there's already a primed default_lrc, just copy it, otherwise @@ -1508,7 +1459,7 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,xe_map_memset (xe, &map,0 ,0 ,LRC_PPHWSP_SIZE );/* PPHWSP */ xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , gt->default_lrc[hwe->class] +LRC_PPHWSP_SIZE , - lrc_size -LRC_PPHWSP_SIZE ); + lrc->size -LRC_PPHWSP_SIZE );if (replay_state)xe_map_memcpy_to (xe, &map,LRC_PPHWSP_SIZE , replay_state, lrc->replay_size);@@ -1516,21 +1467,16 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe,void *init_data =empty_lrc_data (hwe);if (!init_data) { - err = -ENOMEM ; - goto err_lrc_finish; +return -ENOMEM ; } -xe_map_memcpy_to (xe, &map,0 , init_data, lrc_size); +xe_map_memcpy_to (xe, &map,0 , init_data, lrc->size);kfree (init_data); } -if (vm) { +if (vm)xe_lrc_set_ppgtt (lrc, vm); -if (vm->xef) -xe_drm_client_add_bo (vm->xef->client, lrc->bo); - } -if (xe_device_has_msix (xe)) {xe_lrc_write_ctx_reg (lrc,CTX_INT_STATUS_REPORT_PTR ,xe_memirq_status_ptr (&tile->memirq, hwe));@@ -1602,12 +1548,93 @@ static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, err =setup_wa_bb (lrc, hwe);if (err) - goto err_lrc_finish; +return err; err =setup_indirect_ctx (lrc, hwe); + +return err; +} + +/** +* xe_lrc_reinit() - Re-initialize LRC +* @lrc: Pointer to the LRC +* @hwe: Hardware Engine drivers/gpu/drm/xe/xe_lrc.c static int xe_lrc_init(struct xe_lrc *lrc, struct xe_hw_engine *hwe, · line 1571
Quoted 10 lines
drivers/gpu/drm/xe/xe_lrc.c +10
-0+* @vm: The VM (address space) +* @replay_state: GPU hang replay state +* @msix_vec: MSI-X interrupt vector (for platforms that support it) +* @init_flags: LRC initialization flags +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags) +{I think you likely want to set
lrc->ring.tail= 0 here (or inxe_lrc_init_ctx), right? Alternatively, you could set bothINDIRECT_CTX_RING_HEADandINDIRECT_CTX_RING_TAILtolrc->ring.tailinxe_lrc_init_ctx.Consider the case where a bunch of work has run on the migration queue and
lrc->ring.tailends up in the middle of the ring, thenxe_lrc_reinitis called. The next submission on the LRC will execute the instructions between 0 (INDIRECT_CTX_RING_HEADis set zero inxe_lrc_init_ctx) and thelrc->ring.tailvalue at the timexe_lrc_reinitwas invoked, which will be stale or invalid if VRAM was clobbered.I would have expected this to show up in testing if you ran something like:
Quoted 7 lines
xe_exec_basic;
echo 1 > /sys/bus/pci/devices/<BDF>/reset;xe_exec_basic;Otherwise this is good cleanup adding
xe_lrc_init_ctxregardless of Xe PCIe FLR so if we work out above, feel free to post this an independent which we can merge ahead of Xe PCIe FLR.
Makes sense.
Raag
Quoted 85 lines
Quoted 85 lines
drivers/gpu/drm/xe/xe_lrc.c +63
-0+return xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags); +} + +static int xe_lrc_init (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u32 ring_size,u16 msix_vec,u32 init_flags) +{ +struct xe_gt *gt = hwe->gt; +const u32 lrc_size =xe_gt_lrc_size (gt, hwe->class); +u32 bo_size = ring_size + lrc_size +LRC_WA_BB_SIZE ; +struct xe_tile *tile =gt_to_tile (gt); +struct xe_device *xe =gt_to_xe (gt); +struct xe_bo *bo; +u32 bo_flags; +int err; + +kref_init (&lrc->refcount); + lrc->gt = gt; + lrc->replay_size =xe_gt_lrc_hang_replay_size (gt, hwe->class); + lrc->size = lrc_size; + lrc->flags =0 ; + lrc->ring.size = ring_size; + lrc->ring.tail =0 ; + +if (gt_engine_needs_indirect_ctx (gt, hwe->class)) { + lrc->flags |=XE_LRC_FLAG_INDIRECT_CTX ; + bo_size +=LRC_INDIRECT_CTX_BO_SIZE ; + } + +if (xe_gt_has_indirect_ring_state (gt)) + lrc->flags |=XE_LRC_FLAG_INDIRECT_RING_STATE ; + + bo_flags =XE_BO_FLAG_VRAM_IF_DGFX (tile) |XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE ; + +if ((vm && vm->xef) || init_flags &XE_LRC_CREATE_USER_CTX )/* userspace */ + bo_flags |=XE_BO_FLAG_PINNED_LATE_RESTORE |XE_BO_FLAG_FORCE_USER_VRAM ; + + bo =xe_bo_create_pin_map_novm (xe, tile, bo_size, + ttm_bo_type_kernel, + bo_flags, false); +if (IS_ERR (lrc->bo)) +return PTR_ERR (lrc->bo); + + lrc->bo = bo; + + bo =xe_bo_create_pin_map_novm (xe, tile,PAGE_SIZE , + ttm_bo_type_kernel, +XE_BO_FLAG_GGTT | +XE_BO_FLAG_GGTT_INVALIDATE | +XE_BO_FLAG_SYSTEM , false); +if (IS_ERR (bo)) { + err =PTR_ERR (bo); + goto err_lrc_finish; + } + lrc->seqno_bo = bo; + +xe_hw_fence_ctx_init (&lrc->fence_ctx, hwe->gt, + hwe->fence_irq, hwe->name); + + err =xe_lrc_init_ctx (lrc, hwe, vm, replay_state, msix_vec, init_flags);if (err) goto err_lrc_finish; +if (vm && vm->xef) +xe_drm_client_add_bo (vm->xef->client, lrc->bo); +return 0 ; err_lrc_finish:drivers/gpu/drm/xe/xe_lrc.h +2
-0@@ -52,6 +52,8 @@ struct xe_lrc_snapshot {struct xe_lrc *xe_lrc_create (struct xe_hw_engine *hwe,struct xe_vm *vm,void *replay_state,u32 ring_size,u16 msix_vec,u32 flags); +int xe_lrc_reinit (struct xe_lrc *lrc,struct xe_hw_engine *hwe,struct xe_vm *vm, +void *replay_state,u16 msix_vec,u32 init_flags);void xe_lrc_destroy (struct kref *ref);/** Signature · 1 line
2.43.0
PATCH v2 7/9 drm/xe/exec_queue: Introduce xe_exec_queue_reinit()
In reply to Raag JadavIn preparation of usecases which require re-initializing an exec queue after PCIe FLR, introduce xe_exec_queue_reinit() helper. All the exec queue LCRs already exist but the context is lost on PCIe FLR which needs re-initialization.
---
v2: Re-initialize migrate context (Matthew Brost)
--- drivers/gpu/drm/xe/xe_exec_queue.c | 34 ++++++++++++++++++++++++++---- drivers/gpu/drm/xe/xe_exec_queue.h | 1 + 2 files changed, 31 insertions(+), 4 deletions(-)
drivers/gpu/drm/xe/xe_exec_queue.c +30 -4
@@ -270,9 +270,8 @@ static struct xe_exec_queue *__xe_exec_queue_alloc(struct xe_device *xe,return q; } -static int __xe_exec_queue_init (struct xe_exec_queue *q,u32 exec_queue_flags) +static u32 xe_lrc_init_flags (struct xe_exec_queue *q,u32 exec_queue_flags) { -int i, err;u32 flags =0 ;/*
@@ -292,6 +291,13 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)if (!(exec_queue_flags &EXEC_QUEUE_FLAG_KERNEL )) flags |=XE_LRC_CREATE_USER_CTX ; +return flags; +} + +static int __xe_exec_queue_init (struct xe_exec_queue *q,u32 exec_queue_flags) +{ +int i, err; + err = q->ops->init (q);if (err)return err;
@@ -310,8 +316,8 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)struct xe_lrc *lrc;xe_gt_sriov_vf_wait_valid_ggtt (q->gt); - lrc =xe_lrc_create (q->hwe, q->vm, q->replay_state, -xe_lrc_ring_size (), q->msix_vec, flags); + lrc =xe_lrc_create (q->hwe, q->vm, q->replay_state,xe_lrc_ring_size (), + q->msix_vec,xe_lrc_init_flags (q, exec_queue_flags));if (IS_ERR (lrc)) { err =PTR_ERR (lrc); goto err_lrc;
@@ -329,6 +335,26 @@ static int __xe_exec_queue_init(struct xe_exec_queue *q, u32 exec_queue_flags)return err; } +/** +* xe_exec_queue_reinit() - Re-initialize exec queue +* @q: exec queue to re-initialize +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_exec_queue_reinit (struct xe_exec_queue *q) +{ +int i, err; + +for (i =0 ; i < q->width; i++) { + err =xe_lrc_reinit (q->lrc[i], q->hwe, q->vm, q->replay_state, + q->msix_vec,xe_lrc_init_flags (q, q->flags)); +if (err) +return err; + } + +return 0 ; +} +static void __xe_exec_queue_fini (struct xe_exec_queue *q) {int i;
drivers/gpu/drm/xe/xe_exec_queue.h +1 -0
@@ -34,6 +34,7 @@ struct xe_exec_queue *xe_exec_queue_create_bind(struct xe_device *xe,void xe_exec_queue_fini (struct xe_exec_queue *q);void xe_exec_queue_destroy (struct kref *ref);void xe_exec_queue_assign_name (struct xe_exec_queue *q,u32 instance); +int xe_exec_queue_reinit (struct xe_exec_queue *q);static inline struct xe_exec_queue *xe_exec_queue_get_unless_zero (struct xe_exec_queue *q)
Signature · 1 line
2.43.0
PATCH v2 8/9 drm/xe/migrate: Introduce xe_migrate_reinit()
In reply to Raag JadavIn preparation of usecases which require re-initializing migrate context after PCIe FLR, introduce xe_migrate_reinit() helper. Migrate exec queue and pt_bo already exist in migrate structure but since their contents live on VRAM, they are lost on PCIe FLR and need re-initialization.
---
v2: Re-initialize migrate context (Matthew Brost)
--- drivers/gpu/drm/xe/xe_gt.c | 10 +++++ drivers/gpu/drm/xe/xe_migrate.c | 65 +++++++++++++++++++++++++-------- drivers/gpu/drm/xe/xe_migrate.h | 1 + 3 files changed, 61 insertions(+), 15 deletions(-)
drivers/gpu/drm/xe/xe_gt.c +10 -0
@@ -966,6 +966,16 @@ void xe_gt_flr_prepare(struct xe_gt *gt)*/ int xe_gt_flr_done (struct xe_gt *gt) { +int err; + +if (xe_gt_is_main_type (gt)) { +struct xe_tile *tile =gt_to_tile (gt); + + err =xe_migrate_reinit (tile->migrate); +if (err) +return err; + } +return xe_uc_flr_done (>->uc); }
drivers/gpu/drm/xe/xe_migrate.c +50 -15
108 lines not on this page · raw message
drivers/gpu/drm/xe/xe_migrate.h +1 -0
@@ -112,6 +112,7 @@ struct xe_migrate_pt_update {struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m); +int xe_migrate_reinit (struct xe_migrate *m); struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, unsigned long npages,
Signature · 1 line
2.43.0
Re: [PATCH v2 8/9] drm/xe/migrate: Introduce xe_migrate_reinit()
Quoted 19 lines · Raag Jadav
In preparation of usecases which require re-initializing migrate context after PCIe FLR, introduce
xe_migrate_reinit()helper. Migrate exec queue andpt_boalready exist in migrate structure but since their contents live on VRAM, they are lost on PCIe FLR and need re-initialization.Signed-off-by Raag Jadav <[email protected]>---
v2: Re-initialize migrate context (Matthew Brost)--- drivers/gpu/drm/xe/xe_gt.c | 10 +++++ drivers/gpu/drm/xe/xe_migrate.c | 65 +++++++++++++++++++++++++-------- drivers/gpu/drm/xe/xe_migrate.h | 1 + 3 files changed, 61 insertions(+), 15 deletions(-)drivers/gpu/drm/xe/xe_gt.c +0
-0
drivers/gpu/drm/xe/xe_gt.c void xe_gt_flr_prepare(struct xe_gt *gt) · line 974
Quoted 9 lines
drivers/gpu/drm/xe/xe_gt.c +6
-0@@ -966,6 +966,16 @@ void xe_gt_flr_prepare(struct xe_gt *gt)*/ int xe_gt_flr_done (struct xe_gt *gt) { +int err; + +if (xe_gt_is_main_type (gt)) { +struct xe_tile *tile =gt_to_tile (gt); + + err =xe_migrate_reinit (tile->migrate);
I think there is work ahead for future platforms where multiple tiles point to the same migrate object, so we’ll need to keep an eye on that. Perhaps calling xe_migrate_reinit twice is harmless too.
Anyways the patch looks correct.
Quoted 140 lines
drivers/gpu/drm/xe/xe_gt.c +4
-0+if (err) +return err; + } +return xe_uc_flr_done (>->uc); }drivers/gpu/drm/xe/xe_migrate.c +50
-15@@ -184,19 +184,11 @@ static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm,xe_assert (xe, pos == vram_limit); } -static int xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, -struct xe_vm *vm,struct drm_exec *exec) +static int xe_migrate_pt_bo_alloc (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,struct drm_exec *exec) { -struct xe_device *xe =tile_to_xe (tile); -u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; -u8 id = tile->id; -u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; -#define VRAM_IDENTITY_MAP_COUNT 2 -u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; -#undef VRAM_IDENTITY_MAP_COUNT -u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; -u64 entry, pt29_ofs; +u32 num_entries =NUM_PT_SLOTS ;/* Can't bump NUM_PT_SLOTS too high */ BUILD_BUG_ON (NUM_PT_SLOTS >SZ_2M /XE_PAGE_SIZE );@@ -216,6 +208,24 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,if (IS_ERR (bo))return PTR_ERR (bo); + m->pt_bo = bo; +return 0 ; +} + +static void xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,u32 *ofs) +{ +struct xe_device *xe =tile_to_xe (tile); +u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; +u8 id = tile->id; +u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; +#define VRAM_IDENTITY_MAP_COUNT 2 +u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; +#undef VRAM_IDENTITY_MAP_COUNT +u32 map_ofs, level, i; +struct xe_bo *bo = m->pt_bo, *batch = tile->mem.kernel_bb_pool->bo; +u64 entry, pt29_ofs; +/* PT30 & PT31 reserved for 2M identity map */ pt29_ofs =xe_bo_size (bo) -3 *XE_PAGE_SIZE ; entry = vm->pt_ops->pde_encode_bo (bo, pt29_ofs);@@ -338,6 +348,12 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, } } +if (ofs) + *ofs = map_ofs; +} + +static void xe_migrate_suballoc_manager_init (struct xe_migrate *m,u32 map_ofs) +{/* * Example layout created above, with root level = 3: * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's @@ -363,9 +379,6 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,drm_suballoc_manager_init (&m->vm_update_sa, (size_t )(map_ofs /XE_PAGE_SIZE -NUM_KERNEL_PDE ) *NUM_VMUSA_UNIT_PER_PAGE ,0 ); - - m->pt_bo = bo; -return 0 ; }/* @@ -416,12 +429,22 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mstruct xe_device *xe =tile_to_xe (tile);struct xe_validation_ctx ctx;struct drm_exec exec; +u32 map_ofs;int err =0 ; xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) { err =xe_vm_drm_exec_lock (vm, &exec); +if (err) +return err; +drm_exec_retry_on_contention (&exec); - err =xe_migrate_prepare_vm (tile, m, vm, &exec); + + err =xe_migrate_pt_bo_alloc (tile, m, vm, &exec); +if (err) +return err; + +xe_migrate_prepare_vm (tile, m, vm, &map_ofs); +xe_migrate_suballoc_manager_init (m, map_ofs);drm_exec_retry_on_contention (&exec);xe_validation_retry_on_oom (&ctx, &err); }@@ -429,6 +452,18 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mreturn err; } +/** +* xe_migrate_reinit() - Re-initialize a migrate context +* @m: The migration context +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_migrate_reinit (struct xe_migrate *m) +{ +xe_migrate_prepare_vm (m->tile, m, m->q->vm,NULL ); +return xe_exec_queue_reinit (m->q); +} +/** * xe_migrate_init() - Initialize a migrate context * @m: The migration context drivers/gpu/drm/xe/xe_migrate.h +1
-0@@ -112,6 +112,7 @@ struct xe_migrate_pt_update {struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m); +int xe_migrate_reinit (struct xe_migrate *m); struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, unsigned long npages,Signature · 1 line
2.43.0
Re: [PATCH v2 8/9] drm/xe/migrate: Introduce xe_migrate_reinit()
Quoted 34 lines · Matthew Brost
Quoted 19 lines · Raag Jadav
In preparation of usecases which require re-initializing migrate context after PCIe FLR, introduce
xe_migrate_reinit()helper. Migrate exec queue andpt_boalready exist in migrate structure but since their contents live on VRAM, they are lost on PCIe FLR and need re-initialization.Signed-off-by Raag Jadav <[email protected]>---
v2: Re-initialize migrate context (Matthew Brost)--- drivers/gpu/drm/xe/xe_gt.c | 10 +++++ drivers/gpu/drm/xe/xe_migrate.c | 65 +++++++++++++++++++++++++-------- drivers/gpu/drm/xe/xe_migrate.h | 1 + 3 files changed, 61 insertions(+), 15 deletions(-)drivers/gpu/drm/xe/xe_gt.c +0
-0drivers/gpu/drm/xe/xe_gt.c void xe_gt_flr_prepare(struct xe_gt *gt) · line 974
Quoted 9 lines
drivers/gpu/drm/xe/xe_gt.c +6
-0@@ -966,6 +966,16 @@ void xe_gt_flr_prepare(struct xe_gt *gt)*/ int xe_gt_flr_done (struct xe_gt *gt) { +int err; + +if (xe_gt_is_main_type (gt)) { +struct xe_tile *tile =gt_to_tile (gt); + + err =xe_migrate_reinit (tile->migrate);I think there is work ahead for future platforms where multiple tiles point to the same migrate object, so we’ll need to keep an eye on that. Perhaps calling
xe_migrate_reinittwice is harmless too.
Quoted 3 lines
Anyways the patch looks correct.
Reviewed-by Matthew Brost <[email protected]>
Thank you.
Raag
Quoted 140 lines
Quoted 140 lines
drivers/gpu/drm/xe/xe_gt.c +4
-0+if (err) +return err; + } +return xe_uc_flr_done (>->uc); }drivers/gpu/drm/xe/xe_migrate.c +50
-15@@ -184,19 +184,11 @@ static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm,xe_assert (xe, pos == vram_limit); } -static int xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, -struct xe_vm *vm,struct drm_exec *exec) +static int xe_migrate_pt_bo_alloc (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,struct drm_exec *exec) { -struct xe_device *xe =tile_to_xe (tile); -u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; -u8 id = tile->id; -u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; -#define VRAM_IDENTITY_MAP_COUNT 2 -u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; -#undef VRAM_IDENTITY_MAP_COUNT -u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; -u64 entry, pt29_ofs; +u32 num_entries =NUM_PT_SLOTS ;/* Can't bump NUM_PT_SLOTS too high */ BUILD_BUG_ON (NUM_PT_SLOTS >SZ_2M /XE_PAGE_SIZE );@@ -216,6 +208,24 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,if (IS_ERR (bo))return PTR_ERR (bo); + m->pt_bo = bo; +return 0 ; +} + +static void xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,u32 *ofs) +{ +struct xe_device *xe =tile_to_xe (tile); +u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; +u8 id = tile->id; +u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; +#define VRAM_IDENTITY_MAP_COUNT 2 +u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; +#undef VRAM_IDENTITY_MAP_COUNT +u32 map_ofs, level, i; +struct xe_bo *bo = m->pt_bo, *batch = tile->mem.kernel_bb_pool->bo; +u64 entry, pt29_ofs; +/* PT30 & PT31 reserved for 2M identity map */ pt29_ofs =xe_bo_size (bo) -3 *XE_PAGE_SIZE ; entry = vm->pt_ops->pde_encode_bo (bo, pt29_ofs);@@ -338,6 +348,12 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, } } +if (ofs) + *ofs = map_ofs; +} + +static void xe_migrate_suballoc_manager_init (struct xe_migrate *m,u32 map_ofs) +{/* * Example layout created above, with root level = 3: * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's @@ -363,9 +379,6 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,drm_suballoc_manager_init (&m->vm_update_sa, (size_t )(map_ofs /XE_PAGE_SIZE -NUM_KERNEL_PDE ) *NUM_VMUSA_UNIT_PER_PAGE ,0 ); - - m->pt_bo = bo; -return 0 ; }/* @@ -416,12 +429,22 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mstruct xe_device *xe =tile_to_xe (tile);struct xe_validation_ctx ctx;struct drm_exec exec; +u32 map_ofs;int err =0 ; xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) { err =xe_vm_drm_exec_lock (vm, &exec); +if (err) +return err; +drm_exec_retry_on_contention (&exec); - err =xe_migrate_prepare_vm (tile, m, vm, &exec); + + err =xe_migrate_pt_bo_alloc (tile, m, vm, &exec); +if (err) +return err; + +xe_migrate_prepare_vm (tile, m, vm, &map_ofs); +xe_migrate_suballoc_manager_init (m, map_ofs);drm_exec_retry_on_contention (&exec);xe_validation_retry_on_oom (&ctx, &err); }@@ -429,6 +452,18 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mreturn err; } +/** +* xe_migrate_reinit() - Re-initialize a migrate context +* @m: The migration context +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_migrate_reinit (struct xe_migrate *m) +{ +xe_migrate_prepare_vm (m->tile, m, m->q->vm,NULL ); +return xe_exec_queue_reinit (m->q); +} +/** * xe_migrate_init() - Initialize a migrate context * @m: The migration context drivers/gpu/drm/xe/xe_migrate.h +1
-0@@ -112,6 +112,7 @@ struct xe_migrate_pt_update {struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m); +int xe_migrate_reinit (struct xe_migrate *m); struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, unsigned long npages,Signature · 1 line
2.43.0
Re: [PATCH v2 8/9] drm/xe/migrate: Introduce xe_migrate_reinit()
Quoted 35 lines · Raag Jadav
Quoted 34 lines · Matthew Brost
Quoted 19 lines · Raag Jadav
In preparation of usecases which require re-initializing migrate context after PCIe FLR, introduce
xe_migrate_reinit()helper. Migrate exec queue andpt_boalready exist in migrate structure but since their contents live on VRAM, they are lost on PCIe FLR and need re-initialization.Signed-off-by Raag Jadav <[email protected]>---
v2: Re-initialize migrate context (Matthew Brost)--- drivers/gpu/drm/xe/xe_gt.c | 10 +++++ drivers/gpu/drm/xe/xe_migrate.c | 65 +++++++++++++++++++++++++-------- drivers/gpu/drm/xe/xe_migrate.h | 1 + 3 files changed, 61 insertions(+), 15 deletions(-)drivers/gpu/drm/xe/xe_gt.c +0
-0drivers/gpu/drm/xe/xe_gt.c void xe_gt_flr_prepare(struct xe_gt *gt) · line 974
Quoted 9 lines
drivers/gpu/drm/xe/xe_gt.c +6
-0@@ -966,6 +966,16 @@ void xe_gt_flr_prepare(struct xe_gt *gt)*/ int xe_gt_flr_done (struct xe_gt *gt) { +int err; + +if (xe_gt_is_main_type (gt)) { +struct xe_tile *tile =gt_to_tile (gt); + + err =xe_migrate_reinit (tile->migrate);I think there is work ahead for future platforms where multiple tiles point to the same migrate object, so we’ll need to keep an eye on that. Perhaps calling
xe_migrate_reinittwice is harmless too.
Quoted 5 lines
Quoted 3 lines
Anyways the patch looks correct.
Reviewed-by Matthew Brost <[email protected]>Thank you.
Similar to LRC this looks like a candidate for independent refactor. I'll retain the tag and send it out separately.
Raag
Quoted 140 lines
Quoted 140 lines
Quoted 140 lines
drivers/gpu/drm/xe/xe_gt.c +4
-0+if (err) +return err; + } +return xe_uc_flr_done (>->uc); }drivers/gpu/drm/xe/xe_migrate.c +50
-15@@ -184,19 +184,11 @@ static void xe_migrate_program_identity(struct xe_device *xe, struct xe_vm *vm,xe_assert (xe, pos == vram_limit); } -static int xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, -struct xe_vm *vm,struct drm_exec *exec) +static int xe_migrate_pt_bo_alloc (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,struct drm_exec *exec) { -struct xe_device *xe =tile_to_xe (tile); -u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; -u8 id = tile->id; -u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; -#define VRAM_IDENTITY_MAP_COUNT 2 -u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; -#undef VRAM_IDENTITY_MAP_COUNT -u32 map_ofs, level, i;struct xe_bo *bo, *batch = tile->mem.kernel_bb_pool->bo; -u64 entry, pt29_ofs; +u32 num_entries =NUM_PT_SLOTS ;/* Can't bump NUM_PT_SLOTS too high */ BUILD_BUG_ON (NUM_PT_SLOTS >SZ_2M /XE_PAGE_SIZE );@@ -216,6 +208,24 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,if (IS_ERR (bo))return PTR_ERR (bo); + m->pt_bo = bo; +return 0 ; +} + +static void xe_migrate_prepare_vm (struct xe_tile *tile,struct xe_migrate *m, +struct xe_vm *vm,u32 *ofs) +{ +struct xe_device *xe =tile_to_xe (tile); +u16 pat_index = xe->pat.idx[XE_CACHE_WB ]; +u8 id = tile->id; +u32 num_entries =NUM_PT_SLOTS , num_level = vm->pt_root[id]->level; +#define VRAM_IDENTITY_MAP_COUNT 2 +u32 num_setup = num_level +VRAM_IDENTITY_MAP_COUNT ; +#undef VRAM_IDENTITY_MAP_COUNT +u32 map_ofs, level, i; +struct xe_bo *bo = m->pt_bo, *batch = tile->mem.kernel_bb_pool->bo; +u64 entry, pt29_ofs; +/* PT30 & PT31 reserved for 2M identity map */ pt29_ofs =xe_bo_size (bo) -3 *XE_PAGE_SIZE ; entry = vm->pt_ops->pde_encode_bo (bo, pt29_ofs);@@ -338,6 +348,12 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m, } } +if (ofs) + *ofs = map_ofs; +} + +static void xe_migrate_suballoc_manager_init (struct xe_migrate *m,u32 map_ofs) +{/* * Example layout created above, with root level = 3: * [PT0...PT7]: kernel PT's for copy/clear; 64 or 4KiB PTE's @@ -363,9 +379,6 @@ static int xe_migrate_prepare_vm(struct xe_tile *tile, struct xe_migrate *m,drm_suballoc_manager_init (&m->vm_update_sa, (size_t )(map_ofs /XE_PAGE_SIZE -NUM_KERNEL_PDE ) *NUM_VMUSA_UNIT_PER_PAGE ,0 ); - - m->pt_bo = bo; -return 0 ; }/* @@ -416,12 +429,22 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mstruct xe_device *xe =tile_to_xe (tile);struct xe_validation_ctx ctx;struct drm_exec exec; +u32 map_ofs;int err =0 ; xe_validation_guard(&ctx, &xe->val, &exec, (struct xe_val_flags) {}, err) { err =xe_vm_drm_exec_lock (vm, &exec); +if (err) +return err; +drm_exec_retry_on_contention (&exec); - err =xe_migrate_prepare_vm (tile, m, vm, &exec); + + err =xe_migrate_pt_bo_alloc (tile, m, vm, &exec); +if (err) +return err; + +xe_migrate_prepare_vm (tile, m, vm, &map_ofs); +xe_migrate_suballoc_manager_init (m, map_ofs);drm_exec_retry_on_contention (&exec);xe_validation_retry_on_oom (&ctx, &err); }@@ -429,6 +452,18 @@ static int xe_migrate_lock_prepare_vm(struct xe_tile *tile, struct xe_migrate *mreturn err; } +/** +* xe_migrate_reinit() - Re-initialize a migrate context +* @m: The migration context +* +* Returns: 0 on success, negative error code otherwise. +*/ +int xe_migrate_reinit (struct xe_migrate *m) +{ +xe_migrate_prepare_vm (m->tile, m, m->q->vm,NULL ); +return xe_exec_queue_reinit (m->q); +} +/** * xe_migrate_init() - Initialize a migrate context * @m: The migration context drivers/gpu/drm/xe/xe_migrate.h +1
-0@@ -112,6 +112,7 @@ struct xe_migrate_pt_update {struct xe_migrate *xe_migrate_alloc (struct xe_tile *tile);int xe_migrate_init (struct xe_migrate *m); +int xe_migrate_reinit (struct xe_migrate *m); struct dma_fence *xe_migrate_to_vram(struct xe_migrate *m, unsigned long npages,Signature · 1 line
2.43.0
PATCH v2 9/9 drm/xe/pci: Introduce PCIe FLR
In reply to Raag JadavWith all the pieces in place, we can finally introduce PCIe Function Level Reset (FLR) handling which re-initializes hardware state without the need for reloading the driver from userspace. All VRAM contents are lost along with hardware state, so the driver takes care of recreating the required kernel bos as part of re-initialization, but user still needs to recreate user bos and reload context after PCIe FLR.
---
v2: Spell out Function Level Reset (Jani)
--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 ++++++++++++++++++++++++++++++++ 4 files changed, 154 insertions(+) create mode 100644 drivers/gpu/drm/xe/xe_pci_err.c
drivers/gpu/drm/xe/Makefile +1 -0
@@ -99,6 +99,7 @@ xe-y += xe_bb.o \ xe_page_reclaim.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pci_rebar.o \ xe_pcode.o \ xe_pm.o \
drivers/gpu/drm/xe/xe_pci.c +1 -0
@@ -1301,6 +1301,7 @@ static struct pci_driver xe_pci_driver = {#ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, };/**
drivers/gpu/drm/xe/xe_pci.h +2 -0
@@ -8,6 +8,8 @@struct pci_dev ; +extern const struct pci_error_handlers xe_pci_err_handlers; +int xe_register_pci_driver (void );void xe_unregister_pci_driver (void );struct xe_device *xe_pci_to_pf_device (struct pci_dev *pdev);
drivers/gpu/drm/xe/xe_pci_err.c new file +150 -0
150 lines not on this page · raw message
Signature · 1 line
2.43.0
Vivi, Rodrigo
maintainerRe: [PATCH v2 9/9] drm/xe/pci: Introduce PCIe FLR
Quoted 12 lines · Raag Jadav
With all the pieces in place, we can finally introduce PCIe Function Level
Reset (FLR) handling which re-initializes hardware state without the need
for reloading the driver from userspace. All VRAM contents are lost along
with hardware state, so the driver takes care of recreating the required
kernel bos as part of re-initialization, but user still needs to recreate
user bos and reload context after PCIe FLR.
Quoted 1 line
Signed-off-by Raag Jadav <[email protected]>
Do we really have everything in place? I'm missing the handling of VF here. Hence we would have the same deadlock pointed out by Michal when Aravind attempted it last time, no?!
We probably need to use the guc relay communication here so PF can inform VF of the FLR, then stop all the workload and wedge the device, before the AER triggers a kill of the VM, no?!
Quoted 217 lines
---
v2: Spell out Function Level Reset (Jani)
---drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 ++++++++++++++++++++++++++++++++ 4 files changed, 154 insertions(+)create mode 100644
drivers/gpu/drm/xe/xe_pci_err.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -99,6 +99,7 @@ xe-y += xe_bb.o \ xe_page_reclaim.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pci_rebar.o \ xe_pcode.o \ xe_pm.o \drivers/gpu/drm/xe/xe_pci.c +1
-0@@ -1301,6 +1301,7 @@ static struct pci_driver xe_pci_driver = { #ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, };
/**drivers/gpu/drm/xe/xe_pci.h +0
-0@@ -8,6 +8,8 @@
structpci_dev;drivers/gpu/drm/xe/xe_pci.h +2
-0+extern const struct pci_error_handlers xe_pci_err_handlers; +int
xe_register_pci_driver(void);
voidxe_unregister_pci_driver(void);
structxe_device*xe_pci_to_pf_device(structpci_dev*pdev);drivers/gpu/drm/xe/xe_pci_err.c new file +93
-0@@ -0,0 +1,150 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include "xe_bo_evict.h" +#include "xe_device.h" +#include "xe_gt.h" +#include "xe_gt_idle.h" +#include "xe_i2c.h" +#include "xe_irq.h" +#include "xe_late_bind_fw.h" +#include "xe_pci.h" +#include "xe_pcode.h" +#include "xe_printk.h" +#include "xe_pxp.h" +#include "xe_wa.h" + +static int xe_flr_prepare (struct xe_device *xe) +{ +struct xe_gt *gt; +int err; +u8 id; + + err =xe_pxp_pm_suspend (xe->pxp); +if (err) +return err; + +xe_late_bind_wait_for_worker_completion (&xe->late_bind); + +for_each_gt (gt, xe, id) +xe_gt_flr_prepare (gt); + +xe_irq_disable (xe); + +// TODO: Drop all user bos +xe_bo_pci_dev_remove_pinned (xe); + +return 0 ; +} + +static int xe_flr_done (struct xe_device *xe) +{ +struct xe_tile *tile; +struct xe_gt *gt; +int err; +u8 id; + +for_each_gt (gt, xe, id) +xe_gt_idle_disable_c6 (gt); + +for_each_tile (tile, xe, id) +xe_wa_apply_tile_workarounds (tile); + + err =xe_pcode_ready (xe, true); +if (err) +return err; + +xe_device_assert_lmem_ready (xe); + + err =xe_bo_restore_map (xe); +if (err) +return err; + +for_each_gt (gt, xe, id) { + err =xe_gt_flr_done (gt); +if (err) +return err; + } + +xe_i2c_pm_resume (xe, true); + +xe_irq_resume (xe); + +for_each_gt (gt, xe, id) { + err =xe_gt_resume (gt); +if (err) +return err; + } + +xe_pxp_pm_resume (xe->pxp); + +xe_late_bind_fw_load (&xe->late_bind); + +return 0 ; +} + +static void xe_pci_reset_prepare (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display) {drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+xe_err (xe,"PCIe FLR not supported\n" ); +return ; + } + +/* Wedge the device to prevent userspace access but don't send the event yet */
drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+atomic_set (&xe->wedged.flag,1 ); + +/* +* The hardware could be in corrupted state and access unreliable, but we try to
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* update data structures and cleanup any pending work to avoid side effects during
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* PCIe FLR. This will be similar to xe_pm_suspend() flow but without migration.
drivers/gpu/drm/xe/xe_pci_err.c new file +15
-0+*/ +if (xe_flr_prepare (xe)) { +xe_err (xe,"Failed to prepare for PCIe FLR\n" ); +return ; + } + +xe_info (xe,"Prepared for PCIe FLR\n" ); +} + +static void xe_pci_reset_done (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display)drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+return ; + +if (!xe_device_wedged (xe)) { + xe_err(xe, "Device in unexpected state, re-initialization aborted\n");
drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+return ; + } + +/* +* We already have the data structures intact, so try to re- initialize the device.
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* This will be similar to xe_pm_resume() flow, except we'll also need to recreate
drivers/gpu/drm/xe/xe_pci_err.c new file +21
-0+* all VRAM contents. +*/ +if (xe_flr_done (xe)) { +xe_err (xe,"Re-initialization failed\n" ); +return ; + } + +/* Unwedge to allow userspace access */ +atomic_set (&xe->wedged.flag,0 ); + +xe_info (xe,"Re-initialization success\n" ); +} + +/* +* PCIe Function Level Reset (FLR) support only. +* TODO: Add PCIe error handlers using similar flow. +*/ +const struct pci_error_handlers xe_pci_err_handlers = { + .reset_prepare = xe_pci_reset_prepare, + .reset_done = xe_pci_reset_done, +};
Re: [PATCH v2 9/9] drm/xe/pci: Introduce PCIe FLR
Quoted 13 lines · Rodrigo
Quoted 12 lines · Raag Jadav
With all the pieces in place, we can finally introduce PCIe Function Level
Reset (FLR) handling which re-initializes hardware state without the need
for reloading the driver from userspace. All VRAM contents are lost along
with hardware state, so the driver takes care of recreating the required
kernel bos as part of re-initialization, but user still needs to recreate
user bos and reload context after PCIe FLR.
Quoted 8 lines
Quoted 1 line
Signed-off-by Raag Jadav <[email protected]>Cc Michał Winiarski <[email protected]>Cc Aravind Iddamsetty <[email protected]>Do we really have everything in place? I'm missing the handling of VF here. Hence we would have the same deadlock pointed out by Michal when Aravind attempted it last time, no?!
There's a whole lot of TODOs in this file, so perhaps poor choice of words.
Quoted 3 lines
We probably need to use the guc relay communication here so PF can inform VF of the FLR, then stop all the workload and wedge the device, before the AER triggers a kill of the VM, no?!
I was hoping to have the base functionality in place and build stuff on top, but I'll let you all make the final call here.
Raag
Quoted 217 lines
Quoted 217 lines
---
v2: Spell out Function Level Reset (Jani)
---drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 ++++++++++++++++++++++++++++++++ 4 files changed, 154 insertions(+)create mode 100644
drivers/gpu/drm/xe/xe_pci_err.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -99,6 +99,7 @@ xe-y += xe_bb.o \ xe_page_reclaim.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pci_rebar.o \ xe_pcode.o \ xe_pm.o \drivers/gpu/drm/xe/xe_pci.c +1
-0@@ -1301,6 +1301,7 @@ static struct pci_driver xe_pci_driver = { #ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, };
/**drivers/gpu/drm/xe/xe_pci.h +0
-0@@ -8,6 +8,8 @@
structpci_dev;drivers/gpu/drm/xe/xe_pci.h +2
-0+extern const struct pci_error_handlers xe_pci_err_handlers; +int
xe_register_pci_driver(void);
voidxe_unregister_pci_driver(void);
structxe_device*xe_pci_to_pf_device(structpci_dev*pdev);drivers/gpu/drm/xe/xe_pci_err.c new file +93
-0@@ -0,0 +1,150 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include "xe_bo_evict.h" +#include "xe_device.h" +#include "xe_gt.h" +#include "xe_gt_idle.h" +#include "xe_i2c.h" +#include "xe_irq.h" +#include "xe_late_bind_fw.h" +#include "xe_pci.h" +#include "xe_pcode.h" +#include "xe_printk.h" +#include "xe_pxp.h" +#include "xe_wa.h" + +static int xe_flr_prepare (struct xe_device *xe) +{ +struct xe_gt *gt; +int err; +u8 id; + + err =xe_pxp_pm_suspend (xe->pxp); +if (err) +return err; + +xe_late_bind_wait_for_worker_completion (&xe->late_bind); + +for_each_gt (gt, xe, id) +xe_gt_flr_prepare (gt); + +xe_irq_disable (xe); + +// TODO: Drop all user bos +xe_bo_pci_dev_remove_pinned (xe); + +return 0 ; +} + +static int xe_flr_done (struct xe_device *xe) +{ +struct xe_tile *tile; +struct xe_gt *gt; +int err; +u8 id; + +for_each_gt (gt, xe, id) +xe_gt_idle_disable_c6 (gt); + +for_each_tile (tile, xe, id) +xe_wa_apply_tile_workarounds (tile); + + err =xe_pcode_ready (xe, true); +if (err) +return err; + +xe_device_assert_lmem_ready (xe); + + err =xe_bo_restore_map (xe); +if (err) +return err; + +for_each_gt (gt, xe, id) { + err =xe_gt_flr_done (gt); +if (err) +return err; + } + +xe_i2c_pm_resume (xe, true); + +xe_irq_resume (xe); + +for_each_gt (gt, xe, id) { + err =xe_gt_resume (gt); +if (err) +return err; + } + +xe_pxp_pm_resume (xe->pxp); + +xe_late_bind_fw_load (&xe->late_bind); + +return 0 ; +} + +static void xe_pci_reset_prepare (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display) {drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+xe_err (xe,"PCIe FLR not supported\n" ); +return ; + } + +/* Wedge the device to prevent userspace access but don't send the event yet */
drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+atomic_set (&xe->wedged.flag,1 ); + +/* +* The hardware could be in corrupted state and access unreliable, but we try to
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* update data structures and cleanup any pending work to avoid side effects during
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* PCIe FLR. This will be similar to xe_pm_suspend() flow but without migration.
drivers/gpu/drm/xe/xe_pci_err.c new file +15
-0+*/ +if (xe_flr_prepare (xe)) { +xe_err (xe,"Failed to prepare for PCIe FLR\n" ); +return ; + } + +xe_info (xe,"Prepared for PCIe FLR\n" ); +} + +static void xe_pci_reset_done (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display)drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+return ; + +if (!xe_device_wedged (xe)) { + xe_err(xe, "Device in unexpected state, re-initialization aborted\n");
drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+return ; + } + +/* +* We already have the data structures intact, so try to re- initialize the device.
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* This will be similar to xe_pm_resume() flow, except we'll also need to recreate
drivers/gpu/drm/xe/xe_pci_err.c new file +21
-0+* all VRAM contents. +*/ +if (xe_flr_done (xe)) { +xe_err (xe,"Re-initialization failed\n" ); +return ; + } + +/* Unwedge to allow userspace access */ +atomic_set (&xe->wedged.flag,0 ); + +xe_info (xe,"Re-initialization success\n" ); +} + +/* +* PCIe Function Level Reset (FLR) support only. +* TODO: Add PCIe error handlers using similar flow. +*/ +const struct pci_error_handlers xe_pci_err_handlers = { + .reset_prepare = xe_pci_reset_prepare, + .reset_done = xe_pci_reset_done, +};
Rodrigo Vivi
maintainerRe: [PATCH v2 9/9] drm/xe/pci: Introduce PCIe FLR
Quoted 23 lines · Raag Jadav
Quoted 13 lines · Rodrigo
Quoted 12 lines · Raag Jadav
With all the pieces in place, we can finally introduce PCIe Function Level
Reset (FLR) handling which re-initializes hardware state without the need
for reloading the driver from userspace. All VRAM contents are lost along
with hardware state, so the driver takes care of recreating the required
kernel bos as part of re-initialization, but user still needs to recreate
user bos and reload context after PCIe FLR.Quoted 8 lines
Quoted 1 line
Signed-off-by Raag Jadav <[email protected]>Cc Michał Winiarski <[email protected]>Cc Aravind Iddamsetty <[email protected]>Do we really have everything in place? I'm missing the handling of VF here. Hence we would have the same deadlock pointed out by Michal when Aravind attempted it last time, no?!
Quoted 8 lines
There's a whole lot of TODOs in this file, so perhaps poor choice of words.
Quoted 3 lines
We probably need to use the guc relay communication here so PF can inform VF of the FLR, then stop all the workload and wedge the device, before the AER triggers a kill of the VM, no?!
I was hoping to have the base functionality in place and build stuff on top, but I'll let you all make the final call here.
I definitely support this approach. Let's gets the basic in and build on top.
Quoted 220 lines
Raag
Quoted 217 lines
Quoted 217 lines
---
v2: Spell out Function Level Reset (Jani)
---drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 ++++++++++++++++++++++++++++++++ 4 files changed, 154 insertions(+)create mode 100644
drivers/gpu/drm/xe/xe_pci_err.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -99,6 +99,7 @@ xe-y += xe_bb.o \ xe_page_reclaim.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pci_rebar.o \ xe_pcode.o \ xe_pm.o \drivers/gpu/drm/xe/xe_pci.c +1
-0@@ -1301,6 +1301,7 @@ static struct pci_driver xe_pci_driver = { #ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, };
/**drivers/gpu/drm/xe/xe_pci.h +0
-0@@ -8,6 +8,8 @@
structpci_dev;drivers/gpu/drm/xe/xe_pci.h +2
-0+extern const struct pci_error_handlers xe_pci_err_handlers; +int
xe_register_pci_driver(void);
voidxe_unregister_pci_driver(void);
structxe_device*xe_pci_to_pf_device(structpci_dev*pdev);drivers/gpu/drm/xe/xe_pci_err.c new file +93
-0@@ -0,0 +1,150 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include "xe_bo_evict.h" +#include "xe_device.h" +#include "xe_gt.h" +#include "xe_gt_idle.h" +#include "xe_i2c.h" +#include "xe_irq.h" +#include "xe_late_bind_fw.h" +#include "xe_pci.h" +#include "xe_pcode.h" +#include "xe_printk.h" +#include "xe_pxp.h" +#include "xe_wa.h" + +static int xe_flr_prepare (struct xe_device *xe) +{ +struct xe_gt *gt; +int err; +u8 id; + + err =xe_pxp_pm_suspend (xe->pxp); +if (err) +return err; + +xe_late_bind_wait_for_worker_completion (&xe->late_bind); + +for_each_gt (gt, xe, id) +xe_gt_flr_prepare (gt); + +xe_irq_disable (xe); + +// TODO: Drop all user bos +xe_bo_pci_dev_remove_pinned (xe); + +return 0 ; +} + +static int xe_flr_done (struct xe_device *xe) +{ +struct xe_tile *tile; +struct xe_gt *gt; +int err; +u8 id; + +for_each_gt (gt, xe, id) +xe_gt_idle_disable_c6 (gt); + +for_each_tile (tile, xe, id) +xe_wa_apply_tile_workarounds (tile); + + err =xe_pcode_ready (xe, true); +if (err) +return err; + +xe_device_assert_lmem_ready (xe); + + err =xe_bo_restore_map (xe); +if (err) +return err; + +for_each_gt (gt, xe, id) { + err =xe_gt_flr_done (gt); +if (err) +return err; + } + +xe_i2c_pm_resume (xe, true); + +xe_irq_resume (xe); + +for_each_gt (gt, xe, id) { + err =xe_gt_resume (gt); +if (err) +return err; + } + +xe_pxp_pm_resume (xe->pxp); + +xe_late_bind_fw_load (&xe->late_bind); + +return 0 ; +} + +static void xe_pci_reset_prepare (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display) {drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+xe_err (xe,"PCIe FLR not supported\n" ); +return ; + } + +/* Wedge the device to prevent userspace access but don't send the event yet */
drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+atomic_set (&xe->wedged.flag,1 ); + +/* +* The hardware could be in corrupted state and access unreliable, but we try to
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* update data structures and cleanup any pending work to avoid side effects during
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* PCIe FLR. This will be similar to xe_pm_suspend() flow but without migration.
drivers/gpu/drm/xe/xe_pci_err.c new file +15
-0+*/ +if (xe_flr_prepare (xe)) { +xe_err (xe,"Failed to prepare for PCIe FLR\n" ); +return ; + } + +xe_info (xe,"Prepared for PCIe FLR\n" ); +} + +static void xe_pci_reset_done (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display)drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+return ; + +if (!xe_device_wedged (xe)) { + xe_err(xe, "Device in unexpected state, re-initialization aborted\n");
drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+return ; + } + +/* +* We already have the data structures intact, so try to re- initialize the device.
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* This will be similar to xe_pm_resume() flow, except we'll also need to recreate
drivers/gpu/drm/xe/xe_pci_err.c new file +21
-0+* all VRAM contents. +*/ +if (xe_flr_done (xe)) { +xe_err (xe,"Re-initialization failed\n" ); +return ; + } + +/* Unwedge to allow userspace access */ +atomic_set (&xe->wedged.flag,0 ); + +xe_info (xe,"Re-initialization success\n" ); +} + +/* +* PCIe Function Level Reset (FLR) support only. +* TODO: Add PCIe error handlers using similar flow. +*/ +const struct pci_error_handlers xe_pci_err_handlers = { + .reset_prepare = xe_pci_reset_prepare, + .reset_done = xe_pci_reset_done, +};
Re: [PATCH v2 9/9] drm/xe/pci: Introduce PCIe FLR
In reply to Vivi, RodrigoQuoted 13 lines · Rodrigo
Quoted 12 lines · Raag Jadav
With all the pieces in place, we can finally introduce PCIe Function Level
Reset (FLR) handling which re-initializes hardware state without the need
for reloading the driver from userspace. All VRAM contents are lost along
with hardware state, so the driver takes care of recreating the required
kernel bos as part of re-initialization, but user still needs to recreate
user bos and reload context after PCIe FLR.
Quoted 7 lines
Quoted 1 line
Signed-off-by Raag Jadav <[email protected]>Cc Michał Winiarski <[email protected]>Cc Aravind Iddamsetty <[email protected]>Do we really have everything in place? I'm missing the handling of VF here. Hence we would have the same deadlock pointed out by Michal when Aravind attempted it last time, no?!
This seems to be different from Aravind's previous attempt. Here we don't remove the driver and as a result we won't attempt to disable VFs, so the deadlock scenario shouldn't apply.
Lukasz
Quoted 221 lines
We probably need to use the guc relay communication here so PF can inform VF of the FLR, then stop all the workload and wedge the device, before the AER triggers a kill of the VM, no?!
Quoted 217 lines
---
v2: Spell out Function Level Reset (Jani)--- drivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150 ++++++++++++++++++++++++++++++++ 4 files changed, 154 insertions(+)create mode 100644
drivers/gpu/drm/xe/xe_pci_err.cdrivers/gpu/drm/xe/Makefile +1
-0@@ -99,6 +99,7 @@ xe-y += xe_bb.o \ xe_page_reclaim.o \ xe_pat.o \ xe_pci.o \ + xe_pci_err.o \ xe_pci_rebar.o \ xe_pcode.o \ xe_pm.o \drivers/gpu/drm/xe/xe_pci.c +1
-0@@ -1301,6 +1301,7 @@ static struct pci_driver xe_pci_driver = { #ifdef CONFIG_PM_SLEEP .driver.pm = &xe_pm_ops, #endif + .err_handler = &xe_pci_err_handlers, }; /**drivers/gpu/drm/xe/xe_pci.h +2
-0@@ -8,6 +8,8 @@ struct pci_dev; +extern const struct pci_error_handlers xe_pci_err_handlers; + int xe_register_pci_driver(void); void xe_unregister_pci_driver(void); struct xe_device *xe_pci_to_pf_device(struct pci_dev *pdev);drivers/gpu/drm/xe/xe_pci_err.c new file +93
-0@@ -0,0 +1,150 @@ +// SPDX-License-Identifier: MIT +/* +* Copyright © 2026 Intel Corporation +*/ + +#include "xe_bo_evict.h" +#include "xe_device.h" +#include "xe_gt.h" +#include "xe_gt_idle.h" +#include "xe_i2c.h" +#include "xe_irq.h" +#include "xe_late_bind_fw.h" +#include "xe_pci.h" +#include "xe_pcode.h" +#include "xe_printk.h" +#include "xe_pxp.h" +#include "xe_wa.h" + +static int xe_flr_prepare (struct xe_device *xe) +{ +struct xe_gt *gt; +int err; +u8 id; + + err =xe_pxp_pm_suspend (xe->pxp); +if (err) +return err; + +xe_late_bind_wait_for_worker_completion (&xe->late_bind); + +for_each_gt (gt, xe, id) +xe_gt_flr_prepare (gt); + +xe_irq_disable (xe); + +// TODO: Drop all user bos +xe_bo_pci_dev_remove_pinned (xe); + +return 0 ; +} + +static int xe_flr_done (struct xe_device *xe) +{ +struct xe_tile *tile; +struct xe_gt *gt; +int err; +u8 id; + +for_each_gt (gt, xe, id) +xe_gt_idle_disable_c6 (gt); + +for_each_tile (tile, xe, id) +xe_wa_apply_tile_workarounds (tile); + + err =xe_pcode_ready (xe, true); +if (err) +return err; + +xe_device_assert_lmem_ready (xe); + + err =xe_bo_restore_map (xe); +if (err) +return err; + +for_each_gt (gt, xe, id) { + err =xe_gt_flr_done (gt); +if (err) +return err; + } + +xe_i2c_pm_resume (xe, true); + +xe_irq_resume (xe); + +for_each_gt (gt, xe, id) { + err =xe_gt_resume (gt); +if (err) +return err; + } + +xe_pxp_pm_resume (xe->pxp); + +xe_late_bind_fw_load (&xe->late_bind); + +return 0 ; +} + +static void xe_pci_reset_prepare (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display) {drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+xe_err (xe,"PCIe FLR not supported\n" ); +return ; + } + +/* Wedge the device to prevent userspace access but don't send the event yet */
drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+atomic_set (&xe->wedged.flag,1 ); + +/* +* The hardware could be in corrupted state and access unreliable, but we try to
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* update data structures and cleanup any pending work to avoid side effects during
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* PCIe FLR. This will be similar to xe_pm_suspend() flow but without migration.
drivers/gpu/drm/xe/xe_pci_err.c new file +15
-0+*/ +if (xe_flr_prepare (xe)) { +xe_err (xe,"Failed to prepare for PCIe FLR\n" ); +return ; + } + +xe_info (xe,"Prepared for PCIe FLR\n" ); +} + +static void xe_pci_reset_done (struct pci_dev *pdev) +{ +struct xe_device *xe =pdev_to_xe_device (pdev); + +/* TODO: Extend support as a follow-up */ + if (!IS_DGFX(xe) || IS_SRIOV_VF(xe) || pci_num_vf(pdev) ||
xe->info.probe_display)drivers/gpu/drm/xe/xe_pci_err.c new file +4
-0+return ; + +if (!xe_device_wedged (xe)) { + xe_err(xe, "Device in unexpected state, re-initialization aborted\n");
drivers/gpu/drm/xe/xe_pci_err.c new file +5
-0+return ; + } + +/* +* We already have the data structures intact, so try to re- initialize the device.
drivers/gpu/drm/xe/xe_pci_err.c new file +1
-0+* This will be similar to xe_pm_resume() flow, except we'll also need to recreate
drivers/gpu/drm/xe/xe_pci_err.c new file +21
-0+* all VRAM contents. +*/ +if (xe_flr_done (xe)) { +xe_err (xe,"Re-initialization failed\n" ); +return ; + } + +/* Unwedge to allow userspace access */ +atomic_set (&xe->wedged.flag,0 ); + +xe_info (xe,"Re-initialization success\n" ); +} + +/* +* PCIe Function Level Reset (FLR) support only. +* TODO: Add PCIe error handlers using similar flow. +*/ +const struct pci_error_handlers xe_pci_err_handlers = { + .reset_prepare = xe_pci_reset_prepare, + .reset_done = xe_pci_reset_done, +};
Vivi, Rodrigo
maintainerQuoted 9 lines · Raag Jadav
Here's my humble attempt at introducing PCIe Function Level Reset (FLR)
support in xe driver. This is ofcourse a half baked implementation and
only limited to reloading GT. This needs to be extended for a lot of different components which I've skipped here for my lack of competence,
so feel free to join in and support them.
Quoted 7 lines
PS: All
xe_exec_basictests and clpeak run smoothly after FLR. Give it
a spin and let me know if any regressions.Trigger it with:
$ echo 1 > /sys/bus/pci/devices/<BDF>/reset
We definitely need this.
Thank you so much for putting it together!
Quoted 48 lines
v2: Re-initialize migrate context (Matthew Brost)
Add kernel doc (Matthew Brost)
Spell out Function Level Reset (Jani)Raag Jadav (9):
drm/xe/uc_fw: Allow re-initializing firmware drm/xe/gt: Introduce FLR helpers drm/xe/irq: Introducexe_irq_disable()drm/xe: Introducexe_device_assert_lmem_ready()drm/xe/bo_evict: Introducexe_bo_restore_map()drm/xe/lrc: Introducexe_lrc_reinit()drm/xe/exec_queue: Introducexe_exec_queue_reinit()drm/xe/migrate: Introducexe_migrate_reinit()drm/xe/pci: Introduce PCIe FLRdrivers/gpu/drm/xe/Makefile | 1 + drivers/gpu/drm/xe/xe_bo_evict.c | 51 ++++++++-- drivers/gpu/drm/xe/xe_bo_evict.h | 2 + drivers/gpu/drm/xe/xe_device.c | 10 +- drivers/gpu/drm/xe/xe_device.h | 1 +
drivers/gpu/drm/xe/xe_exec_queue.c| 34 ++++++-drivers/gpu/drm/xe/xe_exec_queue.h| 1 + drivers/gpu/drm/xe/xe_gsc.c | 14 +++ drivers/gpu/drm/xe/xe_gsc.h | 1 + drivers/gpu/drm/xe/xe_gt.c | 32 ++++++ drivers/gpu/drm/xe/xe_gt.h | 2 +drivers/gpu/drm/xe/xe_guc.c | 14 +++ drivers/gpu/drm/xe/xe_guc.h | 1 + drivers/gpu/drm/xe/xe_huc.c | 14 +++ drivers/gpu/drm/xe/xe_huc.h | 1 + drivers/gpu/drm/xe/xe_irq.c | 13 ++- drivers/gpu/drm/xe/xe_irq.h | 1 + drivers/gpu/drm/xe/xe_lrc.c | 149 ++++++++++++++++------------
drivers/gpu/drm/xe/xe_lrc.h | 2 + drivers/gpu/drm/xe/xe_migrate.c | 65 ++++++++++--- drivers/gpu/drm/xe/xe_migrate.h | 1 + drivers/gpu/drm/xe/xe_pci.c | 1 + drivers/gpu/drm/xe/xe_pci.h | 2 + drivers/gpu/drm/xe/xe_pci_err.c | 150
+++++++++++++++++++++++++++++
drivers/gpu/drm/xe/xe_uc.c | 36 +++++++ drivers/gpu/drm/xe/xe_uc.h | 2 + drivers/gpu/drm/xe/xe_uc_fw.c | 39 ++++++-- drivers/gpu/drm/xe/xe_uc_fw.h | 1 +28 files changed, 544 insertions(+), 97 deletions(-)create mode 100644
drivers/gpu/drm/xe/xe_pci_err.c