drm/amdkfd: Fix SVM page migration hang on MES-based GPUs
Subsystems (3)
cover drm/amdkfd: Fix SVM page migration hang on MES-based GPUs
This patch addresses a GPU queue hang issue during SVM (Shared Virtual Memory) page migration on MES-based GPUs, specifically observed on gfx1151 (Ryzen AI MAX).
Problem:
--------
KFDSVMRangeTest.MultiThreadMigrationTest/1 fails with a queue consumption timeout when XNACK mode 1 is enabled. The GPU compute queue hangs with packets submitted but never consumed.
Root Cause:
-----------
MES (Micro Engine Scheduler) does not perform heavy-weight TLB invalidation after unmapping queues, unlike HWS which does this automatically. This causes in-flight SDMA DMA descriptors to access memory that has been unmapped, leading to page faults and queue hangs.
Fix:
----
Add kfd_flush_tlb() with TLB_FLUSH_HEAVYWEIGHT after MES removes/suspends queues in evict_process_queues_cpsch() and suspend_queues().
Testing:
--------
Platform: gfx1151, ROCm 7.15.0, kernel 7.0.0-28-generic
Results:
- Baseline (no fix): 100% failure (50 runs)
- With TLB flush: 7-10% failure (100 runs)
The TLB flush significantly reduces failures from 100% to ~10%. The residual failures require further investigation and may be addressed in a follow-up patch.
Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension
.../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.0
PATCH 1/1 drm/amdkfd: Add TLB flush after MES queue eviction/suspension
MES (Micro Engine Scheduler) does not perform heavy-weight TLB invalidation after unmapping queues, unlike HWS which does this automatically. This causes a race condition where in-flight SDMA DMA descriptors can access memory that has been unmapped, leading to page faults and GPU queue hangs during SVM page migration.
The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest/1 failures on gfx1151 (Ryzen AI MAX) with XNACK mode 1 enabled - the GPU compute queue hangs with packets submitted but never consumed.
Add kfd_flush_tlb() with TLB_FLUSH_HEAVYWEIGHT in two MES code paths:
1. evict_process_queues_cpsch() - after MES removes queues
2. suspend_queues() - after MES suspends queues and mem_fence completes
This ensures all in-flight memory accesses from unmapped queues are flushed before memory is freed or migrated.
Testing on gfx1151 shows this reduces failure rate from 100% to approximately 7-10%. The residual failures require further investigation.
--- .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +12 -1
@@ -1450,6 +1450,14 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,dqm_evict_mqd_bo (dqm, q); } +/* +* Heavy-weight TLB flush after MES removes queues to ensure +* in-flight SDMA accesses complete before memory is freed/migrated. +* HWS does this automatically, MES does not. +*/ +if (dqm->dev->kfd->shared_resources.enable_mes) +kfd_flush_tlb (pdd,TLB_FLUSH_HEAVYWEIGHT ); +if (!dqm->dev->kfd->shared_resources.enable_mes) { pdd->last_evict_timestamp =get_jiffies_64 (); retval = execute_queues_cpsch(dqm,
@@ -3736,8 +3744,11 @@ int suspend_queues(struct kfd_process *p,if (!per_device_suspended) {dqm_unlock (dqm);mutex_unlock (&p->event_mutex); -if (total_suspended) +if (total_suspended) {amdgpu_amdkfd_debug_mem_fence (dqm->dev->adev); +/* Heavy-weight TLB flush after MES suspends queues */ +kfd_flush_tlb (pdd,TLB_FLUSH_HEAVYWEIGHT ); + }continue ; }
Signature · 1 line
2.43.0
Re: [PATCH 1/1] drm/amdkfd: Add TLB flush after MES queue eviction/suspension
Quoted 30 lines · Priya Hosur
MES (Micro Engine Scheduler) does not perform heavy-weight TLB invalidation after unmapping queues, unlike HWS which does this automatically. This causes a race condition where in-flight SDMA DMA descriptors can access memory that has been unmapped, leading to page faults and GPU queue hangs during SVM page migration.
The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest/1 failures on gfx1151 (Ryzen AI MAX) with XNACK mode 1 enabled - the GPU compute queue hangs with packets submitted but never consumed.
Add
kfd_flush_tlb()withTLB_FLUSH_HEAVYWEIGHTin two MES code paths:
1.evict_process_queues_cpsch()- after MES removes queues
2.suspend_queues()- after MES suspends queues andmem_fencecompletesThis ensures all in-flight memory accesses from unmapped queues are flushed before memory is freed or migrated.
Testing on gfx1151 shows this reduces failure rate from 100% to approximately 7-10%. The residual failures require further investigation.
Signed-off-by Priya Hosur <[email protected]>--- .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +0
-0
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c static int evict_process_queues_cpsch(struct device_queue_manager *dqm, · line 1456
Quoted 7 lines
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +4
-0@@ -1450,6 +1450,14 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,dqm_evict_mqd_bo (dqm, q); } +/* +* Heavy-weight TLB flush after MES removes queues to ensure +* in-flight SDMA accesses complete before memory is freed/migrated. +* HWS does this automatically, MES does not.
I'm not sure why you call out SDMA specifically here. This affects in-flight memory accesses from compute jobs as well. Just remove "SDMA" from the comment. With that fixed, the patch is
Quoted 20 lines
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +8
-1+*/ +if (dqm->dev->kfd->shared_resources.enable_mes) +kfd_flush_tlb (pdd,TLB_FLUSH_HEAVYWEIGHT ); +if (!dqm->dev->kfd->shared_resources.enable_mes) { pdd->last_evict_timestamp =get_jiffies_64 (); retval = execute_queues_cpsch(dqm,@@ -3736,8 +3744,11 @@ int suspend_queues(struct kfd_process *p,if (!per_device_suspended) {dqm_unlock (dqm);mutex_unlock (&p->event_mutex); -if (total_suspended) +if (total_suspended) {amdgpu_amdkfd_debug_mem_fence (dqm->dev->adev); +/* Heavy-weight TLB flush after MES suspends queues */ +kfd_flush_tlb (pdd,TLB_FLUSH_HEAVYWEIGHT ); + }continue ; }