drm/amdkfd: Add TLB flush after MES queue eviction/suspension
Subsystems (3)
v4 cover drm/amdkfd: Add TLB flush after MES queue eviction/suspension
This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.
v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap in amdgpu_userq_unmap_all(), not per-queue unmap
v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush in amdgpu_userq_unmap_helper() (amdgpu_userq.c)
v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tag
Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension
.../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
Signature · 1 line
2.43.0
PATCH v4 1/1 drm/amdkfd: Add TLB flush after MES queue eviction/suspension
MES (Micro Engine Scheduler) does not perform heavy-weight TLB invalidation after unmapping queues, unlike HWS which does this automatically. This causes a race condition where in-flight DMA descriptors can access memory that has been unmapped, leading to page faults and GPU queue hangs during SVM page migration.
The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest failures on gfx1151 (Strix Point) with XNACK mode 1 enabled - the GPU compute queue hangs with packets submitted but never consumed.
Add kfd_flush_tlb() calls after MES queue removal in two locations:
- evict_process_queues_cpsch(): after all queues removed during eviction
- suspend_queues(): after debug/criu queue suspension (with mem_fence barrier)
This ensures all in-flight memory accesses from unmapped queues are flushed before memory is freed or migrated.
--- .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +12 -1
@@ -1450,6 +1450,14 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,dqm_evict_mqd_bo (dqm, q); } +/* +* Heavy-weight TLB flush after MES removes queues to ensure +* in-flight memory accesses complete before memory is freed/migrated. +* HWS does this automatically, MES does not. +*/ +if (dqm->dev->kfd->shared_resources.enable_mes) +kfd_flush_tlb (pdd); +if (!dqm->dev->kfd->shared_resources.enable_mes) { pdd->last_evict_timestamp =get_jiffies_64 (); retval = execute_queues_cpsch(dqm,
@@ -3736,8 +3744,11 @@ int suspend_queues(struct kfd_process *p,if (!per_device_suspended) {dqm_unlock (dqm);mutex_unlock (&p->event_mutex); -if (total_suspended) +if (total_suspended) {amdgpu_amdkfd_debug_mem_fence (dqm->dev->adev); +/* Heavy-weight TLB flush after MES suspends queues */ +kfd_flush_tlb (pdd); + }continue ; }
Signature · 1 line
2.43.0
Quoted 7 lines · Priya Hosur
This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.
v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap inamdgpu_userq_unmap_all(), not per-queue unmap
Sorry, that was not the intention. I was thinking there is a unmap_all API in MES in which case this could be done within FW itself instead.
I believe Alex was also asking to check about batch unmap cases of user queue. Ex: - amdgpu_eviction_fence_suspend_worker there could still be inflight transactions similar to KFD case.
Thanks,
Lijo
Quoted 15 lines
v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush inamdgpu_userq_unmap_helper()(amdgpu_userq.c)v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tagPriya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension.../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
Quoted 7 lines · Lijo
Quoted 3 lines · Priya Hosur
This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.
Quoted 8 lines
Quoted 4 lines
v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap inamdgpu_userq_unmap_all(), not per-queue unmapSorry, that was not the intention. I was thinking there is a
unmap_allAPI in MES in which case this could be done within FW itself instead.
There is a suspend_all and I believe the reset detect and detect_and_reset calls end up suspending all queues the type specified (gfx, compute, sdma).
Quoted 3 lines
I believe Alex was also asking to check about batch unmap cases of user queue. Ex: -
amdgpu_eviction_fence_suspend_workerthere could still be inflight transactions similar to KFD case.
RIght. When we evict a KGD userq process, we evict all of the queues for that GPU VM.
Alex
Quoted 20 lines
Thanks,
LijoQuoted 15 lines
v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush inamdgpu_userq_unmap_helper()(amdgpu_userq.c)v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tagPriya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension.../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
Christian König
maintainerQuoted 21 lines · Alex Deucher
Quoted 7 lines · Lijo
Quoted 3 lines · Priya Hosur
This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.
Quoted 8 lines
Quoted 4 lines
v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap inamdgpu_userq_unmap_all(), not per-queue unmapSorry, that was not the intention. I was thinking there is a
unmap_allAPI in MES in which case this could be done within FW itself instead.There is a
suspend_alland I believe the reset detect anddetect_and_resetcalls end up suspending all queues the type specified (gfx, compute, sdma).
Quoted 6 lines
Quoted 3 lines
I believe Alex was also asking to check about batch unmap cases of user queue. Ex: -
amdgpu_eviction_fence_suspend_workerthere could still be inflight transactions similar to KFD case.RIght. When we evict a KGD userq process, we evict all of the queues for that GPU VM.
Yeah I completely agree with Lijo.
An unmap operation means that the MES *must* make sure that all work from the unmapped queue is finished. Usually by issuing something similar to an EOP packet/event.
A TLB invalidation is necessary when the debugger wants to attach and we only suspended the queue, e.g. in other words prevent work from continuing but doesn't take it of the HW or wait for it to finish.
So something here is clearly badly going wrong.
Regards,
Christian.
Quoted 23 lines
Alex
Quoted 20 lines
Thanks,
LijoQuoted 15 lines
v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush inamdgpu_userq_unmap_helper()(amdgpu_userq.c)v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tagPriya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension.../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++ 1 file changed, 10 insertions(+)