Skip to contentSkip to conversation

drm/amdkfd: Add TLB flush after MES queue eviction/suspension

PATCH In review Priya Hosur 4 participants 5 messages Aug 16 – Aug 24

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover drm/amdkfd: Add TLB flush after MES queue eviction/suspension

This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.

v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap in amdgpu_userq_unmap_all(), not per-queue unmap

v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush in amdgpu_userq_unmap_helper() (amdgpu_userq.c)

v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tag

Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension

 .../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/1 drm/amdkfd: Add TLB flush after MES queue eviction/suspension

MES (Micro Engine Scheduler) does not perform heavy-weight TLB invalidation after unmapping queues, unlike HWS which does this automatically. This causes a race condition where in-flight DMA descriptors can access memory that has been unmapped, leading to page faults and GPU queue hangs during SVM page migration.

The issue manifests as KFDSVMRangeTest.MultiThreadMigrationTest failures on gfx1151 (Strix Point) with XNACK mode 1 enabled - the GPU compute queue hangs with packets submitted but never consumed.

Add kfd_flush_tlb() calls after MES queue removal in two locations:
- evict_process_queues_cpsch(): after all queues removed during eviction
- suspend_queues(): after debug/criu queue suspension (with mem_fence barrier)

This ensures all in-flight memory accesses from unmapped queues are flushed before memory is freed or migrated.

Change-Id Iaa884d4a7ad1199446bc45f4ad8a9a179ab386e6
---
 .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c   | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +12 -1
index a23384571193..6002c8a65fbe 100644
@@ -1450,6 +1450,14 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
 		dqm_evict_mqd_bo(dqm, q);
 	}
 
+	/*
+	 * Heavy-weight TLB flush after MES removes queues to ensure
+	 * in-flight memory accesses complete before memory is freed/migrated.
+	 * HWS does this automatically, MES does not.
+	 */
+	if (dqm->dev->kfd->shared_resources.enable_mes)
+		kfd_flush_tlb(pdd);
+
 	if (!dqm->dev->kfd->shared_resources.enable_mes) {
 		pdd->last_evict_timestamp = get_jiffies_64();
 		retval = execute_queues_cpsch(dqm,
@@ -3736,8 +3744,11 @@ int suspend_queues(struct kfd_process *p,
 		if (!per_device_suspended) {
 			dqm_unlock(dqm);
 			mutex_unlock(&p->event_mutex);
-			if (total_suspended)
+			if (total_suspended) {
 				amdgpu_amdkfd_debug_mem_fence(dqm->dev->adev);
+				/* Heavy-weight TLB flush after MES suspends queues */
+				kfd_flush_tlb(pdd);
+			}
 			continue;
 		}
 
Signature · 1 line
2.43.0
#2 lore raw
In reply to Priya Hosur
Quoted 7 lines · Priya Hosur

This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.

v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap in amdgpu_userq_unmap_all(), not per-queue unmap

Sorry, that was not the intention. I was thinking there is a unmap_all API in MES in which case this could be done within FW itself instead.

I believe Alex was also asking to check about batch unmap cases of user queue. Ex: - amdgpu_eviction_fence_suspend_worker there could still be inflight transactions similar to KFD case.

Thanks,
Lijo

Quoted 15 lines

v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush in amdgpu_userq_unmap_helper() (amdgpu_userq.c)

v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tag

Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension

  .../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
#3 lore raw
Quoted 7 lines · Lijo
Quoted 3 lines · Priya Hosur

This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.

Quoted 8 lines
Quoted 4 lines

v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap in amdgpu_userq_unmap_all(), not per-queue unmap

Sorry, that was not the intention. I was thinking there is a unmap_all API in MES in which case this could be done within FW itself instead.

There is a suspend_all and I believe the reset detect and detect_and_reset calls end up suspending all queues the type specified (gfx, compute, sdma).

Quoted 3 lines

I believe Alex was also asking to check about batch unmap cases of user queue. Ex: - amdgpu_eviction_fence_suspend_worker there could still be inflight transactions similar to KFD case.

RIght. When we evict a KGD userq process, we evict all of the queues for that GPU VM.

Alex

Quoted 20 lines

Thanks,
Lijo

Quoted 15 lines

v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush in amdgpu_userq_unmap_helper() (amdgpu_userq.c)

v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tag

Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension

  .../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
maintainer
#4 lore raw
Quoted 21 lines · Alex Deucher
Quoted 7 lines · Lijo
Quoted 3 lines · Priya Hosur

This patch adds heavy-weight TLB flush after MES queue eviction/suspension to fix SVM page migration hangs on gfx1151 with XNACK mode enabled.

Quoted 8 lines
Quoted 4 lines

v4:
- Dropped userq TLB flush patch per Felix/Lijo feedback: TLB flush is only needed after batch unmap in amdgpu_userq_unmap_all(), not per-queue unmap

Sorry, that was not the intention. I was thinking there is a unmap_all API in MES in which case this could be done within FW itself instead.

There is a suspend_all and I believe the reset detect and detect_and_reset calls end up suspending all queues the type specified (gfx, compute, sdma).

Quoted 6 lines
Quoted 3 lines

I believe Alex was also asking to check about batch unmap cases of user queue. Ex: - amdgpu_eviction_fence_suspend_worker there could still be inflight transactions similar to KFD case.

RIght. When we evict a KGD userq process, we evict all of the queues for that GPU VM.

Yeah I completely agree with Lijo.

An unmap operation means that the MES *must* make sure that all work from the unmapped queue is finished. Usually by issuing something similar to an EOP packet/event.

A TLB invalidation is necessary when the debugger wants to attach and we only suspended the queue, e.g. in other words prevent work from continuing but doesn't take it of the HW or wait for it to finish.

So something here is clearly badly going wrong.

Regards,
Christian.

Quoted 23 lines

Alex

Quoted 20 lines

Thanks,
Lijo

Quoted 15 lines

v3:
- Split into 2 patches per Alex's review
- Added userq TLB flush in amdgpu_userq_unmap_helper() (amdgpu_userq.c)

v2:
- Removed "SDMA" from comment as this affects all in-flight memory accesses, not just SDMA (Felix Kuehling)
- Added Reviewed-by tag

Priya Hosur (1):
drm/amdkfd: Add TLB flush after MES queue eviction/suspension

  .../drm/amd/amdkfd/kfd_device_queue_manager.c | 10 ++++++++++
  1 file changed, 10 insertions(+)