Skip to contentSkip to conversation

drm/amdgpu: drain SDMA before process eviction during GPU reset

PATCH In review Sunday Clement 2 participants 2 messages Jul 21 – Jul 22

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: drain SDMA before process eviction during GPU reset

Host kernel panics (Oops in free_swap_cache) when GPU reset is triggered under multi-process load. The crash is caused by SDMA engines writing to memory while munmap clears page table entries during process eviction.

The race condition:
amdgpu_device_halt_activities() loops over all devices in the reset list and calls amdgpu_amdkfd_pre_reset() on each device. The first call to amdgpu_amdkfd_pre_reset() triggers global process eviction across all GPUs via kgd2kfd_suspend_process(), which starts munmap clearing PTEs. However, SDMA engines on other devices in the reset list still have in-flight DMA transactions that have not been drained.

Fix by draining ALL SDMA rings on ALL devices in the reset list BEFORE calling amdgpu_amdkfd_pre_reset() on any device. Also flush the HDP write cache to ensure SDMA writes reach system memory before PTEs are cleared. This guarantees no in-flight DMAs exist when process eviction begins.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 +++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +30 -1
index 472e96ae884e..0cd3dc4cffb2 100644
@@ -5340,7 +5340,36 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev,
 					  bool need_emergency_restart)
 {
 	struct amdgpu_device *tmp_adev = NULL;
-	int i;
+	int i, r;
+	struct amdgpu_ring *ring;
+
+    /*
+     * Drain all SDMA on all devices before any KFD pre_reset.
+     *
+     * amdgpu_amdkfd_pre_reset() triggers global process eviction across
+     * all GPUs, which starts munmap clearing PTEs. We must ensure no
+     * in-flight SDMA DMAs on any device when eviction begins, otherwise
+     * SDMA writes race with munmap, corrupting PTEs.
+     */
+	list_for_each_entry(tmp_adev, device_list, reset_list) {
+		for (i = 0; i < tmp_adev->sdma.num_instances; i++) {
+			ring = &tmp_adev->sdma.instance[i].ring;
+			if (ring->sched.timeout == MAX_SCHEDULE_TIMEOUT)
+				continue;
+
+			r = amdgpu_fence_wait_empty(ring);
+			if (r)
+				dev_warn(tmp_adev->dev,
+					"SDMA%d ring drain timeout before reset, proceeding\n", i);
+		}
+
+		/*
+		 * Flush HDP write cache to ensure SDMA writes reach system
+		 * memory before munmap. Prevents cached writes from arriving
+		 * after pages are freed.
+		 */
+		amdgpu_device_flush_hdp(tmp_adev, NULL);
+	}
 
 	/* block all schedulers and reset given job's ring */
 	list_for_each_entry(tmp_adev, device_list, reset_list) {
Signature · 1 line
2.43.0
#1 lore raw
Quoted 13 lines · Sunday Clement

Host kernel panics (Oops in free_swap_cache) when GPU reset is triggered under multi-process load. The crash is caused by SDMA engines writing to memory while munmap clears page table entries during process eviction.

The race condition:
amdgpu_device_halt_activities() loops over all devices in the reset list and calls amdgpu_amdkfd_pre_reset() on each device. The first call to amdgpu_amdkfd_pre_reset() triggers global process eviction across all GPUs via kgd2kfd_suspend_process(), which starts munmap clearing PTEs. However, SDMA engines on other devices in the reset list still have in-flight DMA transactions that have not been drained.

Quoted 6 lines

Fix by draining ALL SDMA rings on ALL devices in the reset list BEFORE calling amdgpu_amdkfd_pre_reset() on any device. Also flush the HDP write cache to ensure SDMA writes reach system memory before PTEs are cleared. This guarantees no in-flight DMAs exist when process eviction begins.

The processes are signaled during pre_reset. Without suspending/notifying any process, how do you expect rings to drain?

Thanks,
Lijo

Quoted 47 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 31 +++++++++++++++++++++-
  1 file changed, 30 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +30 -1
index 472e96ae884e..0cd3dc4cffb2 100644
@@ -5340,7 +5340,36 @@ static void amdgpu_device_halt_activities(struct amdgpu_device *adev,
  					  bool need_emergency_restart)
  {
  	struct amdgpu_device *tmp_adev = NULL;
-	int i;
+	int i, r;
+	struct amdgpu_ring *ring;
+
+    /*
+     * Drain all SDMA on all devices before any KFD pre_reset.
+     *
+     * amdgpu_amdkfd_pre_reset() triggers global process eviction across
+     * all GPUs, which starts munmap clearing PTEs. We must ensure no
+     * in-flight SDMA DMAs on any device when eviction begins, otherwise
+     * SDMA writes race with munmap, corrupting PTEs.
+     */
+	list_for_each_entry(tmp_adev, device_list, reset_list) {
+		for (i = 0; i < tmp_adev->sdma.num_instances; i++) {
+			ring = &tmp_adev->sdma.instance[i].ring;
+			if (ring->sched.timeout == MAX_SCHEDULE_TIMEOUT)
+				continue;
+
+			r = amdgpu_fence_wait_empty(ring);
+			if (r)
+				dev_warn(tmp_adev->dev,
+					"SDMA%d ring drain timeout before reset, proceeding\n", i);
+		}
+
+		/*
+		 * Flush HDP write cache to ensure SDMA writes reach system
+		 * memory before munmap. Prevents cached writes from arriving
+		 * after pages are freed.
+		 */
+		amdgpu_device_flush_hdp(tmp_adev, NULL);
+	}
 
  	/* block all schedulers and reset given job's ring */
  	list_for_each_entry(tmp_adev, device_list, reset_list) {