Skip to contentSkip to conversation

drm/amdgpu: don't disable ttm buffer funcs on reset

PATCH Superseded Pierre-Eric Pelloux-Prayer 6 participants 8 messages Jun 16 – Jun 17

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 drm/amdgpu: don't disable ttm buffer funcs on reset

Access to hw is already suspended so there are no reasons to disable ttm buffer funcs.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 --
 1 file changed, 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -2
index 6608780ffef2..57e10afb4714 100644
@@ -3167,8 +3167,6 @@ static int amdgpu_device_ip_suspend(struct amdgpu_device *adev)
 		amdgpu_virt_request_full_gpu(adev, false);
 	}
 
-	amdgpu_ttm_disable_buffer_funcs(adev);
-
 	r = amdgpu_device_ip_suspend_phase1(adev);
 	if (r)
 		return r;
Signature · 1 line
2.43.0
#1 lore raw

PATCH 2/2 drm/amdgpu: delay ttm buffer func enablement on xgmi

When amdgpu_init_minimal_xgmi is used, SDMA engines init is delayed so amdgpu_ttm_enable_buffer_funcs must be called later.

Without this, the check for num_buffer_funcs_scheds will fail and using ttm buffer funcs later will fail.

Fixes 3a5da695c8d1 ("drm/amdgpu: only use working sdma schedulers for ttm")
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c   | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 57e10afb4714..2e8954fa5c40 100644
@@ -2479,7 +2479,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	amdgpu_ttm_enable_buffer_funcs(adev);
+	if (amdgpu_ip_member_of_hwini(adev, AMD_IP_BLOCK_TYPE_SDMA))
+		amdgpu_ttm_enable_buffer_funcs(adev);
 
 	/* Don't init kfd if whole hive need to be reset during init */
 	if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -0
index faac80a8a6eb..c0dca209335e 100644
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
 		if (r && r != -EHWPOISON)
 			dev_err(tmp_adev->dev,
 				"error during bad page data initialization");
+
+		amdgpu_ttm_enable_buffer_funcs(tmp_adev);
 	}
 }
 
Signature · 1 line
2.43.0
maintainer Acked-by on v1 2/2
#2 lore raw

Re: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

Quoted 9 lines · Pierre-Eric Pelloux-Prayer

When amdgpu_init_minimal_xgmi is used, SDMA engines init is delayed so amdgpu_ttm_enable_buffer_funcs must be called later.

Without this, the check for num_buffer_funcs_scheds will fail and using ttm buffer funcs later will fail.

Fixes 3a5da695c8d1 ("drm/amdgpu: only use working sdma schedulers for ttm")
Quoted 32 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c   | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 57e10afb4714..2e8954fa5c40 100644
@@ -2479,7 +2479,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
 	if (r)
 		goto init_failed;
 
-	amdgpu_ttm_enable_buffer_funcs(adev);
+	if (amdgpu_ip_member_of_hwini(adev, AMD_IP_BLOCK_TYPE_SDMA))
+		amdgpu_ttm_enable_buffer_funcs(adev);
 
 	/* Don't init kfd if whole hive need to be reset during init */
 	if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -0
index faac80a8a6eb..c0dca209335e 100644
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
 		if (r && r != -EHWPOISON)
 			dev_err(tmp_adev->dev,
 				"error during bad page data initialization");
+
+		amdgpu_ttm_enable_buffer_funcs(tmp_adev);
 	}
 }
 
#3 lore raw

Re: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

In reply to Pierre-Eric Pelloux-Prayer
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Asynchronous enablement of TTM buffer functions creates a data race with userspace, potentially causing divide-by-zero or NULL pointer dereferences.

Pre-existing issues:
- [Critical] Lockless iteration over hive->device_list in the asynchronous reset worker exposes the list to concurrent modifications and use-after-free.
- [High] Ignoring the return value of amdgpu_reset_do_xgmi_reset_on_init and unconditionally enabling buffer functions can cause hardware hangs. --

commit 1c14f2e51df95b3cf38be6f03564ce97cfb88996
Author: Pierre-Eric Pelloux-Prayer <[email protected]> drm/amdgpu: delay ttm buffer func enablement on xgmi

This commit delays SDMA engine initialization when amdgpu_init_minimal_xgmi is used. It moves the amdgpu_ttm_enable_buffer_funcs() call to the amdgpu_xgmi_reset_on_init_work() worker to avoid initialization failures.

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +0 -0
index faac80a8a6eb2..c0dca209335e3 100644

[ ... ]

drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work) · line 1667

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -0
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
 		if (r && r != -EHWPOISON)
 			dev_err(tmp_adev->dev,
 				"error during bad page data initialization");
+
+		amdgpu_ttm_enable_buffer_funcs(tmp_adev);

[Severity: Critical]
This isn't a bug introduced by this patch, but inserting this function call here highlights a lockless iteration issue.

Earlier in amdgpu_xgmi_reset_on_init_work(), we drop the hive_lock:

    mutex_unlock(&hive->hive_lock);
    amdgpu_device_unlock_reset_domain(tmp_adev->reset_domain);

    list_for_each_entry(tmp_adev, &hive->device_list, gmc.xgmi.head) {
        ...
        amdgpu_ttm_enable_buffer_funcs(tmp_adev);
    }

If a device is unbound from the driver concurrently (e.g., via sysfs), could amdgpu_xgmi_remove_device() acquire hive->hive_lock and delete the device from the list?

If so, this loop could traverse a corrupted list or access a freed tmp_adev pointer. Does the list traversal need to be protected?

[Severity: High]
This is also a pre-existing issue, but does this loop execute safely if the hardware reset fails?

Right before this list iteration, the reset executes but the return value is ignored:

    amdgpu_reset_do_xgmi_reset_on_init(&reset_context);
    mutex_unlock(&hive->hive_lock);

If the ASIC fails to respond or times out, we proceed to call amdgpu_ttm_enable_buffer_funcs() on dead hardware. Could this result in immediate job timeouts and scheduler hangs when TTM buffer moves are directed to the SDMA engines?

[Severity: High]
Since amdgpu_xgmi_reset_on_init_work() executes asynchronously, could userspace already have access the device before this worker finishes?

If drm_dev_register() has exposed an earlier device in the hive, a concurrent thread could trigger buffer moves. When amdgpu_ttm_enable_buffer_funcs() runs, it sets the enabled flag without memory barriers:

    adev->mman.num_move_entities = num_move_entities;
    ...
    adev->mman.buffer_funcs_enabled = true;

Could a concurrent operation see adev->mman.buffer_funcs_enabled == true but read a stale value of 0 for adev->mman.num_move_entities?

This looks like it might cause a divide-by-zero kernel panic in amdgpu_move_blit() when it calculates the entity:

    e = atomic_inc_return(&adev->mman.next_move_entity) %
        adev->mman.num_move_entities;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Reviewed-by on v1 2/2
#4 lore raw

Re: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

In reply to Pierre-Eric Pelloux-Prayer
Quoted 10 lines · Pierre-Eric Pelloux-Prayer

When amdgpu_init_minimal_xgmi is used, SDMA engines init is delayed so amdgpu_ttm_enable_buffer_funcs must be called later.

Without this, the check for num_buffer_funcs_scheds will fail and using ttm buffer funcs later will fail.

Fixes 3a5da695c8d1 ("drm/amdgpu: only use working sdma schedulers for ttm")
Quoted 35 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c   | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 57e10afb4714..2e8954fa5c40 100644
@@ -2479,7 +2479,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
        if (r)
                goto init_failed;
 
-       amdgpu_ttm_enable_buffer_funcs(adev);
+       if (amdgpu_ip_member_of_hwini(adev, AMD_IP_BLOCK_TYPE_SDMA))
+               amdgpu_ttm_enable_buffer_funcs(adev);
 
        /* Don't init kfd if whole hive need to be reset during init */
        if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) {
drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -1
index faac80a8a6eb..c0dca209335e 100644
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
                if (r && r != -EHWPOISON)
                        dev_err(tmp_adev->dev,
                                "error during bad page data initialization");
+
+               amdgpu_ttm_enable_buffer_funcs(tmp_adev);
        }
 }
 
--

2.43.0

#5 lore raw

RE: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

In reply to Pierre-Eric Pelloux-Prayer

Public

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Pierre-Eric Pelloux-Prayer
Sent: Tuesday, June 16, 2026 6:21 PM
To: Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>; David Airlie <[email protected]>; Simona Vetter <[email protected]>; Pelloux-Prayer, Pierre-Eric <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

When amdgpu_init_minimal_xgmi is used, SDMA engines init is delayed so amdgpu_ttm_enable_buffer_funcs must be called later.

Without this, the check for num_buffer_funcs_scheds will fail and using ttm buffer funcs later will fail.

Fixes 3a5da695c8d1 ("drm/amdgpu: only use working sdma schedulers for ttm")
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c   | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 57e10afb4714..2e8954fa5c40 100644
@@ -2479,7 +2479,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
        if (r)
                goto init_failed;
 
-       amdgpu_ttm_enable_buffer_funcs(adev);
+       if (amdgpu_ip_member_of_hwini(adev, AMD_IP_BLOCK_TYPE_SDMA))
+               amdgpu_ttm_enable_buffer_funcs(adev);
 
        /* Don't init kfd if whole hive need to be reset during init */
        if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c

index faac80a8a6eb..c0dca209335e 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -0
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
                if (r && r != -EHWPOISON)
                        dev_err(tmp_adev->dev,
                                "error during bad page data initialization");
+
+               amdgpu_ttm_enable_buffer_funcs(tmp_adev);
 

<lijo> This gets enabled in the resume path after xgmi reset. It is not needed here.

Thanks,
Lijo

        }

}

--
2.43.0

#6 lore raw

Re: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

Le 16/06/2026 à 16:41, Lazar, Lijo a écrit :

Quoted 44 lines

Public

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Pierre-Eric Pelloux-Prayer
Sent: Tuesday, June 16, 2026 6:21 PM
To: Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>; David Airlie <[email protected]>; Simona Vetter <[email protected]>; Pelloux-Prayer, Pierre-Eric <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: [PATCH v1 2/2] drm/amdgpu: delay ttm buffer func enablement on xgmi

When amdgpu_init_minimal_xgmi is used, SDMA engines init is delayed so amdgpu_ttm_enable_buffer_funcs must be called later.

Without this, the check for num_buffer_funcs_scheds will fail and using ttm buffer funcs later will fail.

Fixes 3a5da695c8d1 ("drm/amdgpu: only use working sdma schedulers for ttm")
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 ++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c   | 2 ++
  2 files changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -1
index 57e10afb4714..2e8954fa5c40 100644
@@ -2479,7 +2479,8 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)
         if (r)
                 goto init_failed;
 
-       amdgpu_ttm_enable_buffer_funcs(adev);
+       if (amdgpu_ip_member_of_hwini(adev, AMD_IP_BLOCK_TYPE_SDMA))
+               amdgpu_ttm_enable_buffer_funcs(adev);
 
         /* Don't init kfd if whole hive need to be reset during init */
         if (adev->init_lvl->level != AMDGPU_INIT_LEVEL_MINIMAL_XGMI) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c

index faac80a8a6eb..c0dca209335e 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c +2 -0
@@ -1663,6 +1663,8 @@ static void amdgpu_xgmi_reset_on_init_work(struct work_struct *work)
                 if (r && r != -EHWPOISON)
                         dev_err(tmp_adev->dev,
                                 "error during bad page data initialization");
+
+               amdgpu_ttm_enable_buffer_funcs(tmp_adev);
 
Quoted 1 line

<lijo> This gets enabled in the resume path after xgmi reset. It is not needed here.

Good point. I've modified the patch in v2.

Thanks,
Pierre-Eric

Quoted 9 lines

Thanks,
Lijo

         }

}

--
2.43.0

maintainer Reviewed-by on v1 1/2
#7 lore raw
In reply to Pierre-Eric Pelloux-Prayer
Quoted 4 lines · Pierre-Eric Pelloux-Prayer

Access to hw is already suspended so there are no reasons to disable ttm buffer funcs.

Quoted 17 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 --
 1 file changed, 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -2
index 6608780ffef2..57e10afb4714 100644
@@ -3167,8 +3167,6 @@ static int amdgpu_device_ip_suspend(struct amdgpu_device *adev)
 		amdgpu_virt_request_full_gpu(adev, false);
 	}
 
-	amdgpu_ttm_disable_buffer_funcs(adev);
-
 	r = amdgpu_device_ip_suspend_phase1(adev);
 	if (r)
 		return r;