drm/amdgpu: Fix TLB flush failures after hibernation resume
Subsystems (2)
cover drm/amdgpu: Fix TLB flush failures after hibernation resume
From: Ionut Nechita <[email protected]>
Hi,
This patch addresses critical TLB flush failures that occur during hibernation resume on AMD GPUs, particularly affecting ROCm workloads.
Problem:
--------
After resuming from hibernation (S4), the amdgpu driver consistently fails TLB invalidation operations with these errors:
amdgpu: TLB flush failed for PASID xxxxx
amdgpu: failed to write reg 28b4 wait reg 28c6
amdgpu: failed to write reg 1a6f4 wait reg 1a706
These failures cause compute workloads to malfunction or crash, making hibernation unreliable for systems running ROCm/OpenCL applications.
Root Cause:
-----------
During resume, the KIQ (Kernel Interface Queue) ring is marked as ready (ring.sched.ready = true) before the GPU hardware has fully initialized. When TLB invalidation attempts to use KIQ for register access during this window, the commands fail because the GPU is not yet stable.
Solution:
---------
This patch introduces a resume_gpu_stable flag that:
- Starts as false during resume
- Forces TLB invalidation to use the reliable MMIO path initially
- Gets set to true after ring tests pass in gfx_v9_0_cp_resume()
- Allows switching to the faster KIQ path once GPU is confirmed stable
This ensures TLB flushes work correctly during early resume while still benefiting from KIQ-based invalidation after the GPU is fully operational.
Testing:
--------
Tested on AMD Cezanne (Renoir) with ROCm workloads across multiple hibernation cycles. The patch eliminates all TLB flush failures and restores reliable hibernation support for compute workloads.
Impact:
-------
Affects all AMD GPUs using KIQ for TLB invalidation, particularly visible on systems with active compute workloads (ROCm, OpenCL).
Ionut Nechita (1):
drm/amdgpu: Fix TLB flush failures after hibernation resume
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 +++++++-- drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 6 +++++- 5 files changed, 29 insertions(+), 3 deletions(-)
Signature · 1 line
2.52.0
PATCH 1/1 drm/amdgpu: Fix TLB flush failures after hibernation resume
From: Ionut Nechita <[email protected]>
After resume from hibernation, the amdgpu driver experiences TLB flush failures with errors:
amdgpu: TLB flush failed for PASID xxxxx
amdgpu: failed to write reg 28b4 wait reg 28c6
amdgpu: failed to write reg 1a6f4 wait reg 1a706
Root Cause:
-----------
The KIQ (Kernel Interface Queue) ring is marked as ready (ring.sched.ready = true) during resume, but the hardware is not fully functional yet. When TLB invalidation attempts to use KIQ for register access, the commands fail because the GPU hasn't completed initialization.
Solution:
---------
1. Add resume_gpu_stable flag (initially false on resume)
2. Force TLB invalidation to use direct MMIO path instead of KIQ when resume_gpu_stable is false
3. After ring tests pass in gfx_v9_0_cp_resume(), set resume_gpu_stable to true
4. From that point forward, use faster KIQ path for TLB invalidation
This ensures TLB flushes work correctly during early resume while still benefiting from KIQ-based invalidation after the GPU is stable.
Changes:
--------
- amdgpu.h: Add resume_gpu_stable flag to amdgpu_device
- amdgpu_device.c: Initialize resume_gpu_stable to false on resume
- amdgpu_gmc.c: Check resume_gpu_stable in flush_gpu_tlb_pasid
- gfx_v9_0.c: Set resume_gpu_stable after ring tests pass
- gmc_v9_0.c: Check resume_gpu_stable before using KIQ path
Tested on AMD Cezanne (Renoir) with ROCm workloads after hibernation.
Result: Eliminates TLB flush failures on resume.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 +++++++-- drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 6 +++++- 5 files changed, 29 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -1225,6 +1225,7 @@ struct amdgpu_device {bool in_s4;bool in_s0ix;suspend_state_t last_suspend_state; +bool resume_gpu_stable;enum pp_mp1_state mp1_state;struct amdgpu_doorbell_index doorbell_index;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6 -0
@@ -5457,6 +5457,12 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) goto exit; } +/* +* Set resume_gpu_stable to false BEFORE KFD resume to ensure +* extended timeouts are used for TLB flushes during hibernation recovery +*/ + adev->resume_gpu_stable = false; + r =amdgpu_amdkfd_resume (adev, !amdgpu_sriov_vf (adev) && !adev->in_runpm);if (r) goto exit;
drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +7 -2
@@ -731,7 +731,12 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,if (!down_read_trylock (&adev->reset_domain->sem))return 0 ; -if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready) { +/* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. +*/ +if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready || + !adev->resume_gpu_stable) {if (adev->gmc.flush_tlb_needs_extra_type_2) adev->gmc.gmc_funcs->flush_gpu_tlb_pasid(adev, pasid, 2, all_hub,
@@ -835,9 +840,9 @@ void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, goto failed_kiq;might_sleep (); +while (r <1 && cnt++ <MAX_KIQ_REG_TRY && !amdgpu_reset_pending (adev->reset_domain)) { -msleep (MAX_KIQ_REG_BAILOUT_INTERVAL ); r =amdgpu_fence_wait_polling (ring, seq,MAX_KIQ_REG_WAIT ); }
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +10 -0
@@ -3985,6 +3985,16 @@ static int gfx_v9_0_cp_resume(struct amdgpu_device *adev)amdgpu_ring_test_helper (ring); } +/* +* After successful ring tests, mark GPU as stable for resume. +* This allows KIQ-based TLB invalidation to be used instead of +* slower direct MMIO path. +*/ +if (!adev->resume_gpu_stable) { + adev->resume_gpu_stable = true; +dev_info (adev->dev,"GPU rings verified, enabling KIQ path\n" ); + } +gfx_v9_0_enable_gui_idle_interrupt (adev, true);return 0 ;
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +5 -1
@@ -855,9 +855,13 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,/* This is necessary for SRIOV as well as for GFXOFF to function * properly under bare metal +* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. */ if (adev->gfx.kiq[inst].ring.sched.ready && - (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev))) { + (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev)) && + adev->resume_gpu_stable) {uint32_t req = hub->vm_inv_eng0_req + hub->eng_distance * eng;uint32_t ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng;
Signature · 1 line
2.52.0
Quoted 67 lines · Ionut Nechita (Sunlight Linux)
From: Ionut Nechita <[email protected]>
After resume from hibernation, the amdgpu driver experiences TLB flush failures with errors:
amdgpu: TLB flush failed for PASID xxxxx
amdgpu: failed to write reg 28b4 wait reg 28c6
amdgpu: failed to write reg 1a6f4 wait reg 1a706Root Cause:
-----------
The KIQ (Kernel Interface Queue) ring is marked as ready (ring.sched.ready = true) during resume, but the hardware is not fully functional yet. When TLB invalidation attempts to use KIQ for register access, the commands fail because the GPU hasn't completed initialization.Solution:
---------
1. Addresume_gpu_stableflag (initially false on resume)
2. Force TLB invalidation to use direct MMIO path instead of KIQ whenresume_gpu_stableis false
3. After ring tests pass ingfx_v9_0_cp_resume(), setresume_gpu_stableto true
4. From that point forward, use faster KIQ path for TLB invalidationThis ensures TLB flushes work correctly during early resume while still benefiting from KIQ-based invalidation after the GPU is stable.
Changes:
--------
-amdgpu.h: Addresume_gpu_stableflag toamdgpu_device
-amdgpu_device.c: Initializeresume_gpu_stableto false on resume
-amdgpu_gmc.c: Checkresume_gpu_stableinflush_gpu_tlb_pasid
-gfx_v9_0.c: Setresume_gpu_stableafter ring tests pass
-gmc_v9_0.c: Checkresume_gpu_stablebefore using KIQ pathTested on AMD Cezanne (Renoir) with ROCm workloads after hibernation.
Result: Eliminates TLB flush failures on resume.Signed-off-by Ionut Nechita <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 +++++++-- drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 6 +++++- 5 files changed, 29 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -1225,6 +1225,7 @@ struct amdgpu_device {bool in_s4;bool in_s0ix;suspend_state_t last_suspend_state; +bool resume_gpu_stable;enum pp_mp1_state mp1_state;struct amdgpu_doorbell_index doorbell_index;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) · line 5464
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +5
-0@@ -5457,6 +5457,12 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) goto exit; } +/* +* Set resume_gpu_stable to false BEFORE KFD resume to ensure +* extended timeouts are used for TLB flushes during hibernation recovery +*/ + adev->resume_gpu_stable = false;
This change disables TLB flushes via KIQ permanently. This is called after gfx_v9_0_cp_resume() so resume_gpu_stable is only set to true between the calls to amdgpu_device_ip_resume() and here.
kiq sched.ready should be handled correctly. kiq sched.ready gets set to false in suspend via:
gfx_v9_0_suspend() -> gfx_v9_0_hw_fini() -> gfx_v9_0_cp_enable()
Then on resume, kiq sched.ready gets set to true again via:
gfx_v9_0_resume() -> gfx_v9_0_hw_init() -> gfx_v9_0_cp_resume() -> gfx_v9_0_kcq_resume() -> amdgpu_gfx_enable_kcq() -> amdgpu_ring_test_helper()
At that point the KIQ hardware is ready. If it weren't, then the above sequence would not have worked. The gfx and compute ring tests are irrelevant.
Alex
Quoted 76 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1
-0+ r =amdgpu_amdkfd_resume (adev, !amdgpu_sriov_vf (adev) && !adev->in_runpm);if (r) goto exit;drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +7
-2@@ -731,7 +731,12 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,if (!down_read_trylock (&adev->reset_domain->sem))return 0 ; -if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready) { +/* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. +*/ +if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready || + !adev->resume_gpu_stable) {if (adev->gmc.flush_tlb_needs_extra_type_2) adev->gmc.gmc_funcs->flush_gpu_tlb_pasid(adev, pasid, 2, all_hub,@@ -835,9 +840,9 @@ void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, goto failed_kiq;might_sleep (); +while (r <1 && cnt++ <MAX_KIQ_REG_TRY && !amdgpu_reset_pending (adev->reset_domain)) { -msleep (MAX_KIQ_REG_BAILOUT_INTERVAL ); r =amdgpu_fence_wait_polling (ring, seq,MAX_KIQ_REG_WAIT ); }drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +10
-0@@ -3985,6 +3985,16 @@ static int gfx_v9_0_cp_resume(struct amdgpu_device *adev)amdgpu_ring_test_helper (ring); } +/* +* After successful ring tests, mark GPU as stable for resume. +* This allows KIQ-based TLB invalidation to be used instead of +* slower direct MMIO path. +*/ +if (!adev->resume_gpu_stable) { + adev->resume_gpu_stable = true; +dev_info (adev->dev,"GPU rings verified, enabling KIQ path\n" ); + } +gfx_v9_0_enable_gui_idle_interrupt (adev, true);return 0 ;drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +5
-2@@ -855,9 +855,13 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,/* This is necessary for SRIOV as well as for GFXOFF to function * properly under bare metal +* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. */ if (adev->gfx.kiq[inst].ring.sched.ready && - (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev))) { + (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev)) && + adev->resume_gpu_stable) {uint32_t req = hub->vm_inv_eng0_req + hub->eng_distance * eng;uint32_t ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng; --2.52.0
Hi Alex,
Thank you for the detailed review and for pointing out the ordering issue.
You're absolutely right - I misunderstood the call sequence. Setting resume_gpu_stable to false in amdgpu_device_resume() happens after gfx_v9_0_cp_resume(), which defeats the purpose and permanently disables the KIQ path.
However, I'm still experiencing the TLB flush failures after hibernation resume on AMD Cezanne (Renoir):
amdgpu: TLB flush failed for PASID xxxxx
amdgpu: failed to write reg 28b4 wait reg 28c6
amdgpu: failed to write reg 1a6f4 wait reg 1a706
If kiq sched.ready is being handled correctly as you described, what else could cause these failures during resume? Are there any known issues with KIQ-based TLB invalidation after hibernation on GFX9?
Should I investigate:
- Timing issues with KIQ command submission during early resume?
- Power/clock gating states affecting KIQ functionality?
- Missing synchronization after KIQ initialization?
Any guidance on the correct direction to investigate would be appreciated.
Thanks,
Ionut
Christian König
maintainerQuoted 4 lines · Ionut Nechita (Sunlight Linux)
From: Ionut Nechita <[email protected]>
After resume from hibernation, the amdgpu driver experiences TLB flush failures with errors:
In general we don't support hibernation with the driver.
Quoted 10 lines
amdgpu: TLB flush failed for PASID xxxxx
amdgpu: failed to write reg 28b4 wait reg 28c6
amdgpu: failed to write reg 1a6f4 wait reg 1a706Root Cause:
-----------
The KIQ (Kernel Interface Queue) ring is marked as ready (ring.sched.ready = true) during resume, but the hardware is not fully functional yet.
Well since that is the root cause we should probably fix that.
Quoted 9 lines
When TLB invalidation attempts to use KIQ
for register access, the commands fail because the GPU hasn't completed initialization.Solution:
---------
1. Addresume_gpu_stableflag (initially false on resume)
2. Force TLB invalidation to use direct MMIO path instead of KIQ whenresume_gpu_stableis false
That is a really bad idea. So absolutely clear NAK to this patch here.
Quoted 6 lines
3. After ring tests pass in
gfx_v9_0_cp_resume(), setresume_gpu_stableto true
4. From that point forward, use faster KIQ path for TLB invalidationThis ensures TLB flushes work correctly during early resume while still benefiting from KIQ-based invalidation after the GPU is stable.
No it doesn't. This patch only works by coincident and not proper engineering.
This only works because the TLB flush is most likely not necesssary.
Question is why the KIQ is not up and running before we do anything with it?
Regards,
Christian.
Quoted 119 lines
Changes:
--------
-amdgpu.h: Addresume_gpu_stableflag toamdgpu_device
-amdgpu_device.c: Initializeresume_gpu_stableto false on resume
-amdgpu_gmc.c: Checkresume_gpu_stableinflush_gpu_tlb_pasid
-gfx_v9_0.c: Setresume_gpu_stableafter ring tests pass
-gmc_v9_0.c: Checkresume_gpu_stablebefore using KIQ pathTested on AMD Cezanne (Renoir) with ROCm workloads after hibernation.
Result: Eliminates TLB flush failures on resume.Signed-off-by Ionut Nechita <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 6 ++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c | 9 +++++++-- drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c | 10 ++++++++++ drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 6 +++++- 5 files changed, 29 insertions(+), 3 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -1225,6 +1225,7 @@ struct amdgpu_device {bool in_s4;bool in_s0ix;suspend_state_t last_suspend_state; +bool resume_gpu_stable;enum pp_mp1_state mp1_state;struct amdgpu_doorbell_index doorbell_index;drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6
-0@@ -5457,6 +5457,12 @@ int amdgpu_device_resume(struct drm_device *dev, bool notify_clients) goto exit; } +/* +* Set resume_gpu_stable to false BEFORE KFD resume to ensure +* extended timeouts are used for TLB flushes during hibernation recovery +*/ + adev->resume_gpu_stable = false; + r =amdgpu_amdkfd_resume (adev, !amdgpu_sriov_vf (adev) && !adev->in_runpm);if (r) goto exit;drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +7
-2@@ -731,7 +731,12 @@ int amdgpu_gmc_flush_gpu_tlb_pasid(struct amdgpu_device *adev, uint16_t pasid,if (!down_read_trylock (&adev->reset_domain->sem))return 0 ; -if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready) { +/* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. +*/ +if (!adev->gmc.flush_pasid_uses_kiq || !ring->sched.ready || + !adev->resume_gpu_stable) {if (adev->gmc.flush_tlb_needs_extra_type_2) adev->gmc.gmc_funcs->flush_gpu_tlb_pasid(adev, pasid, 2, all_hub,@@ -835,9 +840,9 @@ void amdgpu_gmc_fw_reg_write_reg_wait(struct amdgpu_device *adev, goto failed_kiq;might_sleep (); +while (r <1 && cnt++ <MAX_KIQ_REG_TRY && !amdgpu_reset_pending (adev->reset_domain)) { -msleep (MAX_KIQ_REG_BAILOUT_INTERVAL ); r =amdgpu_fence_wait_polling (ring, seq,MAX_KIQ_REG_WAIT ); }drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +10
-0@@ -3985,6 +3985,16 @@ static int gfx_v9_0_cp_resume(struct amdgpu_device *adev)amdgpu_ring_test_helper (ring); } +/* +* After successful ring tests, mark GPU as stable for resume. +* This allows KIQ-based TLB invalidation to be used instead of +* slower direct MMIO path. +*/ +if (!adev->resume_gpu_stable) { + adev->resume_gpu_stable = true; +dev_info (adev->dev,"GPU rings verified, enabling KIQ path\n" ); + } +gfx_v9_0_enable_gui_idle_interrupt (adev, true);return 0 ;drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +5
-1@@ -855,9 +855,13 @@ static void gmc_v9_0_flush_gpu_tlb(struct amdgpu_device *adev, uint32_t vmid,/* This is necessary for SRIOV as well as for GFXOFF to function * properly under bare metal +* +* After hibernation resume, KIQ may report as ready but not be fully +* functional. Use direct MMIO path until GPU is confirmed stable. */ if (adev->gfx.kiq[inst].ring.sched.ready && - (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev))) { + (amdgpu_sriov_runtime (adev) || !amdgpu_sriov_vf (adev)) && + adev->resume_gpu_stable) {uint32_t req = hub->vm_inv_eng0_req + hub->eng_distance * eng;uint32_t ack = hub->vm_inv_eng0_ack + hub->eng_distance * eng;
From: Ionut Nechita <[email protected]>
Quoted 1 line · Christian König
Question is why the KIQ is not up and running before we do anything with it?
Thank you for the feedback. I completely understand that my patch is just a workaround and not proper engineering - you're absolutely right that the real issue is KIQ being marked as ready before it's actually functional.
I don't have experience with GPU drivers and video subsystems, so I'm not familiar with the proper initialization sequence for KIQ. I'd prefer not to keep a workaround for this issue in my tree.
Is there a proper fix available, or could you point me in the right direction? I'm happy to test any patches on my AMD Cezanne (Renoir) hardware where I can reliably reproduce the issue after hibernation.
Also, regarding hibernation support: you mentioned that hibernation is not generally supported with the driver. Should I expect other issues beyond this TLB flush problem, or is this the main blocker?
Thanks for your time,
Ionut
Quoted 12 lines · Ionut Nechita (Sunlight Linux)
From: Ionut Nechita <[email protected]>
Quoted 1 line · Christian König
Question is why the KIQ is not up and running before we do anything with it?
Thank you for the feedback. I completely understand that my patch is just a workaround and not proper engineering - you're absolutely right that the real issue is KIQ being marked as ready before it's actually functional.
Quoted 7 lines
I don't have experience with GPU drivers and video subsystems, so I'm not familiar with the proper initialization sequence for KIQ. I'd prefer not to keep a workaround for this issue in my tree.
Is there a proper fix available, or could you point me in the right direction? I'm happy to test any patches on my AMD Cezanne (Renoir) hardware where I can reliably reproduce the issue after hibernation.
Can you get a stack trace when this happens so we can see the call chain?
Quoted 4 lines
Also, regarding hibernation support: you mentioned that hibernation is not generally supported with the driver. Should I expect other issues beyond this TLB flush problem, or is this the main blocker?
The biggest issue with hibernation is that it's not compatible with secure boot so most distros don't officially support it. The other issue is that when we go into hibernation, we need to evict the contents of VRAM somewhere and at the point when that happens, swap is already offline. So in a lot of cases, we don't have enough memory to back up the VRAM contents. There were patches to the Linux PM core, but I can't recall if they've all landed yet. There's also the possibility that the user's swap partition is too small.
Alex
Quoted 25 lines · Alex Deucher
Quoted 12 lines · Ionut Nechita (Sunlight Linux)
From: Ionut Nechita <[email protected]>
Quoted 1 line · Christian König
Question is why the KIQ is not up and running before we do anything with it?
Thank you for the feedback. I completely understand that my patch is just a workaround and not proper engineering - you're absolutely right that the real issue is KIQ being marked as ready before it's actually functional.
Quoted 7 lines
I don't have experience with GPU drivers and video subsystems, so I'm not familiar with the proper initialization sequence for KIQ. I'd prefer not to keep a workaround for this issue in my tree.
Is there a proper fix available, or could you point me in the right direction? I'm happy to test any patches on my AMD Cezanne (Renoir) hardware where I can reliably reproduce the issue after hibernation.
Can you get a stack trace when this happens so we can see the call chain?
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Also, regarding hibernation support: you mentioned that hibernation is not generally supported with the driver. Should I expect other issues beyond this TLB flush problem, or is this the main blocker?
The biggest issue with hibernation is that it's not compatible with secure boot so most distros don't officially support it.
And by extension of this it doesn't get as much testing as s2idle/s3 do.
Quoted 6 lines
The other
issue is that when we go into hibernation, we need to evict the contents of VRAM somewhere and at the point when that happens, swap is already offline. So in a lot of cases, we don't have enough memory to back up the VRAM contents. There were patches to the Linux PM core, but I can't recall if they've all landed yet.
Yeah everything should have landed now, so swap will still be enabled.
There's also the
Quoted 3 lines
possibility that the user's swap partition is too small.
Alex
I heard something about /sys/power/reserved_size being too small by default still, so if you're having problems you might increase that.
Quoted 44 lines · Mario Limonciello (AMD) (kernel.org)
Quoted 25 lines · Alex Deucher
Quoted 12 lines · Ionut Nechita (Sunlight Linux)
From: Ionut Nechita <[email protected]>
Quoted 1 line · Christian König
Question is why the KIQ is not up and running before we do anything with it?
Thank you for the feedback. I completely understand that my patch is just a workaround and not proper engineering - you're absolutely right that the real issue is KIQ being marked as ready before it's actually functional.
Quoted 7 lines
I don't have experience with GPU drivers and video subsystems, so I'm not familiar with the proper initialization sequence for KIQ. I'd prefer not to keep a workaround for this issue in my tree.
Is there a proper fix available, or could you point me in the right direction? I'm happy to test any patches on my AMD Cezanne (Renoir) hardware where I can reliably reproduce the issue after hibernation.
Can you get a stack trace when this happens so we can see the call chain?
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Also, regarding hibernation support: you mentioned that hibernation is not generally supported with the driver. Should I expect other issues beyond this TLB flush problem, or is this the main blocker?
The biggest issue with hibernation is that it's not compatible with secure boot so most distros don't officially support it.
And by extension of this it doesn't get as much testing as s2idle/s3 do.
Quoted 6 lines
The other
issue is that when we go into hibernation, we need to evict the contents of VRAM somewhere and at the point when that happens, swap is already offline. So in a lot of cases, we don't have enough memory to back up the VRAM contents. There were patches to the Linux PM core, but I can't recall if they've all landed yet.
Quoted 9 lines
Yeah everything should have landed now, so swap will still be enabled.
There's also the
Quoted 3 lines
possibility that the user's swap partition is too small.
Alex
I heard something about
/sys/power/reserved_sizebeing too small by default still, so if you're having problems you might increase that.
Sorry not reserved_size, /sys/power/image_size.
Here's where it was mentioned.
https://gitlab.freedesktop.org/drm/amd/-/issues/4882#note_3287247
Hi Mario,
Thanks for the additional context about /sys/power/image_size and the GitLab issue reference.
My platform configuration appears to be well set up for hibernation:
- 30 GB RAM / 30 GB swap available
- /sys/power/image_size: ~12 GB (13033648128 bytes)
- Not using secure boot
- Swap partition is adequately sized for VRAM eviction
Based on the GitLab discussion you referenced, my image_size should be sufficient. So the known hibernation limitations shouldn't be blocking factors on my system - the TLB flush failure appears to be the main remaining issue.
I'll capture stack traces as Alex requested and follow up with the debugging information in the next few days.
Thanks,
Ionut
Hi Alex,
Thank you for the feedback and for taking the time to review this issue.
I'll add debug code to capture the full stack trace when the TLB flush failures occur. I'll test this on my AMD Cezanne system over the next few days when I have more time available, and will send you the complete call chain information.
Regarding the hibernation limitations you mentioned - I understand the challenges with secure boot compatibility and VRAM eviction. In my case, I'm not using secure boot, and my system has sufficient RAM and swap space to handle the VRAM backup, so those particular issues shouldn't affect my setup.
I'll follow up with the stack traces and additional debugging information in the next few days.
Thanks again,
Ionut
Christian König
maintainerHi Ionut,
Quoted 9 lines · Ionut Nechita (Sunlight Linux)
Hi Alex,
Thank you for the feedback and for taking the time to review this issue.
I'll add debug code to capture the full stack trace when the TLB flush failures occur. I'll test this on my AMD Cezanne system over the next few days when I have more time available, and will send you the complete call chain information.
Quoted 5 lines
Regarding the hibernation limitations you mentioned - I understand the challenges with secure boot compatibility and VRAM eviction. In my case, I'm not using secure boot, and my system has sufficient RAM and swap space to handle the VRAM backup, so those particular issues shouldn't affect my setup.
the problem is since the AMD GPU doesn't official support hibernation you are pretty much the only one able to reproduce the bug.
In other words I can't go to my manager to get a Cezanne system plus time to work on this.
What Alex, Mario, I and all the other devs can do is to explain to you how the driver works and try to further debug what's going on here.
Quoted 2 lines
I'll follow up with the stack traces and additional debugging information in the next few days.
That would certainly be helpful.
Thanks,
Christian.
Quoted 3 lines
Thanks again,
Ionut
Hi Alex,
Thank you for the quick response and for the information about hibernation support.
Here's the stack trace showing the call chain when the TLB flush failures occur. The issue happens in two places:
1. During resume (hibernation restore):
Call Trace: dump_stack_lvl+0x5b/0x80 amdgpu_gmc_fw_reg_write_reg_wait+0x1c7/0x1d0 [amdgpu] gmc_v9_0_hw_init+0x2e2/0x390 [amdgpu] gmc_v9_0_resume+0x26/0x70 [amdgpu] amdgpu_ip_block_resume+0x27/0x50 [amdgpu] amdgpu_device_ip_resume_phase1+0x55/0x90 [amdgpu] amdgpu_device_resume+0x69/0x380 [amdgpu] amdgpu_pmops_resume+0x46/0x80 [amdgpu] dpm_run_callback+0x4a/0x150 device_resume+0x1df/0x2f0 async_resume+0x21/0x30 async_run_entry_fn+0x36/0x160 process_one_work+0x193/0x350 worker_thread+0x2d7/0x410
2. Subsequent failures during command submission:
Call Trace: dump_stack_lvl+0x5b/0x80 amdgpu_gmc_fw_reg_write_reg_wait+0x1c7/0x1d0 [amdgpu] amdgpu_gmc_flush_gpu_tlb+0xd0/0x280 [amdgpu] amdgpu_gart_invalidate_tlb.part.0+0x59/0x90 [amdgpu] amdgpu_ttm_alloc_gart+0x146/0x180 [amdgpu] amdgpu_cs_parser_bos.isra.0+0x5d6/0x7d0 [amdgpu] amdgpu_cs_ioctl+0xbd0/0x1aa0 [amdgpu] drm_ioctl_kernel+0xa6/0x100 drm_ioctl+0x262/0x520 amdgpu_drm_ioctl+0x4a/0x80 [amdgpu]
Error message: "amdgpu 0000:04:00.0: amdgpu: failed to write reg 1a6f4 wait reg 1a706"
Full dmesg log available at: https://gitlab.freedesktop.org/-/project/4522/uploads/6a285ad2e24f4807e5d75c3f4ed5a7a1/dmesg-dump-stack.txt
Regarding the hibernation support issues you mentioned - I understand the limitations with secure boot and VRAM eviction. In my case, I have secure boot disabled and sufficient swap space, so the primary issue I'm encountering is this TLB flush failure.
I'm happy to test any patches or help with further debugging if needed.
Thanks,
Ionut
Christian König
maintainerHi Ionut,
Quoted 38 lines · Ionut Nechita (Sunlight Linux)
Hi Alex,
Thank you for the quick response and for the information about hibernation support.
Here's the stack trace showing the call chain when the TLB flush failures occur. The issue happens in two places:
1. During resume (hibernation restore):
Call Trace: dump_stack_lvl+0x5b/0x80 amdgpu_gmc_fw_reg_write_reg_wait+0x1c7/0x1d0 [amdgpu] gmc_v9_0_hw_init+0x2e2/0x390 [amdgpu] gmc_v9_0_resume+0x26/0x70 [amdgpu] amdgpu_ip_block_resume+0x27/0x50 [amdgpu] amdgpu_device_ip_resume_phase1+0x55/0x90 [amdgpu] amdgpu_device_resume+0x69/0x380 [amdgpu] amdgpu_pmops_resume+0x46/0x80 [amdgpu] dpm_run_callback+0x4a/0x150 device_resume+0x1df/0x2f0 async_resume+0x21/0x30 async_run_entry_fn+0x36/0x160 process_one_work+0x193/0x350 worker_thread+0x2d7/0x4102. Subsequent failures during command submission:
Call Trace: dump_stack_lvl+0x5b/0x80 amdgpu_gmc_fw_reg_write_reg_wait+0x1c7/0x1d0 [amdgpu] amdgpu_gmc_flush_gpu_tlb+0xd0/0x280 [amdgpu] amdgpu_gart_invalidate_tlb.part.0+0x59/0x90 [amdgpu] amdgpu_ttm_alloc_gart+0x146/0x180 [amdgpu] amdgpu_cs_parser_bos.isra.0+0x5d6/0x7d0 [amdgpu] amdgpu_cs_ioctl+0xbd0/0x1aa0 [amdgpu] drm_ioctl_kernel+0xa6/0x100 drm_ioctl+0x262/0x520 amdgpu_drm_ioctl+0x4a/0x80 [amdgpu]
Quoted 1 line
Error message: "amdgpu 0000:04:00.0: amdgpu: failed to write reg 1a6f4 wait reg 1a706"
well thanks for doing this, but that unfortunately doesn't help us solve this. That result is perfectly expected.
Can you check if the KIQ ring test is executed correctly after hibernation? E.g. what happens when amdgpu_ring_test_ring(kiq_ring) is called? Is that called at all?
Thanks,
Christian.
Quoted 8 lines
Full dmesg log available at: https://gitlab.freedesktop.org/-/project/4522/uploads/6a285ad2e24f4807e5d75c3f4ed5a7a1/dmesg-dump-stack.txt
Regarding the hibernation support issues you mentioned - I understand the limitations with secure boot and VRAM eviction. In my case, I have secure boot disabled and sufficient swap space, so the primary issue I'm encountering is this TLB flush failure.
I'm happy to test any patches or help with further debugging if needed.
Thanks,
Ionut
Hi Christian,
Thank you for the guidance on checking the KIQ ring test.
I apologize for the late reply - I haven't had time to investigate this further as I've been busy with other work. I will try to look into the KIQ ring test behavior after hibernation resume in the coming period and report back with my findings.
Thanks,
Ionut