Skip to contentSkip to conversation

drm/amdgpu: clear UVD VCPU buffer when err_event_athub generated

PATCH In review Le Ma 3 participants 9 messages Oct 28, 2019 – Oct 29, 2019 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Le Ma

author
#0 lore raw

PATCH 1/4 drm/amdgpu: clear UVD VCPU buffer when err_event_athub generated

The err_event_athub error will mess up the buffer and cause UVD resume hang.

Change-Id If17a2161fb9b1b52eac08de00d2e935191bdbf99
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9 -1
index b2c364b..b4dd89a 100644
@@ -39,6 +39,8 @@
 #include "cikd.h"
 #include "uvd/uvd_4_2_d.h"
 
+#include "amdgpu_ras.h"
+
 /* 1 second timeout */
 #define UVD_IDLE_TIMEOUT	msecs_to_jiffies(1000)
 
@@ -372,7 +374,13 @@ int amdgpu_uvd_suspend(struct amdgpu_device *adev)
 		if (!adev->uvd.inst[j].saved_bo)
 			return -ENOMEM;
 
-		memcpy_fromio(adev->uvd.inst[j].saved_bo, ptr, size);
+		/* re-write 0 since err_event_athub will corrupt VCPU buffer */
+		if (amdgpu_ras_intr_triggered()) {
+			DRM_WARN("UVD VCPU state may lost due to RAS ERREVENT_ATHUB_INTERRUPT\n");
+			memset(adev->uvd.inst[j].saved_bo, 0, size);
+		} else {
+			memcpy_fromio(adev->uvd.inst[j].saved_bo, ptr, size);
+		}
 	}
 	return 0;
 }
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1572262269-14985-1-git-send-email-le.ma-5C7GfCeVMHo@public.gmane.org

author
#2 lore raw

PATCH 2/4 drm/amdgpu: reset err_event_athub flag if gpu recovery succeeded

Otherwise next err_event_athub error cannot call gpu reset.

Change-Id I5cd293f30f23876bf2a1860681bcb50f47713ecd
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -0
index 676cad1..51d74bb 100644
@@ -4089,6 +4089,9 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
 		}
 	}
 
+	if (!r && in_ras_intr)
+		atomic_set(&amdgpu_ras_in_intr, 0);
+
 skip_sched_resume:
 	list_for_each_entry(tmp_adev, device_list_handle, gmc.xgmi.head) {
 		/*unlock kfd: SRIOV would do it separately */
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1572262269-14985-2-git-send-email-le.ma-5C7GfCeVMHo@public.gmane.org

#4 lore raw

RE: [PATCH 2/4] drm/amdgpu: reset err_event_athub flag if gpu recovery succeeded

Regards,
Guchun

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Le Ma
Sent: Monday, October 28, 2019 7:31 PM
To: [email protected]
Cc: Ma, Le <[email protected]>
Subject: [PATCH 2/4] drm/amdgpu: reset err_event_athub flag if gpu recovery succeeded

Otherwise next err_event_athub error cannot call gpu reset.

Change-Id I5cd293f30f23876bf2a1860681bcb50f47713ecd
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 676cad1..51d74bb 100644
@@ -4089,6 +4089,9 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev,
 		}
 	}
 
+	if (!r && in_ras_intr)
+		atomic_set(&amdgpu_ras_in_intr, 0);

[Guchun]To access this atomic variable, maybe it's better we create a new function like reset or clear in amdgpu_ras.h or .c first, then we can call that function here, like we we do to amdgpu_ras_intr_triggered in this same function. This will do assist to modularity of ras driver.

skip_sched_resume:
list_for_each_entry(tmp_adev, device_list_handle, gmc.xgmi.head) {
/*unlock kfd: SRIOV would do it separately */

Signature · 10 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BYAPR12MB280615A3803ADC31A4AE9C8EF1610-ZGDeBxoHBPk0CuAkIMgl3QdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

author
#6 lore raw

RE: [PATCH 2/4] drm/amdgpu: reset err_event_athub flag if gpu recovery succeeded

Quoted 110 lines

-----Original Message-----

From: Chen, Guchun <[email protected]>

Sent: Tuesday, October 29, 2019 9:28 AM

To: Ma, Le <[email protected]>; [email protected]

Cc Ma, Le <[email protected]>

Subject: RE: [PATCH 2/4] drm/amdgpu: reset err_event_athub flag if gpu

recovery succeeded

Regards,

Guchun

-----Original Message-----

From: amd-gfx <[email protected]<mailto:[email protected]>> On Behalf Of Le Ma

Sent: Monday, October 28, 2019 7:31 PM

To: [email protected]<mailto:[email protected]>

Subject: [PATCH 2/4] drm/amdgpu: reset err_event_athub flag if gpu recovery

succeeded

Otherwise next err_event_athub error cannot call gpu reset.

Change-Id: I5cd293f30f23876bf2a1860681bcb50f47713ecd

---

 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 3 +++
 1 file changed, 3 insertions(+)
(quoted diff) +0 -0

b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

index 676cad1..51d74bb 100644

--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

(quoted diff) +2 -0
@@ -4089,6 +4089,9 @@ int amdgpu_device_gpu_recover(struct
 
 amdgpu_device *adev,
 
                     }
 
         }
 
 
 
+      if (!r && in_ras_intr)
 
+                  atomic_set(&amdgpu_ras_in_intr, 0);

[Guchun]To access this atomic variable, maybe it's better we create a new

function like reset or clear in amdgpu_ras.h or .c first, then we can call that

function here, like we we do to amdgpu_ras_intr_triggered in this same

function. This will do assist to modularity of ras driver.

[Le] Agree with you. We could make it paired with amdgpu_ras_intr_triggered.

skip_sched_resume:

         list_for_each_entry(tmp_adev, device_list_handle, gmc.xgmi.head) {

                     /*unlock kfd: SRIOV would do it separately */

--

2.7.4

Quoted 7 lines

_______________________________________________

amd-gfx mailing list

[email protected]<mailto:[email protected]>

https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#7 lore raw

PATCH 3/4 drm/amdgpu: bypass some cleanup work after err_event_athub

In reply to a message not in the archive

PSP lost connection when err_event_athub occurs. These cleanup work can be skipped in BACO reset.

Change-Id If54a3735edd6ccbb58d40a5f8833392981f8ce37
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  6 ++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c    |  7 +++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c    | 20 +++++++++++---------
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      |  6 ++++--
 4 files changed, 28 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +6 -0
index 51d74bb..72d9892 100644
@@ -2274,6 +2274,12 @@ static int amdgpu_device_ip_suspend_phase2(struct amdgpu_device *adev)
 		/* displays are handled in phase1 */
 		if (adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_DCE)
 			continue;
+		/* PSP lost connection when err_event_athub occurs */
+		if (amdgpu_ras_intr_triggered() &&
+		    adev->ip_blocks[i].version->type == AMD_IP_BLOCK_TYPE_PSP) {
+			adev->ip_blocks[i].status.hw = false;
+			continue;
+		}
 		/* XXX handle errors */
 		r = adev->ip_blocks[i].version->funcs->suspend(adev);
 		/* XXX handle errors */
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +7 -0
index fd7a73f..fce206f 100644
@@ -167,6 +167,13 @@ psp_cmd_submit_buf(struct psp_context *psp,
 	while (*((unsigned int *)psp->fence_buf) != index) {
 		if (--timeout == 0)
 			break;
+		/*
+		 * Shouldn't wait for timeout when err_event_athub occurs,
+		 * because gpu reset thread triggered and lock resource should
+		 * be released for psp resume sequence.
+		 */
+		if (amdgpu_ras_intr_triggered())
+			break;
 		msleep(1);
 		amdgpu_asic_invalidate_hdp(psp->adev, NULL);
 	}
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +11 -9
index 796326b..dab90c2 100644
@@ -558,15 +558,17 @@ int amdgpu_ras_feature_enable(struct amdgpu_device *adev,
 	if (!(!!enable ^ !!amdgpu_ras_is_feature_enabled(adev, head)))
 		return 0;
 
-	ret = psp_ras_enable_features(&adev->psp, &info, enable);
-	if (ret) {
-		DRM_ERROR("RAS ERROR: %s %s feature failed ret %d\n",
-				enable ? "enable":"disable",
-				ras_block_str(head->block),
-				ret);
-		if (ret == TA_RAS_STATUS__RESET_NEEDED)
-			return -EAGAIN;
-		return -EINVAL;
+	if (!amdgpu_ras_intr_triggered()) {
+		ret = psp_ras_enable_features(&adev->psp, &info, enable);
+		if (ret) {
+			DRM_ERROR("RAS ERROR: %s %s feature failed ret %d\n",
+					enable ? "enable":"disable",
+					ras_block_str(head->block),
+					ret);
+			if (ret == TA_RAS_STATUS__RESET_NEEDED)
+				return -EAGAIN;
+			return -EINVAL;
+		}
 	}
 
 	/* setup the obj */
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +4 -2
index 9fe95e7..9c2dba62 100644
@@ -3736,8 +3736,10 @@ static int gfx_v9_0_hw_fini(void *handle)
 	amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0);
 	amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0);
 
-	/* disable KCQ to avoid CPC touch memory not valid anymore */
-	gfx_v9_0_kcq_disable(adev);
+	/* DF freeze and kcq disable will fail */
+	if (!amdgpu_ras_intr_triggered())
+		/* disable KCQ to avoid CPC touch memory not valid anymore */
+		gfx_v9_0_kcq_disable(adev);
 
 	if (amdgpu_sriov_vf(adev)) {
 		gfx_v9_0_cp_gfx_enable(adev, false);
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#8 lore raw

PATCH 4/4 drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

In reply to a message not in the archive

From: Le Ma <[email protected]>

Change-Id Ia8a61a4b3bd529f0f691e43e69b299d7d151c0c2
---
 drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c +5 -1
index 0db458f..876690a 100644
@@ -324,7 +324,11 @@ static void nbio_v7_4_handle_ras_controller_intr_no_bifring(struct amdgpu_device
 						RAS_CNTLR_INTERRUPT_CLEAR, 1);
 		WREG32_SOC15(NBIO, 0, mmBIF_DOORBELL_INT_CNTL, bif_doorbell_intr_cntl);
 
-		amdgpu_ras_global_ras_isr(adev);
+		/*
+		 * ras_controller_int is dedicated for nbif ras error,
+		 * not the global interrupt for sync flood
+		 */
+		amdgpu_ras_reset_gpu(adev, true);
 	}
 }
 
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1572262269-14985-4-git-send-email-le.ma-5C7GfCeVMHo@public.gmane.org

#10 lore raw

RE: [PATCH 4/4] drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

Regards,
Guchun

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Le Ma
Sent: Monday, October 28, 2019 7:31 PM
To: [email protected]
Cc: Ma, Le <[email protected]>
Subject: [PATCH 4/4] drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

From: Le Ma <[email protected]>

Change-Id Ia8a61a4b3bd529f0f691e43e69b299d7d151c0c2
---
 drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c +5 -1
index 0db458f..876690a 100644
@@ -324,7 +324,11 @@ static void nbio_v7_4_handle_ras_controller_intr_no_bifring(struct amdgpu_device
 						RAS_CNTLR_INTERRUPT_CLEAR, 1);
 		WREG32_SOC15(NBIO, 0, mmBIF_DOORBELL_INT_CNTL, bif_doorbell_intr_cntl);
 
-		amdgpu_ras_global_ras_isr(adev);
+		/*
+		 * ras_controller_int is dedicated for nbif ras error,
+		 * not the global interrupt for sync flood
+		 */
+		amdgpu_ras_reset_gpu(adev, true);

[Guchun]We need to add one printing here to tell audience, who and why resets gpu? And moreover, in the removed global ras isr handler amdgpu_ras_global_ras_isr, we call amdgpu_ras_reset_gpu with is_baco parameter "false", but now we use "true" here?
}
}

Signature · 10 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BYAPR12MB2806A8C355785EFB07D88E2EF1610-ZGDeBxoHBPk0CuAkIMgl3QdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

author
#12 lore raw

RE: [PATCH 4/4] drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Tuesday, October 29, 2019 9:37 AM
To: Ma, Le <[email protected]>; [email protected]
Cc: Ma, Le <[email protected]>
Subject: RE: [PATCH 4/4] drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

Regards,

Guchun

-----Original Message-----

From: amd-gfx <[email protected]<mailto:[email protected]>> On Behalf Of Le Ma

Sent: Monday, October 28, 2019 7:31 PM

To: [email protected]<mailto:[email protected]>

Subject: [PATCH 4/4] drm/amdgpu: remove ras global recovery handling from ras_controller_int handler

From: Le Ma <[email protected]<mailto:[email protected]>>

Change-Id: Ia8a61a4b3bd529f0f691e43e69b299d7d151c0c2

---

drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c | 6 +++++-

1 file changed, 5 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c +0 -0

index 0db458f..876690a 100644

--- a/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c

+++ b/drivers/gpu/drm/amd/amdgpu/nbio_v7_4.c

@@ -324,7 +324,11 @@ static void nbio_v7_4_handle_ras_controller_intr_no_bifring(struct amdgpu_device

                                                                       RAS_CNTLR_INTERRUPT_CLEAR, 1);

                       WREG32_SOC15(NBIO, 0, mmBIF_DOORBELL_INT_CNTL, bif_doorbell_intr_cntl);

- amdgpu_ras_global_ras_isr(adev);

+ /*

+ * ras_controller_int is dedicated for nbif ras error,

+ * not the global interrupt for sync flood

+ */

+ amdgpu_ras_reset_gpu(adev, true);

[Guchun]We need to add one printing here to tell audience, who and why resets gpu? And moreover, in the removed global ras isr handler amdgpu_ras_global_ras_isr, we call amdgpu_ras_reset_gpu with is_baco parameter "false", but now we use "true" here?

[Le] We may consider add printing here to indicate it’s ras controller interrupt issue. The is_baco parameter is unused and has no effect. Anyway, I will revise and hold on patch #2 and #4 when baco based RAS recovery totally works as Hawking’s comment.

           }

}

--

2.7.4

_______________________________________________

amd-gfx mailing list

[email protected]<mailto:[email protected]>

https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Reviewed-by Hawking Zhang
#13 lore raw
In reply to a message not in the archive

We should hold on patch #2 and patch #4 until we have baco based RAS recovery works since current ras recovery policy is changed by these two patches.

Other than that, the Series is

Regards,
Hawking
-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Le Ma
Sent: 2019年10月28日 19:31
To: [email protected]
Cc: Ma, Le <[email protected]>
Subject: [PATCH 1/4] drm/amdgpu: clear UVD VCPU buffer when err_event_athub generated

The err_event_athub error will mess up the buffer and cause UVD resume hang.

Change-Id If17a2161fb9b1b52eac08de00d2e935191bdbf99
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9 -1
index b2c364b..b4dd89a 100644
@@ -39,6 +39,8 @@
 #include "cikd.h"
 #include "uvd/uvd_4_2_d.h"
 
+#include "amdgpu_ras.h"
+
 /* 1 second timeout */
 #define UVD_IDLE_TIMEOUT	msecs_to_jiffies(1000)
 
@@ -372,7 +374,13 @@ int amdgpu_uvd_suspend(struct amdgpu_device *adev)
 		if (!adev->uvd.inst[j].saved_bo)
 			return -ENOMEM;
 
-		memcpy_fromio(adev->uvd.inst[j].saved_bo, ptr, size);
+		/* re-write 0 since err_event_athub will corrupt VCPU buffer */
+		if (amdgpu_ras_intr_triggered()) {
+			DRM_WARN("UVD VCPU state may lost due to RAS ERREVENT_ATHUB_INTERRUPT\n");
+			memset(adev->uvd.inst[j].saved_bo, 0, size);
+		} else {
+			memcpy_fromio(adev->uvd.inst[j].saved_bo, ptr, size);
+		}
 	}
 	return 0;
 }
Signature · 10 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx