Skip to contentSkip to conversation

drm/amdgpu: fix for suspend/resume sequence under sriov

PATCH In review Victor Zhao 4 participants 5 messages Nov 3, 2022 – Nov 9, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: fix for suspend/resume sequence under sriov

- clear kiq ring after suspend/resume under sriov to aviod kiq ring test failure
- update irq after resume to fix kiq interrput loss

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 2 ++
 2 files changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 522820eeaa59..5b9f992e4607 100644
@@ -4197,6 +4197,8 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
 	}
 
 	/* Make sure IB tests flushed */
+	if (amdgpu_sriov_vf(adev))
+		amdgpu_irq_gpu_reset_resume_helper(adev);
 	flush_delayed_work(&adev->delayed_init_work);
 
 	if (adev->in_s0ix) {
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 -0
index 7853d3ca58cf..49d34c7bbf20 100644
@@ -6909,6 +6909,8 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)
 		mutex_unlock(&adev->srbm_mutex);
 	} else {
 		memset((void *)mqd, 0, sizeof(*mqd));
+		if (amdgpu_sriov_vf(adev) && adev->in_suspend)
+			amdgpu_ring_clear_ring(ring);
 		mutex_lock(&adev->srbm_mutex);
 		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
 		amdgpu_ring_init_mqd(ring);
Signature · 1 line
2.25.1
#1 lore raw

[AMD Official Use Only - General]

Hi Alex,

This is a patch fixing the sriov suspend/resume sequence. Please help review.

Thanks,
Victor

-----Original Message-----
From: Victor Zhao <[email protected]>
Sent: Thursday, November 3, 2022 12:06 PM
To: [email protected]; Deucher, Alexander <[email protected]>
Cc: Zhao, Victor <[email protected]>
Subject: [PATCH] drm/amdgpu: fix for suspend/resume sequence under sriov

- clear kiq ring after suspend/resume under sriov to aviod kiq ring test failure
- update irq after resume to fix kiq interrput loss

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 2 ++
 2 files changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 522820eeaa59..5b9f992e4607 100644
@@ -4197,6 +4197,8 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
 	}
 
 	/* Make sure IB tests flushed */
+	if (amdgpu_sriov_vf(adev))
+		amdgpu_irq_gpu_reset_resume_helper(adev);
 	flush_delayed_work(&adev->delayed_init_work);
 
 	if (adev->in_s0ix) {
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 -0
index 7853d3ca58cf..49d34c7bbf20 100644
@@ -6909,6 +6909,8 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)
 		mutex_unlock(&adev->srbm_mutex);
 	} else {
 		memset((void *)mqd, 0, sizeof(*mqd));
+		if (amdgpu_sriov_vf(adev) && adev->in_suspend)
+			amdgpu_ring_clear_ring(ring);
 		mutex_lock(&adev->srbm_mutex);
 		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
 		amdgpu_ring_init_mqd(ring);

--
2.25.1

#2 lore raw

[AMD Official Use Only - General]

@Deucher, Alexander<mailto:[email protected]>

Following up on this to see if the patch reviewed and approved for up streaming?

Thanks,

SIRISH

-----Original Message-----
From: Zhao, Victor <[email protected]>
Sent: Thursday, November 3, 2022 12:08 AM
To: Zhao, Victor <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>
Cc: Zhang, Bokun <[email protected]>; Chickanayakanahalli, Sirish <[email protected]>
Subject: RE: [PATCH] drm/amdgpu: fix for suspend/resume sequence under sriov

[AMD Official Use Only - General]

Hi Alex,

This is a patch fixing the sriov suspend/resume sequence. Please help review.

Thanks,

Victor

-----Original Message-----

From: Victor Zhao <[email protected]<mailto:[email protected]>>

Sent: Thursday, November 3, 2022 12:06 PM

To: [email protected]<mailto:[email protected]>; Deucher, Alexander <[email protected]<mailto:[email protected]>>

Cc Zhao, Victor <[email protected]<mailto:[email protected]>>

Subject: [PATCH] drm/amdgpu: fix for suspend/resume sequence under sriov

- clear kiq ring after suspend/resume under sriov to aviod kiq ring test failure

- update irq after resume to fix kiq interrput loss

---

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++

drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c | 2 ++

2 files changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0

index 522820eeaa59..5b9f992e4607 100644

--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c

@@ -4197,6 +4197,8 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)

               }



                /* Make sure IB tests flushed */

+ if (amdgpu_sriov_vf(adev))

+ amdgpu_irq_gpu_reset_resume_helper(adev);

               flush_delayed_work(&adev->delayed_init_work);



                if (adev->in_s0ix) {
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +0 -0

index 7853d3ca58cf..49d34c7bbf20 100644

--- a/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c

+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c

@@ -6909,6 +6909,8 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)

                               mutex_unlock(&adev->srbm_mutex);

               } else {

                               memset((void *)mqd, 0, sizeof(*mqd));

+ if (amdgpu_sriov_vf(adev) && adev->in_suspend)

+ amdgpu_ring_clear_ring(ring);

                               mutex_lock(&adev->srbm_mutex);

                               nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);

                               amdgpu_ring_init_mqd(ring);

--

2.25.1

Acked-by on v1 1/1
#3 lore raw
In reply to Victor Zhao
Quoted 29 lines · Victor Zhao

- clear kiq ring after suspend/resume under sriov to aviod kiq ring test failure
- update irq after resume to fix kiq interrput loss

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 2 ++
 2 files changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index 522820eeaa59..5b9f992e4607 100644
@@ -4197,6 +4197,8 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
        }
 
        /* Make sure IB tests flushed */
+       if (amdgpu_sriov_vf(adev))
+               amdgpu_irq_gpu_reset_resume_helper(adev);
        flush_delayed_work(&adev->delayed_init_work);
 
        if (adev->in_s0ix) {
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +0 -0
index 7853d3ca58cf..49d34c7bbf20 100644

drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring) · line 6913

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 -0
@@ -6909,6 +6909,8 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)
                mutex_unlock(&adev->srbm_mutex);
        } else {
                memset((void *)mqd, 0, sizeof(*mqd));
+               if (amdgpu_sriov_vf(adev) && adev->in_suspend)
+                       amdgpu_ring_clear_ring(ring);

gfx_v8_0.c, gfx_v9_0.c, and gfx_v11_0.c need a similar fix. With those fixed as well, the patch is:

Alex

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +0 -1
                mutex_lock(&adev->srbm_mutex);
                nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
                amdgpu_ring_init_mqd(ring);
--

2.25.1

#4 lore raw
In reply to Victor Zhao

Am 03.11.22 um 05:06 schrieb Victor Zhao:

Quoted 3 lines

- clear kiq ring after suspend/resume under sriov to aviod kiq ring test failure
- update irq after resume to fix kiq interrput loss

Good to see that somebody takes a look into this. Is that enough to get suspend/resume with SRIOV going?

Quoted 12 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
  drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c     | 2 ++
  2 files changed, 4 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index 522820eeaa59..5b9f992e4607 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_device_resume(struct drm_device *dev, bool fbcon) · line 4201

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
@@ -4197,6 +4197,8 @@ int amdgpu_device_resume(struct drm_device *dev, bool fbcon)
  	}
 
  	/* Make sure IB tests flushed */
+	if (amdgpu_sriov_vf(adev))
+		amdgpu_irq_gpu_reset_resume_helper(adev);

This is a pretty clear NAK because that should happen during resume anyway. If this doesn't happen we have a bug somewhere else and that here just hides it.

drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring) · line 6913

Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
  	flush_delayed_work(&adev->delayed_init_work);
 
  	if (adev->in_s0ix) {
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +2 -0
index 7853d3ca58cf..49d34c7bbf20 100644
@@ -6909,6 +6909,8 @@ static int gfx_v10_0_kiq_init_queue(struct amdgpu_ring *ring)
  		mutex_unlock(&adev->srbm_mutex);
  	} else {
  		memset((void *)mqd, 0, sizeof(*mqd));
+		if (amdgpu_sriov_vf(adev) && adev->in_suspend)
+			amdgpu_ring_clear_ring(ring);

Is there any good reason to not always clear the KIQ ring here? E.g. also on bare metal and during load/reset?

Regards,
Christian.

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/gfx_v10_0.c +0 -0
  		mutex_lock(&adev->srbm_mutex);
  		nv_grbm_select(adev, ring->me, ring->pipe, ring->queue, 0);
  		amdgpu_ring_init_mqd(ring);