Skip to contentSkip to conversation

drm/amdgpu: Restore msix after FLR

PATCH Superseded Peng Ju Zhou 2 participants 5 messages Jun 25, 2021 – Jul 1, 2021

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 drm/amdgpu: Restore msix after FLR

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +11 -0
index 90f50561b43a..26e63cb5d8d5 100644
@@ -277,6 +277,16 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
 	return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev)
+{
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+}
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +568,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev)
 {
 	int i, j, k;
 
+	amdgpu_restore_msix(adev);
 	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
 		if (!adev->irq.client[i].sources)
 			continue;
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

[AMD Official Use Only]

Ping

----------------------------------------------------------------------
BW
Pengju Zhou

Quoted 51 lines

-----Original Message-----
From: Peng Ju Zhou <[email protected]>
Sent: Friday, June 25, 2021 2:44 PM
To: [email protected]
Cc: Deng, Emily <[email protected]>; Zhou, Peng Ju <[email protected]>
Subject: [PATCH v3] drm/amdgpu: Restore msix after FLR

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +10 -1
index 90f50561b43a..26e63cb5d8d5 100644
@@ -277,6 +277,16 @@ static bool amdgpu_msi_ok(struct amdgpu_device
 *adev)
 	return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev) {
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap +
 PCI_MSIX_FLAGS, &ctrl);
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap +
 PCI_MSIX_FLAGS, ctrl);
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap +
+PCI_MSIX_FLAGS, ctrl); }
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +568,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct
 amdgpu_device *adev)  {
 	int i, j, k;
 
+	amdgpu_restore_msix(adev);
 	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
 		if (!adev->irq.client[i].sources)
 			continue;
--
Quoted 1 line

2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw
In reply to Peng Ju Zhou
Quoted 4 lines · Peng Ju Zhou

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

Do we need to store whether we enabled msix in the first place and then decide whether to enable it again in this case?

Alex

Quoted 39 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +11 -1
index 90f50561b43a..26e63cb5d8d5 100644
@@ -277,6 +277,16 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
        return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev)
+{
+       u16 ctrl;
+
+       pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
+       ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+       ctrl |= PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+}
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +568,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev)
 {
        int i, j, k;
 
+       amdgpu_restore_msix(adev);
        for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
                if (!adev->irq.client[i].sources)
                        continue;
--

2.17.1

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

[AMD Official Use Only]

Hi Alex
The function amdgpu_restore_msix is used for reset the msix during board reset(sriov reset or asic reset), it moves from host to guest, so I think a flag to indicate if msix enabled is not needed.

----------------------------------------------------------------------
BW
Pengju Zhou

Quoted 71 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Tuesday, June 29, 2021 10:28 PM
To: Zhou, Peng Ju <[email protected]>
Cc: amd-gfx list <[email protected]>; Deng, Emily <[email protected]>
Subject: Re: [PATCH v3] drm/amdgpu: Restore msix after FLR

Quoted 4 lines · Peng Ju Zhou

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

Do we need to store whether we enabled msix in the first place and then decide whether to enable it again in this case?

Alex

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device

Quoted 13 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +0 -0
index 90f50561b43a..26e63cb5d8d5 100644
@@ -277,6 +277,16 @@ static bool amdgpu_msi_ok(struct amdgpu_device

*adev)

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev)

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +4 -0
        return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev) {
+       u16 ctrl;
+
+       pci_read_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, &ctrl);

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device · line 285

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +2 -0
+       ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, ctrl);

Quoted 17 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +4 -1
+       ctrl |= PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap +
+PCI_MSIX_FLAGS, ctrl); }
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +568,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct
 amdgpu_device *adev)  {
        int i, j, k;
 
+       amdgpu_restore_msix(adev);
        for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
                if (!adev->irq.client[i].sources)
                        continue;
--

2.17.1

Quoted 5 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.freedesktop.org%2Fmailman%2Flistinfo%2Famd-

gfx&amp;data=04%7C01%7CPe

Quoted 1 line

ngJu.Zhou%40amd.com%7C409895f80e0d43ecba3808d93b0a15fc%7C3dd8961 fe4884

Quoted 1 line
Quoted 9 lines

e608e11a82d994e183d%7C0%7C0%7C637605736778199787%7CUnknown%7C TWFpbGZsb

Quoted 1 line

3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%

Quoted 1 line

7C1000&amp;sdata=w%2FgDzhoAjDraAMiyfx3XTPxx1QNff3OY%2BZWn1NYq% 2Ffo%3D&

Quoted 1 line

amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw
Quoted 6 lines · Peng Ju

[AMD Official Use Only]

Hi Alex
The function amdgpu_restore_msix is used for reset the msix during board reset(sriov reset or asic reset), it moves from host to guest, so I think a flag to indicate if msix enabled is not needed.

The function ultimately enables MSIX. What if it was not enabled in the first place?

Alex

Quoted 87 lines

----------------------------------------------------------------------
BW
Pengju Zhou

Quoted 80 lines

-----Original Message-----
From: Alex Deucher <[email protected]>
Sent: Tuesday, June 29, 2021 10:28 PM
To: Zhou, Peng Ju <[email protected]>
Cc: amd-gfx list <[email protected]>; Deng, Emily <[email protected]>
Subject: Re: [PATCH v3] drm/amdgpu: Restore msix after FLR

Quoted 4 lines · Peng Ju Zhou

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

Do we need to store whether we enabled msix in the first place and then decide whether to enable it again in this case?

Alex

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device

Quoted 13 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +0 -0
index 90f50561b43a..26e63cb5d8d5 100644
@@ -277,6 +277,16 @@ static bool amdgpu_msi_ok(struct amdgpu_device

*adev)

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev)

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +4 -0
        return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev) {
+       u16 ctrl;
+
+       pci_read_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, &ctrl);

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device · line 285

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +2 -0
+       ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, ctrl);

Quoted 17 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +4 -1
+       ctrl |= PCI_MSIX_FLAGS_ENABLE;
+       pci_write_config_word(adev->pdev, adev->pdev->msix_cap +
+PCI_MSIX_FLAGS, ctrl); }
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +568,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct
 amdgpu_device *adev)  {
        int i, j, k;
 
+       amdgpu_restore_msix(adev);
        for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
                if (!adev->irq.client[i].sources)
                        continue;
--

2.17.1

Quoted 5 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.freedesktop.org%2Fmailman%2Flistinfo%2Famd-

gfx&amp;data=04%7C01%7CPe

Quoted 1 line

ngJu.Zhou%40amd.com%7C409895f80e0d43ecba3808d93b0a15fc%7C3dd8961 fe4884

Quoted 1 line

e608e11a82d994e183d%7C0%7C0%7C637605736778199787%7CUnknown%7C TWFpbGZsb

Quoted 1 line

3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0% 3D%

Quoted 1 line

7C1000&amp;sdata=w%2FgDzhoAjDraAMiyfx3XTPxx1QNff3OY%2BZWn1NYq% 2Ffo%3D&

Quoted 1 line

amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx