Skip to contentSkip to conversation

drm/amdgpu: Restore msix after FLR

PATCH Superseded Peng Ju Zhou 3 participants 4 messages Jul 2, 2021 – Jul 7, 2021

Subsystems (2)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 drm/amdgpu: Restore msix after FLR

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +14 -0
index 90f50561b43a..034420c38352 100644
@@ -277,6 +277,19 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
 	return true;
 }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev)
+{
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
+	if (!(ctrl & PCI_MSIX_FLAGS_ENABLE))
+		return;
+
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+}
 /**
  * amdgpu_irq_init - initialize interrupt handling
  *
@@ -558,6 +571,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev)
 {
 	int i, j, k;
 
+	amdgpu_restore_msix(adev);
 	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
 		if (!adev->irq.client[i].sources)
 			continue;
Signature · 6 lines
2.17.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

Am 02.07.21 um 05:23 schrieb Peng Ju Zhou:

Quoted 15 lines

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +0 -0
index 90f50561b43a..034420c38352 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev) · line 280

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -0
@@ -277,6 +277,19 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
  	return true;
  }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev)

I think this function should be static and maybe add a one line comment like "Clear and re-set the MSIX flags if they where set before to trigger re-enabling it".

With that done feel free to add an Acked-by: Christian König <[email protected]>, but I would also wait what Alex has to say about that.

Regards,
Christian

Quoted 16 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +12 -0
+{
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
+	if (!(ctrl & PCI_MSIX_FLAGS_ENABLE))
+		return;
+
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+}
  /**
   * amdgpu_irq_init - initialize interrupt handling
   *

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev) · line 577

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -0
@@ -558,6 +571,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev)
  {
  	int i, j, k;
 
+	amdgpu_restore_msix(adev);
  	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
  		if (!adev->irq.client[i].sources)
  			continue;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw
In reply to Peng Ju Zhou
Quoted 20 lines · Peng Ju Zhou

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +2 -0
index 90f50561b43a..034420c38352 100644
@@ -277,6 +277,19 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
  	return true;
  }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev)
+{

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev) · line 291

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +10 -0
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
+	if (!(ctrl & PCI_MSIX_FLAGS_ENABLE))
+		return;
+
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);

This field behavior is determined by PCIE spec and toggling is not required to enable MSIX. Also, FLR will clear this field on PF, so the ENABLE flag check to see if MSIX is already enabled doesn't make sense for PF case.

From the code logic, a rough guess is this code is trying to reset the VF's config space field in PF after VF FLR and enable MSI-X back. If that is the case, make an explicit check so that this is only done on VF devices.

Thanks,
Lijo

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev) · line 577

Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +2 -0
+}
  /**
   * amdgpu_irq_init - initialize interrupt handling
   *
@@ -558,6 +571,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev)
  {
  	int i, j, k;
 
+	amdgpu_restore_msix(adev);
  	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
  		if (!adev->irq.client[i].sources)
  			continue;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

[AMD Official Use Only]

Adding VF check and static for function, patch sent out as v5.

----------------------------------------------------------------------
BW
Pengju Zhou

Quoted 61 lines

-----Original Message-----
From: Lazar, Lijo <[email protected]>
Sent: Monday, July 5, 2021 1:52 PM
To: Zhou, Peng Ju <[email protected]>; [email protected]
Cc: Deng, Emily <[email protected]>
Subject: Re: [PATCH v4] drm/amdgpu: Restore msix after FLR

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device

Quoted 16 lines · Peng Ju Zhou

From: "Emily.Deng" <[email protected]>

After FLR, the msix will be cleared, so need to re-enable it.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 14 ++++++++++++++
  1 file changed, 14 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +0 -0
index 90f50561b43a..034420c38352 100644
@@ -277,6 +277,19 @@ static bool amdgpu_msi_ok(struct amdgpu_device

*adev)

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev)

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +4 -0
  	return true;
  }
 
+void amdgpu_restore_msix(struct amdgpu_device *adev) {
+	u16 ctrl;
+
+	pci_read_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, &ctrl);

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device *adev)

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +5 -0
+	if (!(ctrl & PCI_MSIX_FLAGS_ENABLE))
+		return;
+
+	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap +

PCI_MSIX_FLAGS, ctrl);

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c static bool amdgpu_msi_ok(struct amdgpu_device · line 291

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +3 -0
+	ctrl |= PCI_MSIX_FLAGS_ENABLE;
+	pci_write_config_word(adev->pdev, adev->pdev->msix_cap +
+PCI_MSIX_FLAGS, ctrl);

This field behavior is determined by PCIE spec and toggling is not required to enable MSIX. Also, FLR will clear this field on PF, so the ENABLE flag check to see if MSIX is already enabled doesn't make sense for PF case.

From the code logic, a rough guess is this code is trying to reset the VF's config
space field in PF after VF FLR and enable MSI-X back. If that is the case, make an explicit check so that this is only done on VF devices.

Thanks,
Lijo

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c void amdgpu_irq_gpu_reset_resume_helper(struct

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -0
+}
  /**
   * amdgpu_irq_init - initialize interrupt handling
   *
@@ -558,6 +571,7 @@ void amdgpu_irq_gpu_reset_resume_helper(struct

amdgpu_device *adev)

Quoted 8 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c void amdgpu_irq_gpu_reset_resume_helper(struct amdgpu_device *adev) · line 577

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -0
  {
  	int i, j, k;
 
+	amdgpu_restore_msix(adev);
  	for (i = 0; i < AMDGPU_IRQ_CLIENTID_MAX; ++i) {
  		if (!adev->irq.client[i].sources)
  			continue;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx