Skip to contentSkip to conversation

drm/amdgpu: Check resize bar register when system uses large bar

PATCH Superseded Ma Jun 5 participants 10 messages Dec 19, 2023 – Jan 9, 2024

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ma Jun

author
#0 lore raw

PATCH 1/1 drm/amdgpu: Check resize bar register when system uses large bar

Print a warnning message if the system can't access the resize bar register when using large bar.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 		__clear_bit(wb, adev->wb.used);
 }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
 /**
  * amdgpu_device_resize_fb_bar - try to resize FB BAR
  *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
 	/* skip if the bios has already enabled large BAR */
 	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
 		return 0;
+	}
 
 	/* Check if the root BUS has 64bit memory resources */
 	root = adev->pdev->bus;
Signature · 1 line
2.34.1
#1 lore raw

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Regards,
Christian.

Quoted 35 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
  		__clear_bit(wb, adev->wb.used);
  }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
  /**
   * amdgpu_device_resize_fb_bar - try to resize FB BAR
   *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
  	/* skip if the bios has already enabled large BAR */
  	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
  		return 0;
+	}
 
  	/* Check if the root BUS has 64bit memory resources */
  	root = adev->pdev->bus;
#2 lore raw

Hi Christian,

Quoted 9 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Regards,
Ma Jun

Quoted 39 lines

Regards,
Christian.

Quoted 35 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
  		__clear_bit(wb, adev->wb.used);
  }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
  /**
   * amdgpu_device_resize_fb_bar - try to resize FB BAR
   *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
  	/* skip if the bios has already enabled large BAR */
  	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
  		return 0;
+	}
 
  	/* Check if the root BUS has 64bit memory resources */
  	root = adev->pdev->bus;
#3 lore raw
Quoted 17 lines · Jun

Hi Christian,

Quoted 9 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

This doesn't yet have review comments with the holidays but I think this is a scalable solution to that specific issue:

https://lore.kernel.org/linux-pci/[email protected]/

Can you try on one of these affected systems and see that it helps?

Quoted 43 lines

Regards,
Ma Jun

Quoted 39 lines

Regards,
Christian.

Quoted 35 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
   		__clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
   	/* skip if the bios has already enabled large BAR */
   	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
   		return 0;
+	}
 
   	/* Check if the root BUS has 64bit memory resources */
   	root = adev->pdev->bus;
#4 lore raw
In reply to Ma, Jun

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

In this case this would print a warning even when there isn't anything wrong.

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

Regards,
Christian.

Quoted 41 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
   		__clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
   	/* skip if the bios has already enabled large BAR */
   	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
   		return 0;
+	}
 
   	/* Check if the root BUS has 64bit memory resources */
   	root = adev->pdev->bus;
#5 lore raw
Quoted 34 lines · Christian König

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

In this case this would print a warning even when there isn't anything wrong.

Quoted 6 lines

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

That system is probably similar to what Mario mentioned:
https://lore.kernel.org/linux-pci/[email protected]/

Alex

Quoted 46 lines

Regards,
Christian.

Quoted 41 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
            __clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+   if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+           DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
    /* skip if the bios has already enabled large BAR */
    if (adev->gmc.real_vram_size &&
-       (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+       (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+           amdgpu_find_rb_register(adev);
            return 0;
+   }
 
    /* Check if the root BUS has 64bit memory resources */
    root = adev->pdev->bus;
#6 lore raw

Hi Alex,

Quoted 39 lines · Alex Deucher
Quoted 37 lines · Christian König

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

In this case this would print a warning even when there isn't anything wrong.

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

Quoted 7 lines
Quoted 3 lines

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

That system is probably similar to what Mario mentioned:
https://lore.kernel.org/linux-pci/[email protected]/

Yes, It's the same problem.

Regards,
Ma Jun

Quoted 48 lines

Alex

Quoted 46 lines

Regards,
Christian.

Quoted 41 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
            __clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+   if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+           DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
    /* skip if the bios has already enabled large BAR */
    if (adev->gmc.real_vram_size &&
-       (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+       (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+           amdgpu_find_rb_register(adev);
            return 0;
+   }
 
    /* Check if the root BUS has 64bit memory resources */
    root = adev->pdev->bus;
#7 lore raw
In reply to Christian König

Hi Christian,

Quoted 30 lines · Christian König

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

Quoted 6 lines

In this case this would print a warning even when there isn't anything wrong.

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

Do you mean MSI-X? There are no extended capability registers related with MSI or MSI-x.

How about reading the 0x100 register in the extended config space since the extended capabilities always start from the offset 0x100 according the pcie spec.

Quoted 3 lines

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

I think the MSI related configuration registers are in the legacy configuration space. So the system don't need to use mmio to access these registers.

Regards,
Ma Jun

Quoted 46 lines

Regards,
Christian.

Quoted 41 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
   		__clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
   	/* skip if the bios has already enabled large BAR */
   	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
   		return 0;
+	}
 
   	/* Check if the root BUS has 64bit memory resources */
   	root = adev->pdev->bus;
#8 lore raw

Am 08.01.24 um 10:24 schrieb Ma, Jun:

Quoted 38 lines

Hi Christian,

Quoted 29 lines · Christian König

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

Quoted 6 lines

In this case this would print a warning even when there isn't anything wrong.

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

Quoted 6 lines

Do you mean MSI-X? There are no extended capability registers related with MSI or MSI-x.

How about reading the 0x100 register in the extended config space since the extended capabilities always start from the offset 0x100 according the pcie spec.

Yeah, that should work as well. At least some extension should be there in the extended config space.

Quoted 6 lines
Quoted 3 lines

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

I think the MSI related configuration registers are in the legacy configuration space. So the system don't need to use mmio to access these registers.

Ah, yes that could explain it.

Thanks,
Christian.

Quoted 47 lines

Regards,
Ma Jun

Quoted 43 lines

Regards,
Christian.

Quoted 40 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
    1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
    		__clear_bit(wb, adev->wb.used);
    }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
    /**
     * amdgpu_device_resize_fb_bar - try to resize FB BAR
     *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
    	/* skip if the bios has already enabled large BAR */
    	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
    		return 0;
+	}
 
    	/* Check if the root BUS has 64bit memory resources */
    	root = adev->pdev->bus;
#9 lore raw
Quoted 39 lines · Christian König

Am 08.01.24 um 10:24 schrieb Ma, Jun:

Quoted 38 lines

Hi Christian,

Quoted 35 lines · Christian König

Am 21.12.23 um 02:58 schrieb Ma, Jun:

Quoted 16 lines

Hi Christian,

Quoted 8 lines · Christian König

Am 19.12.23 um 06:58 schrieb Ma Jun:

Quoted 2 lines

Print a warnning message if the system can't access the resize bar register when using large bar.

Well pretty clear NAK, we have embedded use cases where this would trigger an incorrect warning.

What should that be good for in the first place?

Some customer platforms do not enable mmconfig for various reasons, such as bios bug, and therefore cannot access the GPU extend configuration space through mmio.

Quoted 8 lines

Therefore, when the system enters the d3cold state and resumes, the amdgpu driver fails to resume because the extend configuration space registers of GPU can't be restored. At this point, Usually we only see some failure dmesg log printed by amdgpu driver, it is difficult to find the root cause.

So I thought it would be helpful to print some warning messages at the beginning to identify problems quickly.

Interesting bug, but we can't do this here. We have a couple of devices where the REBAR cap isn't enabled for some reason (or not correctly enabled).

In this case this would print a warning even when there isn't anything wrong.

What we could potentially do is to check for the MSI extension, that should always be there if I'm not completely mistaken.

Quoted 9 lines
Quoted 6 lines

Do you mean MSI-X? There are no extended capability registers related with MSI or MSI-x.

How about reading the 0x100 register in the extended config space since the extended capabilities always start from the offset 0x100 according the pcie spec.

Yeah, that should work as well. At least some extension should be there in the extended config space.

Ok, I'll submit a new patch.

Regards,
Ma Jun

Quoted 59 lines
Quoted 6 lines
Quoted 3 lines

But how does this hardware platform even works without the extended mmio space? I mean we can't even enable/disable MSI in that configuration if I'm not completely mistaken.

I think the MSI related configuration registers are in the legacy configuration space. So the system don't need to use mmio to access these registers.

Ah, yes that could explain it.

Thanks,
Christian.

Quoted 46 lines

Regards,
Ma Jun

Quoted 43 lines

Regards,
Christian.

Quoted 40 lines

Regards,
Ma Jun

Quoted 37 lines

Regards,
Christian.

Quoted 34 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 +++++++++-
   1 file changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +9 -1
index 4b694696930e..e7aedb4bd66e 100644
@@ -1417,6 +1417,12 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
   		__clear_bit(wb, adev->wb.used);
   }
 
+static inline void amdgpu_find_rb_register(struct amdgpu_device *adev)
+{
+	if (!pci_find_ext_capability(adev->pdev, PCI_EXT_CAP_ID_REBAR))
+		DRM_WARN("System can't access the resize bar register,please check!!\n");
+}
+
   /**
    * amdgpu_device_resize_fb_bar - try to resize FB BAR
    *
@@ -1444,8 +1450,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
   	/* skip if the bios has already enabled large BAR */
   	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size)) {
+		amdgpu_find_rb_register(adev);
   		return 0;
+	}
 
   	/* Check if the root BUS has 64bit memory resources */
   	root = adev->pdev->bus;