Skip to contentSkip to conversation

drm/amdgpu: resize VRAM BAR for CPU access on gfx10

PATCH In review Alan Swanson 3 participants 5 messages May 21, 2020 – May 22, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 drm/amdgpu: resize VRAM BAR for CPU access on gfx10

Try to resize BAR0 to let CPU access all of VRAM on Navi. Syncs code with previous gfx generations from commit d6895ad39f3b39 ("drm/amdgpu: resize VRAM BAR for CPU access v6").

---
Unfortunately cannot test this with my RX5700 on my AMD X470 motherboard as its BIOS "Above 4G decoding" option does not remap the PCI root bus. Trying "pci=nocrs" also unsuccessful. However, I assume it's just absent due to missed commit overlap during gfx10 bring up and should work.

 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +10 -4
index edaa50d85..ba2b7ac0c 100644
@@ -686,17 +686,23 @@ static void gmc_v10_0_vram_gtt_location(struct amdgpu_device *adev,
  */
 static int gmc_v10_0_mc_init(struct amdgpu_device *adev)
 {
-	/* Could aper size report 0 ? */
-	adev->gmc.aper_base = pci_resource_start(adev->pdev, 0);
-	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
+	int r;
 
 	/* size in MB on si */
 	adev->gmc.mc_vram_size =
 		adev->nbio.funcs->get_memsize(adev) * 1024ULL * 1024ULL;
 	adev->gmc.real_vram_size = adev->gmc.mc_vram_size;
-	adev->gmc.visible_vram_size = adev->gmc.aper_size;
+
+	if (!(adev->flags & AMD_IS_APU)) {
+		r = amdgpu_device_resize_fb_bar(adev);
+		if (r)
+			return r;
+	}
+	adev->gmc.aper_base = pci_resource_start(adev->pdev, 0);
+	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->gmc.visible_vram_size = adev->gmc.aper_size;
 	if (adev->gmc.visible_vram_size > adev->gmc.real_vram_size)
 		adev->gmc.visible_vram_size = adev->gmc.real_vram_size;
 
Signature · 6 lines
2.26.2

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

PATCH 2/2 drm/amdgpu: Advise if unable to resize BAR

Even with the "Above 4G decoding" (or similar) BIOS option enabled, many BIOS do not assign the PCI root bus a 64-bit address space.

If available, "MMIOH Base" and "MMIO High Size" (or similar) BIOS options should allow mapping to the desired address spaces.

---
Useful to know why bar resizing isn't happening.

 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -1
index 2f0e8da7b..39a7f7212 100644
@@ -919,8 +919,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	}
 
 	/* Trying to resize is pointless without a root hub window above 4GB */
-	if (!res)
+	if (!res) {
+		DRM_INFO("Unable to resize BAR as PCI bus address space below 4GB.");
 		return 0;
+	}
 
 	/* Disable memory decoding while we change the BAR addresses and size */
 	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
Signature · 6 lines
2.26.2

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#2 lore raw

Re: [PATCH 2/2] drm/amdgpu: Advise if unable to resize BAR

Quoted 10 lines · Alan Swanson

Even with the "Above 4G decoding" (or similar) BIOS option enabled, many BIOS do not assign the PCI root bus a 64-bit address space.

If available, "MMIOH Base" and "MMIO High Size" (or similar) BIOS options should allow mapping to the desired address spaces.

---
Useful to know why bar resizing isn't happening.

This will spam a lot of people and probably cause confusion. I'd prefer to drop this one.

Alex

Quoted 23 lines
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -2
index 2f0e8da7b..39a7f7212 100644
@@ -919,8 +919,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
        }
 
        /* Trying to resize is pointless without a root hub window above 4GB */
-       if (!res)
+       if (!res) {
+               DRM_INFO("Unable to resize BAR as PCI bus address space below 4GB.");
                return 0;
+       }
 
        /* Disable memory decoding while we change the BAR addresses and size */
        pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
--

2.26.2

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

Re: [PATCH 2/2] drm/amdgpu: Advise if unable to resize BAR

Am 21.05.20 um 23:32 schrieb Alex Deucher:

Quoted 12 lines · Alan Swanson
Quoted 9 lines

Even with the "Above 4G decoding" (or similar) BIOS option enabled, many BIOS do not assign the PCI root bus a 64-bit address space.

If available, "MMIOH Base" and "MMIO High Size" (or similar) BIOS options should allow mapping to the desired address spaces.

---
Useful to know why bar resizing isn't happening.

This will spam a lot of people and probably cause confusion. I'd prefer to drop this one.

Agreed, you can just look at /proc/iomem to figure out if resources above 4GB are available or not.

Christian.

Quoted 26 lines

Alex

Quoted 22 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -2
index 2f0e8da7b..39a7f7212 100644
@@ -919,8 +919,10 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
         }
 
         /* Trying to resize is pointless without a root hub window above 4GB */
-       if (!res)
+       if (!res) {
+               DRM_INFO("Unable to resize BAR as PCI bus address space below 4GB.");
                 return 0;
+       }
 
         /* Disable memory decoding while we change the BAR addresses and size */
         pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
--

2.26.2

Quoted 8 lines
Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw
In reply to Alan Swanson
Quoted 6 lines · Alan Swanson

Try to resize BAR0 to let CPU access all of VRAM on Navi. Syncs code with previous gfx generations from commit d6895ad39f3b39 ("drm/amdgpu: resize VRAM BAR for CPU access v6").

Tested and applied. Thanks!

Alex

Quoted 45 lines

---
Unfortunately cannot test this with my RX5700 on my AMD X470 motherboard as its BIOS "Above 4G decoding" option does not remap the PCI root bus. Trying "pci=nocrs" also unsuccessful. However, I assume it's just absent due to missed commit overlap during gfx10 bring up and should work.

 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +10 -5
index edaa50d85..ba2b7ac0c 100644
@@ -686,17 +686,23 @@ static void gmc_v10_0_vram_gtt_location(struct amdgpu_device *adev,
  */
 static int gmc_v10_0_mc_init(struct amdgpu_device *adev)
 {
-       /* Could aper size report 0 ? */
-       adev->gmc.aper_base = pci_resource_start(adev->pdev, 0);
-       adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
+       int r;
 
        /* size in MB on si */
        adev->gmc.mc_vram_size =
                adev->nbio.funcs->get_memsize(adev) * 1024ULL * 1024ULL;
        adev->gmc.real_vram_size = adev->gmc.mc_vram_size;
-       adev->gmc.visible_vram_size = adev->gmc.aper_size;
+
+       if (!(adev->flags & AMD_IS_APU)) {
+               r = amdgpu_device_resize_fb_bar(adev);
+               if (r)
+                       return r;
+       }
+       adev->gmc.aper_base = pci_resource_start(adev->pdev, 0);
+       adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
        /* In case the PCI BAR is larger than the actual amount of vram */
+       adev->gmc.visible_vram_size = adev->gmc.aper_size;
        if (adev->gmc.visible_vram_size > adev->gmc.real_vram_size)
                adev->gmc.visible_vram_size = adev->gmc.real_vram_size;
 
--

2.26.2

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx