Skip to contentSkip to conversation

amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

PATCH In review Darren Salt 2 participants 7 messages Dec 10, 2020 – Dec 11, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 amdgpu: resize BAR0 to the maximum available size, even if it doesn't cover VRAM

This allows BAR0 resizing to be done for cards which don't advertise support for a size large enough to cover the VRAM but which do advertise at least one size larger than the default. For example, my RX 5600 XT, which advertises 256MB, 512MB and 1GB.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 44 +++++++++++++++++-----
 1 file changed, 35 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +35 -9
index 355fa0057c26..ec3610b4110b 100644
@@ -1078,6 +1078,11 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 		__clear_bit(wb, adev->wb.used);
 }
 
+static inline u32 bytes_to_size_pci(u64 bytes)
+{
+	return order_base_2(((bytes >> 20) | 1)) - 1;
+}
+
 /**
  * amdgpu_device_resize_fb_bar - try to resize FB BAR
  *
@@ -1090,20 +1095,25 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
 int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 {
 	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
-	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	u32 rbar_size = bytes_to_size_pci(space_needed);
 	struct pci_bus *root;
 	struct resource *res;
+	u64 current_bytes;
+	u32 current_size;
 	unsigned i;
 	u16 cmd;
 	int r;
+	bool nospc = false;
 
 	/* Bypass for VF */
 	if (amdgpu_sriov_vf(adev))
 		return 0;
 
-	/* skip if the bios has already enabled large BAR */
-	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	current_bytes = pci_resource_len(adev->pdev, 0);
+	current_size = bytes_to_size_pci(current_bytes);
+
+	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
+	if (current_size >= rbar_size)
 		return 0;
 
 	/* Check if the root BUS has 64bit memory resources */
@@ -1121,6 +1131,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 	if (!res)
 		return 0;
 
+	dev_dbg(adev->dev, "BIOS-allocated BAR0 was %lluMB; trying to get %lluMB",
+	        current_bytes >> 20, space_needed >> 20);
+
 	/* Disable memory decoding while we change the BAR addresses and size */
 	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
 	pci_write_config_word(adev->pdev, PCI_COMMAND,
@@ -1133,11 +1146,24 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
 	pci_release_resource(adev->pdev, 0);
 
-	r = pci_resize_resource(adev->pdev, 0, rbar_size);
-	if (r == -ENOSPC)
-		DRM_INFO("Not enough PCI address space for a large BAR.");
-	else if (r && r != -ENOTSUPP)
-		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+	while (rbar_size > current_size) {
+		r = pci_resize_resource(adev->pdev, 0, rbar_size);
+		if (r == 0 || r == -ENOTSUPP) {
+			break;
+		} else if (r == -ENOSPC) {
+			if (!nospc) {
+				/* Warn only the first time */
+				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
+				nospc = true;
+			}
+			--rbar_size;
+		} else if (r == -EINVAL) {
+			--rbar_size;
+		} else {
+			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
+			break;
+		}
+	}
 
 	pci_assign_unassigned_bus_resources(adev->pdev->bus);
 
Signature · 6 lines
2.20.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#1 lore raw

Am 10.12.20 um 01:57 schrieb Darren Salt:

Quoted 4 lines

This allows BAR0 resizing to be done for cards which don't advertise support for a size large enough to cover the VRAM but which do advertise at least one size larger than the default. For example, my RX 5600 XT, which advertises 256MB, 512MB and 1GB.

I've never seen such a configuration except for engineering samples. Can you send me a dump of the relevant PCI configuration space?

In general we could do this, but instead of just blindly trying different values we should just pick a supported one in the first place.

Regards,
Christian.

Quoted 82 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 44 +++++++++++++++++-----
  1 file changed, 35 insertions(+), 9 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +28 -9
index 355fa0057c26..ec3610b4110b 100644
@@ -1078,6 +1078,11 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
  		__clear_bit(wb, adev->wb.used);
  }
 
+static inline u32 bytes_to_size_pci(u64 bytes)
+{
+	return order_base_2(((bytes >> 20) | 1)) - 1;
+}
+
  /**
   * amdgpu_device_resize_fb_bar - try to resize FB BAR
   *
@@ -1090,20 +1095,25 @@ void amdgpu_device_wb_free(struct amdgpu_device *adev, u32 wb)
  int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
  {
  	u64 space_needed = roundup_pow_of_two(adev->gmc.real_vram_size);
-	u32 rbar_size = order_base_2(((space_needed >> 20) | 1)) - 1;
+	u32 rbar_size = bytes_to_size_pci(space_needed);
  	struct pci_bus *root;
  	struct resource *res;
+	u64 current_bytes;
+	u32 current_size;
  	unsigned i;
  	u16 cmd;
  	int r;
+	bool nospc = false;
 
  	/* Bypass for VF */
  	if (amdgpu_sriov_vf(adev))
  		return 0;
 
-	/* skip if the bios has already enabled large BAR */
-	if (adev->gmc.real_vram_size &&
-	    (pci_resource_len(adev->pdev, 0) >= adev->gmc.real_vram_size))
+	current_bytes = pci_resource_len(adev->pdev, 0);
+	current_size = bytes_to_size_pci(current_bytes);
+
+	/* Skip if the BIOS has already enabled large BAR, covering the VRAM */
+	if (current_size >= rbar_size)
  		return 0;
 
  	/* Check if the root BUS has 64bit memory resources */
@@ -1121,6 +1131,9 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
  	if (!res)
  		return 0;
 
+	dev_dbg(adev->dev, "BIOS-allocated BAR0 was %lluMB; trying to get %lluMB",
+	        current_bytes >> 20, space_needed >> 20);
+
  	/* Disable memory decoding while we change the BAR addresses and size */
  	pci_read_config_word(adev->pdev, PCI_COMMAND, &cmd);
  	pci_write_config_word(adev->pdev, PCI_COMMAND,
@@ -1133,11 +1146,24 @@ int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev)
 
  	pci_release_resource(adev->pdev, 0);
 
-	r = pci_resize_resource(adev->pdev, 0, rbar_size);
-	if (r == -ENOSPC)
-		DRM_INFO("Not enough PCI address space for a large BAR.");
-	else if (r && r != -ENOTSUPP)
-		DRM_ERROR("Problem resizing BAR0 (%d).", r);
+	while (rbar_size > current_size) {
+		r = pci_resize_resource(adev->pdev, 0, rbar_size);
+		if (r == 0 || r == -ENOTSUPP) {
+			break;
+		} else if (r == -ENOSPC) {
+			if (!nospc) {
+				/* Warn only the first time */
+				dev_info(adev->dev, "Not enough PCI address space for a large BAR.");
+				nospc = true;
+			}
+			--rbar_size;

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_device_resize_fb_bar(struct amdgpu_device *adev) · line 1169

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +7 -0
+		} else if (r == -EINVAL) {
+			--rbar_size;
+		} else {
+			dev_err(adev->dev, "Problem resizing BAR0 (%d).", r);
+			break;
+		}
+	}
 
  	pci_assign_unassigned_bus_resources(adev->pdev->bus);
 

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw

I demand that Christian König may or may not have written...

Quoted 8 lines

Am 10.12.20 um 01:57 schrieb Darren Salt:

Quoted 4 lines

This allows BAR0 resizing to be done for cards which don't advertise support for a size large enough to cover the VRAM but which do advertise at least one size larger than the default. For example, my RX 5600 XT, which advertises 256MB, 512MB and 1GB.

I've never seen such a configuration except for engineering samples. Can you send me a dump of the relevant PCI configuration space?

“lspci -nn -v -xxxx” output is attached. (Sapphire RX 5600 XT Pulse; not an early one.)

My current kernel has another patch, applied on top of this patch, which allows ignoring the size list. As such, that BAR is currently 8GB instead of the 1GB which it should be. I've not noticed any significant problems as yet.

If the card should be advertising larger sizes too then its VBIOS needs fixing; but as a lot of these already out there won't get that fix, some sort of override (quirk, I expect, with a module option for cards not covered) would, I think, be warranted.

Quoted 2 lines

In general we could do this, but instead of just blindly trying different values we should just pick a supported one in the first place.

By using pci_rebar_get_possible_sizes() etc.? That looks reasonable to me. It'll also require some patching in the PCI subsystem to expose relevant functions.

28:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:731f] (rev ca) (prog-if 00 [VGA controller])
Subsystem: Sapphire Technology Limited Device [1da2:e416]
Flags: bus master, fast devsel, latency 0, IRQ 41
Memory at 600000000 (64-bit, prefetchable) [size=8G]
Memory at 500000000 (64-bit, prefetchable) [size=256M]
I/O ports at e000 [size=256]
Memory at fcb00000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at fcb80000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] #15
Capabilities: [240] Power Budgeting <?>
Capabilities: [270] #19
Capabilities: [2a0] Access Control Services
Capabilities: [2b0] Address Translation Service (ATS)
Capabilities: [2c0] Page Request Interface (PRI)
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting
Capabilities: [400] #25
Capabilities: [410] #26
Capabilities: [440] #27
Kernel driver in use: amdgpu
00: 02 10 1f 73 07 04 10 00 ca 00 00 03 10 00 80 00
10: 0c 00 00 00 06 00 00 00 0c 00 00 00 05 00 00 00
20: 01 e0 00 00 00 00 b0 fc 00 00 00 00 a2 1d 16 e4
30: 00 00 b8 fc 48 00 00 00 00 00 00 00 ff 01 00 00
40: 00 00 00 00 00 00 00 00 09 50 08 00 a2 1d 16 e4
50: 01 64 03 f0 08 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 10 a0 12 00 a1 8f 00 00 30 29 09 00
70: 04 0d 40 00 40 00 04 11 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 9f 09 73 00 40 04 00 00
90: 1e 00 80 01 04 00 1f 00 00 00 00 00 00 00 00 00
a0: 05 00 81 00 00 40 e0 fe 00 00 00 00 22 40 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 0b 00 01 15 01 00 01 01 00 00 00 00 00 00 00 00
110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
150: 01 00 02 20 00 00 00 00 00 00 00 00 30 20 46 00
160: 00 20 00 00 00 20 00 00 a0 00 00 00 00 00 00 00
170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 15 00 01 24 00 70 00 00 40 0d 00 00 e0 1f 00 00
210: 02 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 04 00 01 27 00 00 00 00 00 81 07 00 00 00 00 00
250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
270: 19 00 01 2a 00 00 00 00 00 00 00 00 00 7f 00 7f
280: 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f
290: 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f 00 00 00 00
2a0: 0d 00 01 2b 00 00 00 00 00 00 00 00 00 00 00 00
2b0: 0f 00 01 2c 20 00 00 80 00 00 00 00 00 00 00 00
2c0: 13 00 01 2d 00 00 00 01 00 01 00 00 00 00 00 00
2d0: 1b 00 01 32 06 10 00 00 00 00 00 00 00 00 00 00
2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
310: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
320: 18 00 01 40 01 10 01 10 00 00 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 25 00 01 41 01 00 00 80 01 00 00 80 00 00 00 00
410: 26 00 01 44 00 00 00 00 00 00 00 00 0f 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0 f0
440: 27 00 01 00 00 00 00 00 38 9c 00 00 38 9c 00 00
450: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
460: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
470: 38 9c 00 00 38 9c 00 00 38 9c 00 00 38 9c 00 00
480: 38 9c 00 00 38 9c 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
720: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
730: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
750: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
890: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw

Am 10.12.20 um 14:59 schrieb Darren Salt:

Quoted 11 lines

I demand that Christian König may or may not have written...

Quoted 7 lines

Am 10.12.20 um 01:57 schrieb Darren Salt:

Quoted 4 lines

This allows BAR0 resizing to be done for cards which don't advertise support for a size large enough to cover the VRAM but which do advertise at least one size larger than the default. For example, my RX 5600 XT, which advertises 256MB, 512MB and 1GB.

I've never seen such a configuration except for engineering samples. Can you send me a dump of the relevant PCI configuration space?

“lspci -nn -v -xxxx” output is attached. (Sapphire RX 5600 XT Pulse; not an early one.)

Thanks. Going to double check tomorrow.

Quoted 4 lines

My current kernel has another patch, applied on top of this patch, which allows ignoring the size list. As such, that BAR is currently 8GB instead of the 1GB which it should be. I've not noticed any significant problems as yet.

Please grab umr, take a look at the amdgpu_vram_mm debugfs file and see if you can get some bytes from a buffer at the end of VRAM.

If that doesn't return 0x0 or 0xffffffff then it is probably working quite fine.

Quoted 4 lines

If the card should be advertising larger sizes too then its VBIOS needs fixing; but as a lot of these already out there won't get that fix, some sort of override (quirk, I expect, with a module option for cards not covered) would, I think, be warranted.

I'm really wondering what the heck is going on here. I've heard from boards which don't have resizeable BARs, but that there should be an artificial 1GB limit sounds strongly like a VBIOS bug to me.

Anyway I agree that a PCI subsystem quirk might be appropriated.

Quoted 3 lines
Quoted 2 lines

In general we could do this, but instead of just blindly trying different values we should just pick a supported one in the first place.

By using pci_rebar_get_possible_sizes() etc.? That looks reasonable to me.

Yes, exactly.

Quoted 2 lines

It'll also require some patching in the PCI subsystem to expose relevant functions.

Just send that to me as a complete and clean patchset.

I'm the one who added the code in the first place and I have no problem arguing with Bjorn why we need that in a driver now.

Thanks,
Christian.

Quoted 6 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw

I demand that Christian König may or may not have written...

Quoted 3 lines

Am 10.12.20 um 14:59 schrieb Darren Salt:

Quoted 2 lines

I demand that Christian König may or may not have written...

[snip]

Quoted 10 lines
Quoted 4 lines

My current kernel has another patch, applied on top of this patch, which allows ignoring the size list. As such, that BAR is currently 8GB instead of the 1GB which it should be. I've not noticed any significant problems as yet.

Please grab umr, take a look at the amdgpu_vram_mm debugfs file and see if you can get some bytes from a buffer at the end of VRAM.

If that doesn't return 0x0 or 0xffffffff then it is probably working quite fine.

I used dd:

  # dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 count=1 skip=6127 | hexdump -C |tail
  1+0 records in
  1+0 records out
  1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696037 s, 15.1 MB/s
  *
  000ff800  ca 53 ca 53 ca 53 ca 53  ca 53 ca 53 ca 53 ca 53  |.S.S.S.S.S.S.S.S|
  *
  000ffa00  c6 63 c6 63 c6 63 c6 63  c6 63 c6 63 c6 63 c6 63  |.c.c.c.c.c.c.c.c|
  *
  000ffe00  ca 59 ca 59 ca 59 ca 59  ca 59 ca 59 ca 59 ca 59  |.Y.Y.Y.Y.Y.Y.Y.Y|
  *
  000fff00  9a 53 9a 53 9a 53 9a 53  9a 53 9a 53 9a 53 9a 53  |.S.S.S.S.S.S.S.S|
  *
  00100000

[snip; VBIOS BAR size limit of 1GB]

Quoted 1 line

Anyway I agree that a PCI subsystem quirk might be appropriated.

My current patch set implements an override, which is definitely useful for testing. As for adding a quirk – I've not looked into how best to do this yet.

[snip]

Quoted 4 lines
Quoted 2 lines

It'll also require some patching in the PCI subsystem to expose relevant functions.

Just send that to me as a complete and clean patchset.

Done, though only to the list.

Quoted 2 lines

I'm the one who added the code in the first place and I have no problem arguing with Bjorn why we need that in a driver now.

That's definitely helpful! ☺
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw

Am 11.12.20 um 02:42 schrieb Darren Salt:

Quoted 1 line

I demand that Christian König may or may not have written...

[SNIP]

Well I did wrote that :)

Quoted 1 line

I used dd: # dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 count=1 skip=6127 | hexdump -C |tail

That won't work. amdgpu_vram uses a MMIO register pair to access VRAM which works even when it isn't CPU visible.

Thinking more about it umr would probably use this as well, so that won't work either.

You could try to use dd on /dev/mem with the offset of the BAR.

Quoted 1 line

Anyway I agree that a PCI subsystem quirk might be appropriated.

I'm going to discuss AMD internally why you have such strange values in the RBAR registers.

Quoted 2 lines
Quoted 1 line

Just send that to me as a complete and clean patchset.

Done, though only to the list.

I have a few comments on the patches. They can use some polishing, but in general the approach looks solid to me.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw

I demand that Christian König may or may not have written...

Quoted 5 lines

Am 11.12.20 um 02:42 schrieb Darren Salt:

Quoted 1 line

I demand that Christian König may or may not have written...

[SNIP]

Well I did wrote that :)

“did write”, surely…

Quoted 10 lines
Quoted 2 lines

I used dd:

# dd if=/sys/kernel/debug/dri/0/amdgpu_vram bs=1048576 count=1 skip=6127 | hexdump -C |tail

That won't work. amdgpu_vram uses a MMIO register pair to access VRAM which works even when it isn't CPU visible.

Thinking more about it umr would probably use this as well, so that won't work either.

You could try to use dd on /dev/mem with the offset of the BAR.

Looks like that's RAM accessed by physical address, so that won't work either. And I do see dd reporting ‘bad address’.

Quoted 4 lines
Quoted 1 line

Anyway I agree that a PCI subsystem quirk might be appropriated.

I'm going to discuss AMD internally why you have such strange values in the RBAR registers.

I'm thinking probably an error by somebody at Sapphire, but we'll see…

Hopefully, that'll sort it out, at least for new cards. I doubt that mine's the only one like this, and it seems likely that most already out there won't be updated (shoudl there be new VBIOS releases as a reault).

Anyway, I have a quirk patch written now – untested as yet, and probably going to be changed due to other changes before I do test it.

[snip]
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx