Skip to contentSkip to conversation

drm/amdgpu: access stolen VRAM directly on KV/KB

PATCH In review Christian König 5 participants 12 messages Nov 7, 2016 – Nov 8, 2016 9 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 drm/amdgpu: access stolen VRAM directly on KV/KB

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +8 -1
index 3a25f72..70585b1 100644
@@ -375,9 +375,16 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+	if (adev->flags & AMD_IS_APU) {
+		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+		adev->mc.aper_size = adev->mc.real_vram_size;
+	}
+#endif
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1478510481-1768-1-git-send-email-deathsimple-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#2 lore raw

PATCH 2/2 drm/amdgpu: access stolen VRAM directly on CZ

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +8 -1
index f7372d3..0323f5d 100644
@@ -462,9 +462,16 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+	if (adev->flags & AMD_IS_APU) {
+		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+		adev->mc.aper_size = adev->mc.real_vram_size;
+	}
+#endif
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1478510481-1768-2-git-send-email-deathsimple-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#4 lore raw

Re: [PATCH 2/2] drm/amdgpu: access stolen VRAM directly on CZ

Quoted 4 lines · Christian König

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

I'm feeling coherency issues, has this approach been validated with the hw team?

Dave.

Quoted 31 lines
---
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +8 -2
index f7372d3..0323f5d 100644
@@ -462,9 +462,16 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
        /* size in MB on si */
        adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
        adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-       adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+       if (adev->flags & AMD_IS_APU) {
+               adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+               adev->mc.aper_size = adev->mc.real_vram_size;
+       }
+#endif
 
        /* In case the PCI BAR is larger than the actual amount of vram */
+       adev->mc.visible_vram_size = adev->mc.aper_size;
        if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
                adev->mc.visible_vram_size = adev->mc.real_vram_size;
 
--

2.7.4

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CAPM=9twmcyE4fOzZXJexdfViNx32vkc39sciQ3Ha+7UhetJJKA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#6 lore raw

Re: [PATCH 2/2] drm/amdgpu: access stolen VRAM directly on CZ

Quoted 7 lines · Dave Airlie
Quoted 4 lines · Christian König

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

I'm feeling coherency issues, has this approach been validated with the hw team?

I raised this concern internally as well, but the response indicated that it should be fine. Apparently x86 defines that write-combined writes complete before any access to UC memory (such as writing to an MMIO register or doorbell to submit a GPU operation).

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Reviewed-by
#7 lore raw
In reply to a message not in the archive
Quoted 6 lines · Christian König

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

The series is

We could also drop the HDP flushes / invalidates in this case, right?

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 9d218698-d588-090a-eb47-d0142cfdb623-otUistvHUpPR7s880joybQ@public.gmane.org

#9 lore raw

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 6 lines · Christian König
Quoted 5 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

Quoted 7 lines
Quoted 1 line

The series is

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on APUs.

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7be5b56d-fae3-d71b-485f-01227172859d-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#11 lore raw
Quoted 10 lines · Christian König

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 9 lines · Christian König
Quoted 6 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

The series is

Quoted 8 lines
Quoted 4 lines

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on APUs.

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

My assumption is that since CPU access to VRAM doesn't go through the HDP at all with this change, HDP flushes / invalidates can't have any useful effect.

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 8372afd6-8a51-1102-99a3-28a719088fa6-otUistvHUpPR7s880joybQ@public.gmane.org

#13 lore raw

Am 08.11.2016 um 08:31 schrieb Michel Dänzer:

Quoted 14 lines · Christian König
Quoted 13 lines

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 12 lines · Christian König
Quoted 6 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

The series is

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on APUs.

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

My assumption is that since CPU access to VRAM doesn't go through the HDP at all with this change, HDP flushes / invalidates can't have any useful effect.

Well writes to GTT still use the HDP if I understand it correctly.

So we still need to handle either flushes or invalidations (I doesn't of hand remember which was that).

Regards,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): f04a0ad6-e2cb-98c9-cb09-8af3218b9dca-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

#15 lore raw
Quoted 20 lines · Christian König

Am 08.11.2016 um 08:31 schrieb Michel Dänzer:

Quoted 19 lines · Christian König
Quoted 18 lines

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 6 lines · Christian König
Quoted 5 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

Quoted 7 lines
Quoted 1 line

The series is

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on APUs.

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

Quoted 8 lines
Quoted 3 lines

My assumption is that since CPU access to VRAM doesn't go through the HDP at all with this change, HDP flushes / invalidates can't have any useful effect.

Well writes to GTT still use the HDP if I understand it correctly.

So we still need to handle either flushes or invalidations (I doesn't of hand remember which was that).

Do you mean GPU access to GTT? If so, not sure why it would be writes only?

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 930d7e26-a894-c119-e5ec-b6dc1dae1ea4-otUistvHUpPR7s880joybQ@public.gmane.org

#17 lore raw
Quoted 35 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Michel Dänzer
Sent: Tuesday, November 08, 2016 4:37 AM
To: Christian König
Cc: [email protected]
Subject: Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

Quoted 17 lines · Christian König

Am 08.11.2016 um 08:31 schrieb Michel Dänzer:

Quoted 16 lines · Christian König
Quoted 15 lines

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 6 lines · Christian König
Quoted 5 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

Quoted 7 lines
Quoted 1 line

The series is

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on

APUs.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

My assumption is that since CPU access to VRAM doesn't go through the HDP at all with this change, HDP flushes / invalidates can't have any useful effect.

Quoted 6 lines
Quoted 4 lines

Well writes to GTT still use the HDP if I understand it correctly.

So we still need to handle either flushes or invalidations (I doesn't of hand remember which was that).

Do you mean GPU access to GTT? If so, not sure why it would be writes only?

The HDP has a read cache and a write queue. It only affects access via the BAR (either the CPU or another PCIe device). AFAIK, the APU access would only use the HDP if you mapped the BAR via the GART.

Alex

Quoted 9 lines

--
Earthling Michel Dänzer | http://www.amd.com
Libre software enthusiast | Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): MWHPR12MB16944B5018FC8CFF710EF3FFF7A60-Gy0DoCVfaSW4WA4dJ5YXGAdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

#19 lore raw

Am 08.11.2016 um 14:36 schrieb Deucher, Alexander:

Quoted 33 lines
Quoted 33 lines

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Michel Dänzer
Sent: Tuesday, November 08, 2016 4:37 AM
To: Christian König
Cc: [email protected]
Subject: Re: [PATCH 1/2] drm/amdgpu: access stolen VRAM directly on KV/KB

Quoted 17 lines · Christian König

Am 08.11.2016 um 08:31 schrieb Michel Dänzer:

Quoted 16 lines · Christian König
Quoted 15 lines

Am 07.11.2016 um 10:29 schrieb Michel Dänzer:

Quoted 6 lines · Christian König
Quoted 5 lines

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

Quoted 7 lines
Quoted 1 line

The series is

We could also drop the HDP flushes / invalidates in this case, right?

Could be, but I'm not 100% sure how those are actually wired up on

APUs.

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Might be that they are NOPs anyway, but could be that at least the flushes are still vital.

My assumption is that since CPU access to VRAM doesn't go through the HDP at all with this change, HDP flushes / invalidates can't have any useful effect.

Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

Well writes to GTT still use the HDP if I understand it correctly.

So we still need to handle either flushes or invalidations (I doesn't of hand remember which was that).

Do you mean GPU access to GTT? If so, not sure why it would be writes only?

The HDP has a read cache and a write queue. It only affects access via the BAR (either the CPU or another PCIe device). AFAIK, the APU access would only use the HDP if you mapped the BAR via the GART.

Ok, my understanding was that the HDP was responsible for both directions. E.g. writes/reads from the CPU side to the BAR as well as writes/reads from the GPU to system memory.

Since this needs intense testing anyway I will just wait for the first two patches to land, wait for a release to make sure they don't cause problems and then disable HDP flushes on APUs as well.

Christian.

Quoted 11 lines

Alex

Quoted 8 lines

--
Earthling Michel Dänzer | http://www.amd.com
Libre software enthusiast | Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#20 lore raw
In reply to a message not in the archive

As a next step, you could also remove HDP flushing on APUs.

Regards,
Felix

Quoted 22 lines · Christian König

From: Christian König <[email protected]>

We don't need to use the PCI BAR on APUs. This allows us to access the full VRAM directly without being limited by the BAR size.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +3 -1
index 3a25f72..70585b1 100644
@@ -375,9 +375,16 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 	/* size in MB on si */
 	adev->mc.mc_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
 	adev->mc.real_vram_size = RREG32(mmCONFIG_MEMSIZE) * 1024ULL * 1024ULL;
-	adev->mc.visible_vram_size = adev->mc.aper_size;
+
+#ifdef CONFIG_X86_64
+	if (adev->flags & AMD_IS_APU) {

drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c static int gmc_v7_0_mc_init(struct amdgpu_device *adev) · line 390

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +5 -0
+		adev->mc.aper_base = RREG32(mmMC_VM_FB_OFFSET) << 22ULL;
+		adev->mc.aper_size = adev->mc.real_vram_size;
+	}
+#endif
 
 	/* In case the PCI BAR is larger than the actual amount of vram */
+	adev->mc.visible_vram_size = adev->mc.aper_size;
 	if (adev->mc.visible_vram_size > adev->mc.real_vram_size)
 		adev->mc.visible_vram_size = adev->mc.real_vram_size;
 

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx