Skip to contentSkip to conversation

drm/amdgpu/gmc: use PCI BARs for APUs in passthrough

PATCH In review Alex Deucher 4 participants 8 messages Mar 10, 2022 – Mar 16, 2022

Subsystems (2)
lore
Share & tools
lore

Alex Deucher

author maintainer
#0 lore raw

PATCH 1/1 drm/amdgpu/gmc: use PCI BARs for APUs in passthrough

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

---
 drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c  | 5 +++--
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c  | 2 +-
 drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c  | 2 +-
 4 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +1 -1
index f60b7bd4dbf5..9887a60e8eae 100644
@@ -814,7 +814,7 @@ static int gmc_v10_0_mc_init(struct amdgpu_device *adev)
 	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
 #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU) {
+	if ((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) {
 		adev->gmc.aper_base = adev->gfxhub.funcs->get_mc_fb_offset(adev);
 		adev->gmc.aper_size = adev->gmc.real_vram_size;
 	}
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +3 -2
index 344d819b4c1b..979da6f510e8 100644
@@ -381,8 +381,9 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
 	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
 #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU &&
-	    adev->gmc.real_vram_size > adev->gmc.aper_size) {
+	if ((adev->flags & AMD_IS_APU) &&
+	    adev->gmc.real_vram_size > adev->gmc.aper_size &&
+	    !amdgpu_passthrough(adev)) {
 		adev->gmc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
 		adev->gmc.aper_size = adev->gmc.real_vram_size;
 	}
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +1 -1
index ca9841d5669f..1932a3e4af7e 100644
@@ -581,7 +581,7 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
 	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
 #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU) {
+	if ((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) {
 		adev->gmc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
 		adev->gmc.aper_size = adev->gmc.real_vram_size;
 	}
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +1 -1
index 431742eb7811..5fa613d41799 100644
@@ -1456,7 +1456,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
 	 */
 
 	/* check whether both host-gpu and gpu-gpu xgmi links exist */
-	if ((adev->flags & AMD_IS_APU) ||
+	if (((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) ||
 	    (adev->gmc.xgmi.supported &&
 	     adev->gmc.xgmi.connected_to_cpu)) {
 		adev->gmc.aper_base =
Signature · 1 line
2.35.1
#1 lore raw
Quoted 3 lines · Alex Deucher

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

Also, I wonder if BAR resizing could work with APUs?

Signature · 2 lines
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer
#2 lore raw

Am 11.03.22 um 11:24 schrieb Michel Dänzer:

Quoted 8 lines · Alex Deucher
Quoted 3 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

Also, I wonder if BAR resizing could work with APUs?

At least trivial to implement.

The host had to resize the BAR for the client or otherwise the client could easily bring down the system.

Christian.

#3 lore raw
Quoted 6 lines · Christian König

Am 11.03.22 um 11:24 schrieb Michel Dänzer:

Quoted 5 lines · Alex Deucher
Quoted 3 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out.  The physical address is not valid in a guest.

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?

Quoted 5 lines
Quoted 1 line

Also, I wonder if BAR resizing could work with APUs?

At least trivial to implement.

The host had to resize the BAR for the client or otherwise the client could easily bring down the system.

Gotcha, makes sense.

Signature · 2 lines
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer
#4 lore raw
Quoted 9 lines · Michel Dänzer
Quoted 6 lines · Christian König

Am 11.03.22 um 11:24 schrieb Michel Dänzer:

Quoted 5 lines · Alex Deucher
Quoted 3 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out.  The physical address is not valid in a guest.

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?

I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?

Signature · 2 lines
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer
#5 lore raw
Quoted 5 lines · Michel Dänzer
Quoted 3 lines · Michel Dänzer
Quoted 2 lines · Christian König

Am 11.03.22 um 11:24 schrieb Michel Dänzer:

Quoted 1 line · Alex Deucher
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?

I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?

Yes, it is needed. I added it in v2 of the patch.

Alex

Quoted 5 lines

--
Earthling Michel Dänzer | https://redhat.com
Libre software enthusiast | Mesa and Xwayland developer

#6 lore raw
Quoted 10 lines · Alex Deucher
Quoted 8 lines · Michel Dänzer
Quoted 6 lines · Michel Dänzer
Quoted 5 lines · Christian König

Am 11.03.22 um 11:24 schrieb Michel Dänzer:

Quoted 4 lines · Alex Deucher
Quoted 3 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.

I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?

I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?

Yes, it is needed. I added it in v2 of the patch.

Ah, great, hadn't seen the v2 patch yet. :)

Signature · 2 lines
Earthling Michel Dänzer            |                  https://redhat.com
Libre software enthusiast          |         Mesa and Xwayland developer
Reviewed-by on v1 1/1
#7 lore raw
In reply to Alex Deucher

Am 10.03.22 um 19:06 schrieb Alex Deucher:

Quoted 5 lines

If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.

Quoted 62 lines
---
  drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +-
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c  | 5 +++--
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c  | 2 +-
  drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c  | 2 +-
  4 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +1 -1
index f60b7bd4dbf5..9887a60e8eae 100644
@@ -814,7 +814,7 @@ static int gmc_v10_0_mc_init(struct amdgpu_device *adev)
  	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
  #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU) {
+	if ((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) {
  		adev->gmc.aper_base = adev->gfxhub.funcs->get_mc_fb_offset(adev);
  		adev->gmc.aper_size = adev->gmc.real_vram_size;
  	}
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +3 -2
index 344d819b4c1b..979da6f510e8 100644
@@ -381,8 +381,9 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev)
  	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
  #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU &&
-	    adev->gmc.real_vram_size > adev->gmc.aper_size) {
+	if ((adev->flags & AMD_IS_APU) &&
+	    adev->gmc.real_vram_size > adev->gmc.aper_size &&
+	    !amdgpu_passthrough(adev)) {
  		adev->gmc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
  		adev->gmc.aper_size = adev->gmc.real_vram_size;
  	}
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +1 -1
index ca9841d5669f..1932a3e4af7e 100644
@@ -581,7 +581,7 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev)
  	adev->gmc.aper_size = pci_resource_len(adev->pdev, 0);
 
  #ifdef CONFIG_X86_64
-	if (adev->flags & AMD_IS_APU) {
+	if ((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) {
  		adev->gmc.aper_base = ((u64)RREG32(mmMC_VM_FB_OFFSET)) << 22;
  		adev->gmc.aper_size = adev->gmc.real_vram_size;
  	}
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +1 -1
index 431742eb7811..5fa613d41799 100644
@@ -1456,7 +1456,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)
  	 */
 
  	/* check whether both host-gpu and gpu-gpu xgmi links exist */
-	if ((adev->flags & AMD_IS_APU) ||
+	if (((adev->flags & AMD_IS_APU) && !amdgpu_passthrough(adev)) ||
  	    (adev->gmc.xgmi.supported &&
  	     adev->gmc.xgmi.connected_to_cpu)) {
  		adev->gmc.aper_base =