drm/amdgpu/gmc: use PCI BARs for APUs in passthrough
Subsystems (2)
Alex Deucher
maintainerPATCH 1/1 drm/amdgpu/gmc: use PCI BARs for APUs in passthrough
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
--- drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 5 +++-- drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +1 -1
@@ -814,7 +814,7 @@ static int gmc_v10_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU ) { +if ((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) { adev->gmc.aper_base = adev->gfxhub.funcs->get_mc_fb_offset (adev); adev->gmc.aper_size = adev->gmc.real_vram_size; }
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +3 -2
@@ -381,8 +381,9 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU && - adev->gmc.real_vram_size > adev->gmc.aper_size) { +if ((adev->flags &AMD_IS_APU ) && + adev->gmc.real_vram_size > adev->gmc.aper_size && + !amdgpu_passthrough (adev)) { adev->gmc.aper_base = ((u64 )RREG32 (mmMC_VM_FB_OFFSET)) <<22 ; adev->gmc.aper_size = adev->gmc.real_vram_size; }
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +1 -1
@@ -581,7 +581,7 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU ) { +if ((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) { adev->gmc.aper_base = ((u64 )RREG32 (mmMC_VM_FB_OFFSET)) <<22 ; adev->gmc.aper_size = adev->gmc.real_vram_size; }
drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +1 -1
@@ -1456,7 +1456,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)*/ /* check whether both host-gpu and gpu-gpu xgmi links exist */ -if ((adev->flags &AMD_IS_APU ) || +if (((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) || (adev->gmc.xgmi.supported && adev->gmc.xgmi.connected_to_cpu)) { adev->gmc.aper_base =
Signature · 1 line
2.35.1
Quoted 3 lines · Alex Deucher
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
I think amdgpu_device_flush_hdp & amdgpu_device_invalidate_hdp need to be modified accordingly.
Also, I wonder if BAR resizing could work with APUs?
Signature · 2 lines
Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and Xwayland developer
Am 11.03.22 um 11:24 schrieb Michel Dänzer:
Quoted 8 lines · Alex Deucher
Quoted 3 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
I think
amdgpu_device_flush_hdp&amdgpu_device_invalidate_hdpneed to be modified accordingly.Also, I wonder if BAR resizing could work with APUs?
At least trivial to implement.
The host had to resize the BAR for the client or otherwise the client could easily bring down the system.
Christian.
Quoted 6 lines · Christian König
Am 11.03.22 um 11:24 schrieb Michel Dänzer:
Quoted 5 lines · Alex Deucher
Quoted 3 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
I think
amdgpu_device_flush_hdp&amdgpu_device_invalidate_hdpneed to be modified accordingly.
I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?
Quoted 5 lines
Quoted 1 line
Also, I wonder if BAR resizing could work with APUs?
At least trivial to implement.
The host had to resize the BAR for the client or otherwise the client could easily bring down the system.
Gotcha, makes sense.
Signature · 2 lines
Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and Xwayland developer
Quoted 9 lines · Michel Dänzer
Quoted 6 lines · Christian König
Am 11.03.22 um 11:24 schrieb Michel Dänzer:
Quoted 5 lines · Alex Deucher
Quoted 3 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
I think
amdgpu_device_flush_hdp&amdgpu_device_invalidate_hdpneed to be modified accordingly.I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?
I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?
Signature · 2 lines
Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and Xwayland developer
Quoted 5 lines · Michel Dänzer
Quoted 3 lines · Michel Dänzer
Quoted 2 lines · Christian König
Am 11.03.22 um 11:24 schrieb Michel Dänzer:
Quoted 1 line · Alex Deucher
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
I think
amdgpu_device_flush_hdp&amdgpu_device_invalidate_hdpneed to be modified accordingly.I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?
I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?
Yes, it is needed. I added it in v2 of the patch.
Alex
Quoted 5 lines
--
Earthling Michel Dänzer | https://redhat.com
Libre software enthusiast | Mesa and Xwayland developer
Quoted 10 lines · Alex Deucher
Quoted 8 lines · Michel Dänzer
Quoted 6 lines · Michel Dänzer
Quoted 5 lines · Christian König
Am 11.03.22 um 11:24 schrieb Michel Dänzer:
Quoted 4 lines · Alex Deucher
Quoted 3 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
I think
amdgpu_device_flush_hdp&amdgpu_device_invalidate_hdpneed to be modified accordingly.I'm surprised you gave your R-b despite this. Surely the HDP cache needs to be handled when going through the BAR?
I'm a bit confused why there's still been no response on this point from either of you. Are you guys checking internally whether or not this is needed?
Yes, it is needed. I added it in v2 of the patch.
Ah, great, hadn't seen the v2 patch yet. :)
Signature · 2 lines
Earthling Michel Dänzer | https://redhat.com Libre software enthusiast | Mesa and Xwayland developer
Am 10.03.22 um 19:06 schrieb Alex Deucher:
Quoted 5 lines
If the GPU is passed through to a guest VM, use the PCI BAR for CPU FB access rather than the physical address of carve out. The physical address is not valid in a guest.
Signed-off-by Alex Deucher <[email protected]>
Quoted 62 lines
--- drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 5 +++-- drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 2 +- drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c | 2 +- 4 files changed, 6 insertions(+), 5 deletions(-)drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +1
-1@@ -814,7 +814,7 @@ static int gmc_v10_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU ) { +if ((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) { adev->gmc.aper_base = adev->gfxhub.funcs->get_mc_fb_offset (adev); adev->gmc.aper_size = adev->gmc.real_vram_size; }drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +3
-2@@ -381,8 +381,9 @@ static int gmc_v7_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU && - adev->gmc.real_vram_size > adev->gmc.aper_size) { +if ((adev->flags &AMD_IS_APU ) && + adev->gmc.real_vram_size > adev->gmc.aper_size && + !amdgpu_passthrough (adev)) { adev->gmc.aper_base = ((u64 )RREG32 (mmMC_VM_FB_OFFSET)) <<22 ; adev->gmc.aper_size = adev->gmc.real_vram_size; }drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +1
-1@@ -581,7 +581,7 @@ static int gmc_v8_0_mc_init(struct amdgpu_device *adev) adev->gmc.aper_size =pci_resource_len (adev->pdev,0 );#ifdef CONFIG_X86_64 -if (adev->flags &AMD_IS_APU ) { +if ((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) { adev->gmc.aper_base = ((u64 )RREG32 (mmMC_VM_FB_OFFSET)) <<22 ; adev->gmc.aper_size = adev->gmc.real_vram_size; }drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +1
-1@@ -1456,7 +1456,7 @@ static int gmc_v9_0_mc_init(struct amdgpu_device *adev)*/ /* check whether both host-gpu and gpu-gpu xgmi links exist */ -if ((adev->flags &AMD_IS_APU ) || +if (((adev->flags &AMD_IS_APU ) && !amdgpu_passthrough (adev)) || (adev->gmc.xgmi.supported && adev->gmc.xgmi.connected_to_cpu)) { adev->gmc.aper_base =