drm/amdgpu: CIK UVD initialization fixes
Subsystems (2)
RFC PATCH 1/2 drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to John OlenderThe drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.
Restore the pre-buddy-allocator behavior of allowing such placements.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,int r; lpfn = (u64 )place->lpfn <<PAGE_SHIFT ; -if (!lpfn) +if (!lpfn || lpfn > man->size) lpfn = man->size; fpfn = (u64 )place->fpfn <<PAGE_SHIFT ;
Signature · 1 line
2.47.2
cover drm/amdgpu: CIK UVD initialization fixes
This patchset addresses two issues affecting UVD initialization on CIK hardware:
* Patch #1 fixes init with <256M vram:
(https://gitlab.freedesktop.org/drm/amd/-/issues/3448)
* Patch #2 fixes init with >256M vram:
(https://gitlab.freedesktop.org/drm/amd/-/issues/3851)
Includes a whitespace fix for the version on the issue tracker.
[Request for Testing]
Patch #2 tries to preserve the existing vcpu bo allocation behavior for UVD 5.0+. However, I don't have any UVD 5.0+ hardware available for regression testing.
If someone with a GCN3-5 card can confirm this patchset does not break UVD 5.0+ init, it'd be greatly appreciated.
John Olender (2):
drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++-------- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 2 files changed, 22 insertions(+), 16 deletions(-)
base-commit: 092f0316dd74beba6685b5c65ff7d25c4870e842
Signature · 1 line
2.47.2
RFC PATCH 2/2 drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
If the vcpu bos are allocated outside the uvd segment, amdgpu_uvd_ring_test_ib() times out waiting on the ring's fence.
See amdgpu_fence_driver_start_ring() for more context.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++----------- 1 file changed, 21 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +21 -15
@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);static int amdgpu_uvd_create_msg_bo_helper (struct amdgpu_device *adev,uint32_t size, -struct amdgpu_bo **bo_ptr) +struct amdgpu_bo **bo_ptr, +bool interruptible) { -struct ttm_operation_ctx ctx = { true, false }; +struct ttm_operation_ctx ctx = { interruptible, false };struct amdgpu_bo *bo =NULL ; +u32 initial_domain =AMDGPU_GEM_DOMAIN_GTT ;void *addr;int r; +if (!interruptible && adev->uvd.address_64_bit) + initial_domain |=AMDGPU_GEM_DOMAIN_VRAM ; + r =amdgpu_bo_create_reserved (adev, size,PAGE_SIZE , -AMDGPU_GEM_DOMAIN_GTT , + initial_domain, &bo,NULL , &addr);if (r)return r;
@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)if (adev->firmware.load_type !=AMDGPU_FW_LOAD_PSP ) bo_size +=AMDGPU_GPU_PAGE_ALIGN (le32_to_cpu (hdr->ucode_size_bytes) +8 ); +/* from uvd v5.0 HW addressing capacity increased to 64 bits */ +if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) + adev->uvd.address_64_bit = true; +for (j =0 ; j < adev->uvd.num_uvd_inst; j++) {if (adev->uvd.harvest_config & (1 << j))continue ; - r =amdgpu_bo_create_kernel (adev, bo_size,PAGE_SIZE , -AMDGPU_GEM_DOMAIN_VRAM | -AMDGPU_GEM_DOMAIN_GTT , - &adev->uvd.inst[j].vcpu_bo, - &adev->uvd.inst[j].gpu_addr, - &adev->uvd.inst[j].cpu_addr); + r =amdgpu_uvd_create_msg_bo_helper (adev, bo_size, + &adev->uvd.inst[j].vcpu_bo, false);if (r) {dev_err (adev->dev,"(%d) failed to allocate UVD bo\n" , r);return r; } + adev->uvd.inst[j].gpu_addr = +amdgpu_bo_gpu_offset (adev->uvd.inst[j].vcpu_bo); + adev->uvd.inst[j].cpu_addr = +amdgpu_bo_kptr (adev->uvd.inst[j].vcpu_bo); }for (i =0 ; i < adev->uvd.max_handles; ++i) {
@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) adev->uvd.filp[i] =NULL ; } -/* from uvd v5.0 HW addressing capacity increased to 64 bits */ -if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) - adev->uvd.address_64_bit = true; - - r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo, + true);if (r)return r;
@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,if (direct) { bo = adev->uvd.ib_bo; }else { - r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo, true);if (r)return r; }
Signature · 1 line
2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to John Olender+ Christian
Quoted 8 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Signed-off-by John Olender <[email protected]>
This looks correct to me.
Quoted 20 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-2@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,int r; lpfn = (u64 )place->lpfn <<PAGE_SHIFT ; -if (!lpfn) +if (!lpfn || lpfn > man->size) lpfn = man->size; fpfn = (u64 )place->fpfn <<PAGE_SHIFT ; --2.47.2
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
In reply to John Olender+ Christian
Quoted 44 lines · John Olender
If the vcpu bos are allocated outside the uvd segment,
amdgpu_uvd_ring_test_ib()times out waiting on the ring's fence.See
amdgpu_fence_driver_start_ring()for more context.Signed-off-by John Olender <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++----------- 1 file changed, 21 insertions(+), 15 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8
-3@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);static int amdgpu_uvd_create_msg_bo_helper (struct amdgpu_device *adev,uint32_t size, -struct amdgpu_bo **bo_ptr) +struct amdgpu_bo **bo_ptr, +bool interruptible) { -struct ttm_operation_ctx ctx = { true, false }; +struct ttm_operation_ctx ctx = { interruptible, false };struct amdgpu_bo *bo =NULL ; +u32 initial_domain =AMDGPU_GEM_DOMAIN_GTT ;void *addr;int r; +if (!interruptible && adev->uvd.address_64_bit) + initial_domain |=AMDGPU_GEM_DOMAIN_VRAM ; + r =amdgpu_bo_create_reserved (adev, size,PAGE_SIZE , -AMDGPU_GEM_DOMAIN_GTT , + initial_domain, &bo,NULL , &addr);if (r)return r;@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)if (adev->firmware.load_type !=AMDGPU_FW_LOAD_PSP ) bo_size +=AMDGPU_GPU_PAGE_ALIGN (le32_to_cpu (hdr->ucode_size_bytes) +8 );
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4
-3+/* from uvd v5.0 HW addressing capacity increased to 64 bits */ +if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) + adev->uvd.address_64_bit = true; +for (j =0 ; j < adev->uvd.num_uvd_inst; j++) {if (adev->uvd.harvest_config & (1 << j))continue ; - r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM | - AMDGPU_GEM_DOMAIN_GTT,
I think we can just make this VRAM only. Or something like:adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM
If that fixes it, this should be tagged with:
allocations")
Alex
Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9
-10- &adev->uvd.inst[j].vcpu_bo, - &adev->uvd.inst[j].gpu_addr, - &adev->uvd.inst[j].cpu_addr); + r =amdgpu_uvd_create_msg_bo_helper (adev, bo_size, + &adev->uvd.inst[j].vcpu_bo, false);if (r) {dev_err (adev->dev,"(%d) failed to allocate UVD bo\n" , r);return r; } + adev->uvd.inst[j].gpu_addr = +amdgpu_bo_gpu_offset (adev->uvd.inst[j].vcpu_bo); + adev->uvd.inst[j].cpu_addr = +amdgpu_bo_kptr (adev->uvd.inst[j].vcpu_bo); }for (i =0 ; i < adev->uvd.max_handles; ++i) {@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) adev->uvd.filp[i] =NULL ; } -/* from uvd v5.0 HW addressing capacity increased to 64 bits */ -if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) - adev->uvd.address_64_bit = true; - - r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo, + true);if (r)return r;@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,if (direct) { bo = adev->uvd.ib_bo; }else { - r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo, true);if (r)return r; } --2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to Alex DeucherQuoted 6 lines · Alex Deucher
+ Christian
Quoted 3 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.
Quoted 6 lines
Quoted 4 lines
Restore the pre-buddy-allocator behavior of allowing such placements.
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>
I was thinking that we should return an error when lpfn > man->size.
Regards,
Arun.
Quoted 20 lines
Quoted 20 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-2@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,int r; lpfn = (u64 )place->lpfn <<PAGE_SHIFT ; -if (!lpfn) +if (!lpfn || lpfn > man->size) lpfn = man->size; fpfn = (u64 )place->fpfn <<PAGE_SHIFT ; --2.47.2
Christian König
maintainerRe: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to Alex DeucherQuoted 7 lines · Alex Deucher
+ Christian
Quoted 4 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.
Quoted 7 lines
Quoted 4 lines
Restore the pre-buddy-allocator behavior of allowing such placements.
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>
Good catch, Arun needs to look at this.
My educated guess is that it would be better to make the drm buddy tolerate being passed end > mm->size.
Regards,
Christian.
Quoted 21 lines
Quoted 21 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-2@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,int r; lpfn = (u64 )place->lpfn <<PAGE_SHIFT ; -if (!lpfn) +if (!lpfn || lpfn > man->size) lpfn = man->size; fpfn = (u64 )place->fpfn <<PAGE_SHIFT ; --2.47.2
Christian König
maintainerRe: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
In reply to Alex DeucherQuoted 6 lines · Alex Deucher
+ Christian
Quoted 3 lines · John Olender
If the vcpu bos are allocated outside the uvd segment,
amdgpu_uvd_ring_test_ib()times out waiting on the ring's fence.
That's incorrect, but pointing to the correct solution.
Quoted 41 lines
Quoted 41 lines
See
amdgpu_fence_driver_start_ring()for more context.Signed-off-by John Olender <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++----------- 1 file changed, 21 insertions(+), 15 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8
-3@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);static int amdgpu_uvd_create_msg_bo_helper (struct amdgpu_device *adev,uint32_t size, -struct amdgpu_bo **bo_ptr) +struct amdgpu_bo **bo_ptr, +bool interruptible) { -struct ttm_operation_ctx ctx = { true, false }; +struct ttm_operation_ctx ctx = { interruptible, false };struct amdgpu_bo *bo =NULL ; +u32 initial_domain =AMDGPU_GEM_DOMAIN_GTT ;void *addr;int r; +if (!interruptible && adev->uvd.address_64_bit) + initial_domain |=AMDGPU_GEM_DOMAIN_VRAM ; + r =amdgpu_bo_create_reserved (adev, size,PAGE_SIZE , -AMDGPU_GEM_DOMAIN_GTT , + initial_domain, &bo,NULL , &addr);if (r)return r;@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)if (adev->firmware.load_type !=AMDGPU_FW_LOAD_PSP ) bo_size +=AMDGPU_GPU_PAGE_ALIGN (le32_to_cpu (hdr->ucode_size_bytes) +8 );
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4
-3+/* from uvd v5.0 HW addressing capacity increased to 64 bits */ +if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) + adev->uvd.address_64_bit = true; +for (j =0 ; j < adev->uvd.num_uvd_inst; j++) {if (adev->uvd.harvest_config & (1 << j))continue ; - r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM | - AMDGPU_GEM_DOMAIN_GTT,I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT:AMDGPU_GEM_DOMAIN_VRAM
Yeah completely agree. It's a good catch, but the solution is incorrect.
On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.
So something like AMDGPU_GEM_DOMAIN_VRAM | (adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : 0) would be correct I think.
Quoted 4 lines
If that fixes it, this should be tagged with:
Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernelallocations")
And CC stable I think.
Regards,
Christian.
Quoted 45 lines
Alex
Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9
-10- &adev->uvd.inst[j].vcpu_bo, - &adev->uvd.inst[j].gpu_addr, - &adev->uvd.inst[j].cpu_addr); + r =amdgpu_uvd_create_msg_bo_helper (adev, bo_size, + &adev->uvd.inst[j].vcpu_bo, false);if (r) {dev_err (adev->dev,"(%d) failed to allocate UVD bo\n" , r);return r; } + adev->uvd.inst[j].gpu_addr = +amdgpu_bo_gpu_offset (adev->uvd.inst[j].vcpu_bo); + adev->uvd.inst[j].cpu_addr = +amdgpu_bo_kptr (adev->uvd.inst[j].vcpu_bo); }for (i =0 ; i < adev->uvd.max_handles; ++i) {@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) adev->uvd.filp[i] =NULL ; } -/* from uvd v5.0 HW addressing capacity increased to 64 bits */ -if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) - adev->uvd.address_64_bit = true; - - r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo, + true);if (r)return r;@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,if (direct) { bo = adev->uvd.ib_bo; }else { - r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo, true);if (r)return r; } --2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to Paneer Selvam, ArunpravinQuoted 12 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.
This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.
With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs. That is, I think it's worth asking:
1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?
Thanks,
John
Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 25 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to
drm_buddy. I think it's fair to consider their style as expected.
Quoted 5 lines
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?
I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.
I will modify this at drm_buddy to match with drm_mm and send the patch.
Regards,
Arun.
Quoted 26 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
In reply to Christian KönigQuoted 72 lines · Christian König
Quoted 6 lines · Alex Deucher
+ Christian
Quoted 3 lines · John Olender
If the vcpu bos are allocated outside the uvd segment,
amdgpu_uvd_ring_test_ib()times out waiting on the ring's fence.That's incorrect, but pointing to the correct solution.
Quoted 41 lines
Quoted 41 lines
See
amdgpu_fence_driver_start_ring()for more context.Signed-off-by John Olender <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++----------- 1 file changed, 21 insertions(+), 15 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8
-3@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);static int amdgpu_uvd_create_msg_bo_helper (struct amdgpu_device *adev,uint32_t size, -struct amdgpu_bo **bo_ptr) +struct amdgpu_bo **bo_ptr, +bool interruptible) { -struct ttm_operation_ctx ctx = { true, false }; +struct ttm_operation_ctx ctx = { interruptible, false };struct amdgpu_bo *bo =NULL ; +u32 initial_domain =AMDGPU_GEM_DOMAIN_GTT ;void *addr;int r; +if (!interruptible && adev->uvd.address_64_bit) + initial_domain |=AMDGPU_GEM_DOMAIN_VRAM ; + r =amdgpu_bo_create_reserved (adev, size,PAGE_SIZE , -AMDGPU_GEM_DOMAIN_GTT , + initial_domain, &bo,NULL , &addr);if (r)return r;@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)if (adev->firmware.load_type !=AMDGPU_FW_LOAD_PSP ) bo_size +=AMDGPU_GPU_PAGE_ALIGN (le32_to_cpu (hdr->ucode_size_bytes) +8 );Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4
-3+/* from uvd v5.0 HW addressing capacity increased to 64 bits */ +if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) + adev->uvd.address_64_bit = true; +for (j =0 ; j < adev->uvd.num_uvd_inst; j++) {if (adev->uvd.harvest_config & (1 << j))continue ; - r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE, - AMDGPU_GEM_DOMAIN_VRAM | - AMDGPU_GEM_DOMAIN_GTT,I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT:AMDGPU_GEM_DOMAIN_VRAMYeah completely agree. It's a good catch, but the solution is incorrect.
On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.
So something like
AMDGPU_GEM_DOMAIN_VRAM| (adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT: 0) would be correct I think.Quoted 1 line
Quoted 12 lines
Quoted 3 lines
If that fixes it, this should be tagged with:
Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernelallocations")
And CC stable I think.
Regards,
Christian.Quoted 3 lines
Alex
Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.
Let me go into a bit of depth about how I arrived at this patch.
First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.
Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.
With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.
Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.
Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.
When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.
So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.
Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().
So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.
Thanks,
John
Quoted 43 lines
Quoted 42 lines
Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9
-10- &adev->uvd.inst[j].vcpu_bo, - &adev->uvd.inst[j].gpu_addr, - &adev->uvd.inst[j].cpu_addr); + r =amdgpu_uvd_create_msg_bo_helper (adev, bo_size, + &adev->uvd.inst[j].vcpu_bo, false);if (r) {dev_err (adev->dev,"(%d) failed to allocate UVD bo\n" , r);return r; } + adev->uvd.inst[j].gpu_addr = +amdgpu_bo_gpu_offset (adev->uvd.inst[j].vcpu_bo); + adev->uvd.inst[j].cpu_addr = +amdgpu_bo_kptr (adev->uvd.inst[j].vcpu_bo); }for (i =0 ; i < adev->uvd.max_handles; ++i) {@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev) adev->uvd.filp[i] =NULL ; } -/* from uvd v5.0 HW addressing capacity increased to 64 bits */ -if (!amdgpu_device_ip_block_version_cmp (adev,AMD_IP_BLOCK_TYPE_UVD ,5 ,0 )) - adev->uvd.address_64_bit = true; - - r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,128 <<10 , &adev->uvd.ib_bo, + true);if (r)return r;@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,if (direct) { bo = adev->uvd.ib_bo; }else { - r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo); + r =amdgpu_uvd_create_msg_bo_helper (adev,4096 , &bo, true);if (r)return r; } --2.47.2
Christian König
maintainerRe: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 9 lines
Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.
Let me go into a bit of depth about how I arrived at this patch.
First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.
Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.
But the effect is the same. E.g. you have more memory CPU accessible than otherwise.
Quoted 16 lines
Now that the issue can be reliably triggered, where does the error code, -110 /
-ETIMEDOUT, come from? It turns out it's inamdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.With that out of the way, what allocator-related change happens when a display is connected at startup? The '
stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.
Quoted 4 lines
Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.
That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.
Quoted 9 lines
When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.
So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.
Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.
Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.
The problem is that this is basically just coincident. On other systems the base address can be completely different.
See function uvd_v4_2_mc_resume() where the mmUVD_LMI_ADDR_EXT and mmUVD_LMI_EXT40_ADDR register is programmed and try to hack those two register writes and see if they really end up in the HW.
I will try to find a Kaveri system which is still working to reproduce the issue.
Thanks,
Christian.
Quoted 10 lines
Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by
amdgpu_fence_driver_start_ring().So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.
Thanks,
John
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 52 lines · Christian König
Quoted 9 lines
Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.
Let me go into a bit of depth about how I arrived at this patch.
First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.
Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.
But the effect is the same. E.g. you have more memory CPU accessible than otherwise.
Quoted 16 lines
Now that the issue can be reliably triggered, where does the error code, -110 /
-ETIMEDOUT, come from? It turns out it's inamdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.With that out of the way, what allocator-related change happens when a display is connected at startup? The '
stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.
Quoted 4 lines
Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.
That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.
Quoted 9 lines
When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.
So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.
Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.
Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.
Quoted 9 lines
The problem is that this is basically just coincident. On other systems the base address can be completely different.
See function
uvd_v4_2_mc_resume()where themmUVD_LMI_ADDR_EXTandmmUVD_LMI_EXT40_ADDRregister is programmed and try to hack those two register writes and see if they really end up in the HW.I will try to find a Kaveri system which is still working to reproduce the issue.
Thanks,
Christian.
I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.
The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.
The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.
Anyway, hopefully these extra configuration details help narrow down the problem.
Thanks,
John
Quoted 10 lines
Quoted 10 lines
Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by
amdgpu_fence_driver_start_ring().So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.
Thanks,
John
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 49 lines · John Olender
Quoted 48 lines · Christian König
Quoted 9 lines
Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.
Let me go into a bit of depth about how I arrived at this patch.
First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.
Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.
But the effect is the same. E.g. you have more memory CPU accessible than otherwise.
Quoted 16 lines
Now that the issue can be reliably triggered, where does the error code, -110 /
-ETIMEDOUT, come from? It turns out it's inamdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.With that out of the way, what allocator-related change happens when a display is connected at startup? The '
stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.
Quoted 4 lines
Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.
That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.
Quoted 9 lines
When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.
So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.
Quoted 7 lines
Quoted 7 lines
Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.
Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.
The problem is that this is basically just coincident. On other systems the base address can be completely different.
See function
uvd_v4_2_mc_resume()where themmUVD_LMI_ADDR_EXTandmmUVD_LMI_EXT40_ADDRregister is programmed and try to hack those two register writes and see if they really end up in the HW.
Okay, I did a read and compare after each write.
Both writes seem to go through on both the Kaveri and s9150:
Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote 0x00001001, read 0x00001001 [same] amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote 0x800900F4, read 0x800900F4 [same]
s9150:
amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote 0x0000F00F, read 0x0000F00F [same] amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote 0x800900F7, read 0x800900F7 [same]
Thanks,
John
Quoted 37 lines
Quoted 6 lines
I will try to find a Kaveri system which is still working to reproduce the issue.
Thanks,
Christian.I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.
The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.
The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.
Anyway, hopefully these extra configuration details help narrow down the problem.
Thanks,
JohnQuoted 10 lines
Quoted 10 lines
Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by
amdgpu_fence_driver_start_ring().So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.
Thanks,
John
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
In reply to Paneer Selvam, ArunpravinQuoted 31 lines · Arunpravin
Quoted 28 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:
Quoted 8 lines
Quoted 2 lines
1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.I will modify this at
drm_buddyto match withdrm_mmand send the patch.
After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Regards,
Arun.
Quoted 30 lines
Regards,
Arun.Quoted 26 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Paneer Selvam, Arunpravin
Reviewed-byRe: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 37 lines · Arunpravin
Quoted 34 lines · Arunpravin
Quoted 30 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?
Quoted 8 lines
Quoted 5 lines
I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.I will modify this at
drm_buddyto match withdrm_mmand send the patch.After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Quoted 34 lines
Regards,
Arun.Quoted 30 lines
Regards,
Arun.Quoted 26 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Christian König
maintainerRe: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 44 lines · Arunpravin
Quoted 41 lines · Arunpravin
Quoted 38 lines · Arunpravin
Quoted 25 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.
Quoted 5 lines
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
I will modify this at
drm_buddyto match withdrm_mmand send the patch.After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Reviewed-by Arunpravin Paneer Selvam <[email protected]>
Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.
Alex most likely won't follow the discussion till the end.
Thanks,
Christian.
Quoted 35 lines
Quoted 34 lines
Regards,
Arun.Quoted 30 lines
Regards,
Arun.Quoted 26 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;
--
2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 41 lines · Christian König
Quoted 39 lines · Arunpravin
Quoted 37 lines · Arunpravin
Quoted 35 lines · Arunpravin
Quoted 24 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.
Quoted 5 lines
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
I will modify this at
drm_buddyto match withdrm_mmand send the patch.After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Reviewed-by Arunpravin Paneer Selvam <[email protected]>Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.
Alex most likely won't follow the discussion till the end.
Sure Christian, I will merge this patch into amd-staging-drm-next.
Thanks,
Arun.
Quoted 33 lines
Thanks,
Christian.Quoted 29 lines
Quoted 29 lines
Regards,
Arun.Quoted 27 lines
Regards,
Arun.Quoted 25 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 45 lines · Arunpravin
Quoted 42 lines · Christian König
Quoted 40 lines · Arunpravin
Quoted 38 lines · Arunpravin
Quoted 36 lines · Arunpravin
Quoted 24 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to
drm_buddy. I think it's fair to consider their style as expected.Quoted 5 lines
With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.I will modify this at
drm_buddyto match withdrm_mmand send the patch.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Reviewed-by Arunpravin Paneer Selvam <[email protected]>Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.
Alex most likely won't follow the discussion till the end.
Sure Christian, I will merge this patch into amd-staging-drm-next.
I see a black screen problem during the amdgpu driver load on the tip of amd-staging-drm-next, once that issue is fixed, I will push this patch into amd-staging-drm-next.
Regards,
Arun.
Quoted 37 lines
Thanks,
Arun.Quoted 33 lines
Thanks,
Christian.Quoted 29 lines
Quoted 29 lines
Regards,
Arun.Quoted 27 lines
Regards,
Arun.Quoted 25 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram
Quoted 51 lines · Arunpravin
Quoted 48 lines · Arunpravin
Quoted 45 lines · Christian König
Quoted 43 lines · Arunpravin
Quoted 41 lines · Arunpravin
Quoted 29 lines · Arunpravin
Quoted 27 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher
+ Christian
Quoted 5 lines · John Olender
The
drm_mmallocator tolerated being passed end >mm->size, but thedrm_buddyallocator does not.Restore the pre-buddy-allocator behavior of allowing such placements.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Signed-off-by John Olender <[email protected]>This looks correct to me.
Reviewed-by Alex Deucher <[email protected]>I was thinking that we should return an error when lpfn >
man->size.Regards,
Arun.This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to
drm_buddy. I think it's fair to consider their style as expected.With that in mind, I'm not sure
amdgpu_vram_mgris the place this change really belongs. That is, I think it's worth asking:Quoted 7 lines
Quoted 2 lines
1) Why does
drm_mmaccept end >mm->sizewithout complaint?
2) Why doesn'tdrm_buddydo the same?I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end >
mm->size. This was done to ensure that, at the driver level, lpfn has the correct value.I will modify this at
drm_buddyto match withdrm_mmand send the patch.After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line
Reviewed-by Arunpravin Paneer Selvam <[email protected]>Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.
Alex most likely won't follow the discussion till the end.
Sure Christian, I will merge this patch into amd-staging-drm-next.
I see a black screen problem during the amdgpu driver load on the tip of amd-staging-drm-next, once that issue is fixed, I will push this patch into amd-staging-drm-next.
I have merged this patch into amd-staging-drm-next.
Thanks,
Arun.
Quoted 41 lines
Regards,
Arun.Quoted 37 lines
Thanks,
Arun.Quoted 33 lines
Thanks,
Christian.Quoted 29 lines
Quoted 29 lines
Regards,
Arun.Quoted 27 lines
Regards,
Arun.Quoted 25 lines
Thanks,
JohnQuoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0
-0
gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index2d7f82e98df9..abdc52b0895a100644drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1
-1@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(structttm_resource_manager *man, int r; lpfn = (u64)place->lpfn << PAGE_SHIFT; - if (!lpfn) + if (!lpfn || lpfn > man->size) lpfn = man->size;fpfn = (u64)place->fpfn <<
PAGE_SHIFT;Signature · 1 line
2.47.2
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
In reply to John OlenderPing.
Quoted 62 lines · John Olender
Quoted 49 lines · John Olender
Quoted 48 lines · Christian König
Quoted 9 lines
Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.
Let me go into a bit of depth about how I arrived at this patch.
First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.
Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.
But the effect is the same. E.g. you have more memory CPU accessible than otherwise.
Quoted 16 lines
Now that the issue can be reliably triggered, where does the error code, -110 /
-ETIMEDOUT, come from? It turns out it's inamdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.With that out of the way, what allocator-related change happens when a display is connected at startup? The '
stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.
Quoted 4 lines
Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.
That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.
Quoted 9 lines
When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.
So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.
Quoted 7 lines
Quoted 7 lines
Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.
Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.
The problem is that this is basically just coincident. On other systems the base address can be completely different.
See function
uvd_v4_2_mc_resume()where themmUVD_LMI_ADDR_EXTandmmUVD_LMI_EXT40_ADDRregister is programmed and try to hack those two register writes and see if they really end up in the HW.Okay, I did a read and compare after each write.
Both writes seem to go through on both the Kaveri and s9150:
Quoted 8 lines
Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote0x00001001, read0x00001001[same] amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote0x800900F4, read0x800900F4[same]s9150:
amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote0x0000F00F, read0x0000F00F[same] amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote0x800900F7, read0x800900F7[same]
I've also confirmed the patch works fine when segments other than [0, 256M) are used.
E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.
Quoted 41 lines
Thanks,
JohnQuoted 37 lines
Quoted 6 lines
I will try to find a Kaveri system which is still working to reproduce the issue.
Thanks,
Christian.I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.
The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.
The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.
Anyway, hopefully these extra configuration details help narrow down the problem.
Thanks,
JohnQuoted 10 lines
Quoted 10 lines
Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by
amdgpu_fence_driver_start_ring().So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.
Thanks,
John
Christian König
maintainerRe: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Hi John,
first of all thanks a lot for taking a look into this.
Quoted 13 lines
Quoted 4 lines
Okay, I did a read and compare after each write.
Both writes seem to go through on both the Kaveri and s9150:
Quoted 8 lines
Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote0x00001001, read0x00001001[same] amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote0x800900F4, read0x800900F4[same]s9150:
amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote0x0000F00F, read0x0000F00F[same] amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote0x800900F7, read0x800900F7[same]
Quoted 5 lines
I've also confirmed the patch works fine when segments other than [0, 256M) are used.
E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.
Oh, that's a very interesting find. Could you try to turn around the way the patch works?
E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.
That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.
When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.
Thanks,
Christian.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I will try to find a Kaveri system which is still working to reproduce the issue.
I unfortunately couldn't find a working box of hand. Would need to search in our HW stash for a box which still works and get that shipped to me.
And that is overhead for this issue I would rather like to avoid.
So if you can come up with a simpler patch which works for you I'm happy to take that.
Regards,
Christian.
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 25 lines · Christian König
Hi John,
first of all thanks a lot for taking a look into this.
Quoted 13 lines
Quoted 4 lines
Okay, I did a read and compare after each write.
Both writes seem to go through on both the Kaveri and s9150:
Quoted 8 lines
Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote0x00001001, read0x00001001[same] amdgpu 0000:00:01.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote0x800900F4, read0x800900F4[same]s9150:
amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote0x0000F00F, read0x0000F00F[same] amdgpu 0000:41:00.0: amdgpu: [drm]uvd_v4_2_mc_resume:mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote0x800900F7, read0x800900F7[same]Quoted 5 lines
I've also confirmed the patch works fine when segments other than [0, 256M) are used.
E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.
Oh, that's a very interesting find. Could you try to turn around the way the patch works?
Quoted 9 lines
E.g. instead of forcing the UVD FW into the first segment, change
amdgpu_uvd_force_into_uvd_segment()so that the BOs are forced into the same segment as the UVD firmware.That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.
When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.
Thanks,
Christian.
Okay, yeah, that should make for a significantly simpler fix.
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
I will try to find a Kaveri system which is still working to reproduce the issue.
I unfortunately couldn't find a working box of hand. Would need to search in our HW stash for a box which still works and get that shipped to me.
And that is overhead for this issue I would rather like to avoid.
So if you can come up with a simpler patch which works for you I'm happy to take that.
Regards,
Christian.
Sounds good. I'll get started on this variant.
Thanks,
John
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 4 lines
Quoted 4 lines
Oh, that's a very interesting find. Could you try to turn around the way the patch works?
E.g. instead of forcing the UVD FW into the first segment, change
amdgpu_uvd_force_into_uvd_segment()so that the BOs are forced into the same segment as the UVD firmware.
I started implementing this and I realized two main problems with this approach.
First, there's currently no guarantee the UVD FW does not cross a 256MB boundary. Checking for this and providing a fallback is going to make this patch... not really any less complex than the original.
Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.
Quoted 4 lines
Quoted 4 lines
That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.
When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.
During my initial investigation, I found out that the UVD FW got placed in the first segment *because* things were already placed there. This is why adding a 'stolen_vga_memory' substitute was an effective workaround.
So, CIK is *already* forced to deal with an overloaded first segment and, with the inverted approach, will continue to do so for typical use cases. Explicitly placing the UVD FW into the first segment just makes this guaranteed.
I did implement a module parameter for testing that allows designating a specific 256MB segment as the legacy UVD segment. I can polish this up so the user has the option to manually relieve some of the first segment pressure on SI and CIK devices.
I haven't run into a situation where I've needed this during normal use, but I can certainly appreciate it being available.
Thanks,
John
Christian König
maintainerRe: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 10 lines · John Olender
Quoted 4 lines
Quoted 4 lines
Oh, that's a very interesting find. Could you try to turn around the way the patch works?
E.g. instead of forcing the UVD FW into the first segment, change
amdgpu_uvd_force_into_uvd_segment()so that the BOs are forced into the same segment as the UVD firmware.I started implementing this and I realized two main problems with this approach.
First, there's currently no guarantee the UVD FW does not cross a 256MB boundary.
There actually is a guarantee for that.
FW BOs are always allocated contiguously, and the allocation backend makes sure that for example an 16MiB (rounded up) FW is always aligned to a 16MiB boundary.
So FW memory never crosses a segment boundary larger than it's rounded up size. Without that tons of things in our driver would fall apart, not just UVD :)
Checking for this and providing a fallback is going to make
Quoted 4 lines
this patch... not really any less complex than the original.
Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.
That's fine with me. I'm just concerned about that 1% of people where it potentially worked before and we are now breaking it.
Quoted 15 lines
Quoted 5 lines
Quoted 5 lines
That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.
When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.
During my initial investigation, I found out that the UVD FW got placed in the first segment *because* things were already placed there. This is why adding a '
stolen_vga_memory' substitute was an effective workaround.So, CIK is *already* forced to deal with an overloaded first segment and, with the inverted approach, will continue to do so for typical use cases. Explicitly placing the UVD FW into the first segment just makes this guaranteed.
Quoted 4 lines
I did implement a module parameter for testing that allows designating a specific 256MB segment as the legacy UVD segment. I can polish this up so the user has the option to manually relieve some of the first segment pressure on SI and CIK devices.
The module parameter is fine for your testing, but that is not something I would accept upstream.
As soon as things are not working automatically any more people will start to complain. That we have a manual work around is only feasible to a minority.
Regards,
Christian.
Quoted 6 lines
I haven't run into a situation where I've needed this during normal use, but I can certainly appreciate it being available.
Thanks,
John
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 11 lines · Christian König
Quoted 8 lines · John Olender
Quoted 4 lines
Quoted 4 lines
Oh, that's a very interesting find. Could you try to turn around the way the patch works?
E.g. instead of forcing the UVD FW into the first segment, change
amdgpu_uvd_force_into_uvd_segment()so that the BOs are forced into the same segment as the UVD firmware.I started implementing this and I realized two main problems with this approach.
Quoted 8 lines
Quoted 2 lines
First, there's currently no guarantee the UVD FW does not cross a 256MB boundary.
There actually is a guarantee for that.
FW BOs are always allocated contiguously, and the allocation backend makes sure that for example an 16MiB (rounded up) FW is always aligned to a 16MiB boundary.
So FW memory never crosses a segment boundary larger than it's rounded up size. Without that tons of things in our driver would fall apart, not just UVD :)
Aha, thanks!
Quoted 14 lines
Checking for this and providing a fallback is going to make
Quoted 4 lines
this patch... not really any less complex than the original.
Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.
That's fine with me. I'm just concerned about that 1% of people where it potentially worked before and we are now breaking it.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.
When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.
Okay this makes much more sense. I read this as general concern and put the brakes on, especially after missing the above.
I'll resume work on the simplified patch.
Thanks again,
John
Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
In reply to Christian KönigQuoted 8 lines · Christian König
Quoted 2 lines
I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT:AMDGPU_GEM_DOMAIN_VRAMYeah completely agree. It's a good catch, but the solution is incorrect.
On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.
So something like
AMDGPU_GEM_DOMAIN_VRAM| (adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT: 0) would be correct I think.
Quoted 9 lines
Quoted 3 lines
If that fixes it, this should be tagged with:
Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernelallocations")
And CC stable I think.
Regards,
Christian.Quoted 1 line
Alex
The simplified patch needs this change to the vcpu bo domain. Would you prefer this change as a separate patch?
Thanks,
John
Christian König
maintainerRe: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment
Quoted 14 lines · John Olender
Quoted 13 lines · Christian König
Quoted 2 lines
I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT:AMDGPU_GEM_DOMAIN_VRAMYeah completely agree. It's a good catch, but the solution is incorrect.
On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.
So something like
AMDGPU_GEM_DOMAIN_VRAM| (adev->uvd.address_64_bit?AMDGPU_GEM_DOMAIN_GTT: 0) would be correct I think.Quoted 3 lines
If that fixes it, this should be tagged with:
Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernelallocations")
And CC stable I think.
Quoted 7 lines
Quoted 4 lines
Regards,
Christian.Quoted 1 line
Alex
The simplified patch needs this change to the vcpu bo domain. Would you prefer this change as a separate patch?
Separate one please.
Just send it as the first one in a series.
Regards,
Christian.
Quoted 3 lines
Thanks,
John