Skip to contentSkip to conversation

drm/amdgpu: CIK UVD initialization fixes

RFC PATCH In review John Olender 4 participants 28 messages Apr 29, 2025 – Jun 5, 2025

Subsystems (2)
lore
Share & tools
lore
#1 lore raw

RFC PATCH 1/2 drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
index 2d7f82e98df9..abdc52b0895a 100644
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
 	int r;
 
 	lpfn = (u64)place->lpfn << PAGE_SHIFT;
-	if (!lpfn)
+	if (!lpfn || lpfn > man->size)
 		lpfn = man->size;
 
 	fpfn = (u64)place->fpfn << PAGE_SHIFT;
Signature · 1 line
2.47.2
#0 lore raw

cover drm/amdgpu: CIK UVD initialization fixes

This patchset addresses two issues affecting UVD initialization on CIK hardware:

* Patch #1 fixes init with <256M vram:
(https://gitlab.freedesktop.org/drm/amd/-/issues/3448)

* Patch #2 fixes init with >256M vram:
(https://gitlab.freedesktop.org/drm/amd/-/issues/3851)
Includes a whitespace fix for the version on the issue tracker.

[Request for Testing]

Patch #2 tries to preserve the existing vcpu bo allocation behavior for UVD 5.0+. However, I don't have any UVD 5.0+ hardware available for regression testing.

If someone with a GCN3-5 card can confirm this patchset does not break UVD 5.0+ init, it'd be greatly appreciated.

John Olender (2):
drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c      | 36 ++++++++++++--------
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c |  2 +-
 2 files changed, 22 insertions(+), 16 deletions(-)

base-commit: 092f0316dd74beba6685b5c65ff7d25c4870e842

Signature · 1 line
2.47.2
#13 lore raw

RFC PATCH 2/2 drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

If the vcpu bos are allocated outside the uvd segment, amdgpu_uvd_ring_test_ib() times out waiting on the ring's fence.

See amdgpu_fence_driver_start_ring() for more context.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++-----------
 1 file changed, 21 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +21 -15
index 74758b5ffc6c..a6b3e75ffa2d 100644
@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);
 
 static int amdgpu_uvd_create_msg_bo_helper(struct amdgpu_device *adev,
 					   uint32_t size,
-					   struct amdgpu_bo **bo_ptr)
+					   struct amdgpu_bo **bo_ptr,
+					   bool interruptible)
 {
-	struct ttm_operation_ctx ctx = { true, false };
+	struct ttm_operation_ctx ctx = { interruptible, false };
 	struct amdgpu_bo *bo = NULL;
+	u32 initial_domain = AMDGPU_GEM_DOMAIN_GTT;
 	void *addr;
 	int r;
 
+	if (!interruptible && adev->uvd.address_64_bit)
+		initial_domain |= AMDGPU_GEM_DOMAIN_VRAM;
+
 	r = amdgpu_bo_create_reserved(adev, size, PAGE_SIZE,
-				      AMDGPU_GEM_DOMAIN_GTT,
+				      initial_domain,
 				      &bo, NULL, &addr);
 	if (r)
 		return r;
@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
 	if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP)
 		bo_size += AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(hdr->ucode_size_bytes) + 8);
 
+	/* from uvd v5.0 HW addressing capacity increased to 64 bits */
+	if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
+		adev->uvd.address_64_bit = true;
+
 	for (j = 0; j < adev->uvd.num_uvd_inst; j++) {
 		if (adev->uvd.harvest_config & (1 << j))
 			continue;
-		r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE,
-					    AMDGPU_GEM_DOMAIN_VRAM |
-					    AMDGPU_GEM_DOMAIN_GTT,
-					    &adev->uvd.inst[j].vcpu_bo,
-					    &adev->uvd.inst[j].gpu_addr,
-					    &adev->uvd.inst[j].cpu_addr);
+		r = amdgpu_uvd_create_msg_bo_helper(adev, bo_size,
+				&adev->uvd.inst[j].vcpu_bo, false);
 		if (r) {
 			dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r);
 			return r;
 		}
+		adev->uvd.inst[j].gpu_addr =
+				amdgpu_bo_gpu_offset(adev->uvd.inst[j].vcpu_bo);
+		adev->uvd.inst[j].cpu_addr =
+				amdgpu_bo_kptr(adev->uvd.inst[j].vcpu_bo);
 	}
 
 	for (i = 0; i < adev->uvd.max_handles; ++i) {
@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
 		adev->uvd.filp[i] = NULL;
 	}
 
-	/* from uvd v5.0 HW addressing capacity increased to 64 bits */
-	if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
-		adev->uvd.address_64_bit = true;
-
-	r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo);
+	r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo,
+			true);
 	if (r)
 		return r;
 
@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
 	if (direct) {
 		bo = adev->uvd.ib_bo;
 	} else {
-		r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo);
+		r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo, true);
 		if (r)
 			return r;
 	}
Signature · 1 line
2.47.2
Reviewed-by on v1 1/2
#2 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to John Olender

+ Christian

Quoted 8 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

This looks correct to me.

Quoted 20 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -2
index 2d7f82e98df9..abdc52b0895a 100644
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
        int r;
 
        lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                lpfn = man->size;
 
        fpfn = (u64)place->fpfn << PAGE_SHIFT;
--

2.47.2

#14 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

In reply to John Olender

+ Christian

Quoted 44 lines · John Olender

If the vcpu bos are allocated outside the uvd segment, amdgpu_uvd_ring_test_ib() times out waiting on the ring's fence.

See amdgpu_fence_driver_start_ring() for more context.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++-----------
 1 file changed, 21 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8 -3
index 74758b5ffc6c..a6b3e75ffa2d 100644
@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);
 
 static int amdgpu_uvd_create_msg_bo_helper(struct amdgpu_device *adev,
                                           uint32_t size,
-                                          struct amdgpu_bo **bo_ptr)
+                                          struct amdgpu_bo **bo_ptr,
+                                          bool interruptible)
 {
-       struct ttm_operation_ctx ctx = { true, false };
+       struct ttm_operation_ctx ctx = { interruptible, false };
        struct amdgpu_bo *bo = NULL;
+       u32 initial_domain = AMDGPU_GEM_DOMAIN_GTT;
        void *addr;
        int r;
 
+       if (!interruptible && adev->uvd.address_64_bit)
+               initial_domain |= AMDGPU_GEM_DOMAIN_VRAM;
+
        r = amdgpu_bo_create_reserved(adev, size, PAGE_SIZE,
-                                     AMDGPU_GEM_DOMAIN_GTT,
+                                     initial_domain,
                                      &bo, NULL, &addr);
        if (r)
                return r;
@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
        if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP)
                bo_size += AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(hdr->ucode_size_bytes) + 8);
 

drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4 -3
+       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
+       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
+               adev->uvd.address_64_bit = true;
+
        for (j = 0; j < adev->uvd.num_uvd_inst; j++) {
                if (adev->uvd.harvest_config & (1 << j))
                        continue;
-               r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE,
-                                           AMDGPU_GEM_DOMAIN_VRAM |
-                                           AMDGPU_GEM_DOMAIN_GTT,

I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM

If that fixes it, this should be tagged with:

Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernel

allocations")

Alex

Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9 -10
-                                           &adev->uvd.inst[j].vcpu_bo,
-                                           &adev->uvd.inst[j].gpu_addr,
-                                           &adev->uvd.inst[j].cpu_addr);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, bo_size,
+                               &adev->uvd.inst[j].vcpu_bo, false);
                if (r) {
                        dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r);
                        return r;
                }
+               adev->uvd.inst[j].gpu_addr =
+                               amdgpu_bo_gpu_offset(adev->uvd.inst[j].vcpu_bo);
+               adev->uvd.inst[j].cpu_addr =
+                               amdgpu_bo_kptr(adev->uvd.inst[j].vcpu_bo);
        }
 
        for (i = 0; i < adev->uvd.max_handles; ++i) {
@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
                adev->uvd.filp[i] = NULL;
        }
 
-       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
-       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
-               adev->uvd.address_64_bit = true;
-
-       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo);
+       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo,
+                       true);
        if (r)
                return r;
 
@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
        if (direct) {
                bo = adev->uvd.ib_bo;
        } else {
-               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo, true);
                if (r)
                        return r;
        }
--

2.47.2

#3 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to Alex Deucher
Quoted 6 lines · Alex Deucher

+ Christian

Quoted 3 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Quoted 6 lines
Quoted 4 lines

Restore the pre-buddy-allocator behavior of allowing such placements.

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

Quoted 20 lines
Quoted 20 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -2
index 2d7f82e98df9..abdc52b0895a 100644
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
         int r;
 
         lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                 lpfn = man->size;
 
         fpfn = (u64)place->fpfn << PAGE_SHIFT;
--

2.47.2

maintainer
#12 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to Alex Deucher
Quoted 7 lines · Alex Deucher

+ Christian

Quoted 4 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Quoted 7 lines
Quoted 4 lines

Restore the pre-buddy-allocator behavior of allowing such placements.

This looks correct to me.

Good catch, Arun needs to look at this.

My educated guess is that it would be better to make the drm buddy tolerate being passed end > mm->size.

Regards,
Christian.

Quoted 21 lines
Quoted 21 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -2
index 2d7f82e98df9..abdc52b0895a 100644
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct ttm_resource_manager *man,
        int r;
 
        lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                lpfn = man->size;
 
        fpfn = (u64)place->fpfn << PAGE_SHIFT;
--

2.47.2

maintainer
#15 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

In reply to Alex Deucher
Quoted 6 lines · Alex Deucher

+ Christian

Quoted 3 lines · John Olender

If the vcpu bos are allocated outside the uvd segment, amdgpu_uvd_ring_test_ib() times out waiting on the ring's fence.

That's incorrect, but pointing to the correct solution.

Quoted 41 lines
Quoted 41 lines

See amdgpu_fence_driver_start_ring() for more context.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++-----------
 1 file changed, 21 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8 -3
index 74758b5ffc6c..a6b3e75ffa2d 100644
@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);
 
 static int amdgpu_uvd_create_msg_bo_helper(struct amdgpu_device *adev,
                                           uint32_t size,
-                                          struct amdgpu_bo **bo_ptr)
+                                          struct amdgpu_bo **bo_ptr,
+                                          bool interruptible)
 {
-       struct ttm_operation_ctx ctx = { true, false };
+       struct ttm_operation_ctx ctx = { interruptible, false };
        struct amdgpu_bo *bo = NULL;
+       u32 initial_domain = AMDGPU_GEM_DOMAIN_GTT;
        void *addr;
        int r;
 
+       if (!interruptible && adev->uvd.address_64_bit)
+               initial_domain |= AMDGPU_GEM_DOMAIN_VRAM;
+
        r = amdgpu_bo_create_reserved(adev, size, PAGE_SIZE,
-                                     AMDGPU_GEM_DOMAIN_GTT,
+                                     initial_domain,
                                      &bo, NULL, &addr);
        if (r)
                return r;
@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
        if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP)
                bo_size += AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(hdr->ucode_size_bytes) + 8);
 
Quoted 13 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4 -3
+       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
+       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
+               adev->uvd.address_64_bit = true;
+
        for (j = 0; j < adev->uvd.num_uvd_inst; j++) {
                if (adev->uvd.harvest_config & (1 << j))
                        continue;
-               r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE,
-                                           AMDGPU_GEM_DOMAIN_VRAM |
-                                           AMDGPU_GEM_DOMAIN_GTT,

I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM

Yeah completely agree. It's a good catch, but the solution is incorrect.

On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.

So something like AMDGPU_GEM_DOMAIN_VRAM | (adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : 0) would be correct I think.

Quoted 4 lines

If that fixes it, this should be tagged with:

Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernel

allocations")

And CC stable I think.

Regards,
Christian.

Quoted 45 lines

Alex

Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9 -10
-                                           &adev->uvd.inst[j].vcpu_bo,
-                                           &adev->uvd.inst[j].gpu_addr,
-                                           &adev->uvd.inst[j].cpu_addr);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, bo_size,
+                               &adev->uvd.inst[j].vcpu_bo, false);
                if (r) {
                        dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r);
                        return r;
                }
+               adev->uvd.inst[j].gpu_addr =
+                               amdgpu_bo_gpu_offset(adev->uvd.inst[j].vcpu_bo);
+               adev->uvd.inst[j].cpu_addr =
+                               amdgpu_bo_kptr(adev->uvd.inst[j].vcpu_bo);
        }
 
        for (i = 0; i < adev->uvd.max_handles; ++i) {
@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
                adev->uvd.filp[i] = NULL;
        }
 
-       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
-       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
-               adev->uvd.address_64_bit = true;
-
-       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo);
+       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo,
+                       true);
        if (r)
                return r;
 
@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
        if (direct) {
                bo = adev->uvd.ib_bo;
        } else {
-               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo, true);
                if (r)
                        return r;
        }
--

2.47.2

#4 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to Paneer Selvam, Arunpravin
Quoted 12 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs. That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

Thanks,
John

Quoted 23 lines
Quoted 22 lines
Quoted 22 lines

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
          int r;
 
          lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                  lpfn = man->size;
 

         fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#5 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 25 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel". The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.

Quoted 5 lines

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs. That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

I will modify this at drm_buddy to match with drm_mm and send the patch.

Regards,
Arun.

Quoted 26 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
          int r;
 
          lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                  lpfn = man->size;
 

         fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#16 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

In reply to Christian König
Quoted 72 lines · Christian König
Quoted 6 lines · Alex Deucher

+ Christian

Quoted 3 lines · John Olender

If the vcpu bos are allocated outside the uvd segment, amdgpu_uvd_ring_test_ib() times out waiting on the ring's fence.

That's incorrect, but pointing to the correct solution.

Quoted 41 lines
Quoted 41 lines

See amdgpu_fence_driver_start_ring() for more context.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 36 ++++++++++++++-----------
 1 file changed, 21 insertions(+), 15 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +8 -3
index 74758b5ffc6c..a6b3e75ffa2d 100644
@@ -139,15 +139,20 @@ static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo);
 
 static int amdgpu_uvd_create_msg_bo_helper(struct amdgpu_device *adev,
                                           uint32_t size,
-                                          struct amdgpu_bo **bo_ptr)
+                                          struct amdgpu_bo **bo_ptr,
+                                          bool interruptible)
 {
-       struct ttm_operation_ctx ctx = { true, false };
+       struct ttm_operation_ctx ctx = { interruptible, false };
        struct amdgpu_bo *bo = NULL;
+       u32 initial_domain = AMDGPU_GEM_DOMAIN_GTT;
        void *addr;
        int r;
 
+       if (!interruptible && adev->uvd.address_64_bit)
+               initial_domain |= AMDGPU_GEM_DOMAIN_VRAM;
+
        r = amdgpu_bo_create_reserved(adev, size, PAGE_SIZE,
-                                     AMDGPU_GEM_DOMAIN_GTT,
+                                     initial_domain,
                                      &bo, NULL, &addr);
        if (r)
                return r;
@@ -319,19 +324,23 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
        if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP)
                bo_size += AMDGPU_GPU_PAGE_ALIGN(le32_to_cpu(hdr->ucode_size_bytes) + 8);
 
Quoted 13 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c int amdgpu_uvd_sw_init(struct amdgpu_device *adev) · line 327

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +4 -3
+       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
+       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
+               adev->uvd.address_64_bit = true;
+
        for (j = 0; j < adev->uvd.num_uvd_inst; j++) {
                if (adev->uvd.harvest_config & (1 << j))
                        continue;
-               r = amdgpu_bo_create_kernel(adev, bo_size, PAGE_SIZE,
-                                           AMDGPU_GEM_DOMAIN_VRAM |
-                                           AMDGPU_GEM_DOMAIN_GTT,

I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM

Yeah completely agree. It's a good catch, but the solution is incorrect.

On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.

So something like AMDGPU_GEM_DOMAIN_VRAM | (adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : 0) would be correct I think.

Quoted 1 line
Quoted 12 lines
Quoted 3 lines

If that fixes it, this should be tagged with:

Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernel

allocations")

And CC stable I think.

Regards,
Christian.

Quoted 3 lines

Alex

Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.

Let me go into a bit of depth about how I arrived at this patch.

First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.

Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.

With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.

Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.

Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.

When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.

So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.

Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().

So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.

Thanks,
John

Quoted 43 lines
Quoted 42 lines
Quoted 42 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +9 -10
-                                           &adev->uvd.inst[j].vcpu_bo,
-                                           &adev->uvd.inst[j].gpu_addr,
-                                           &adev->uvd.inst[j].cpu_addr);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, bo_size,
+                               &adev->uvd.inst[j].vcpu_bo, false);
                if (r) {
                        dev_err(adev->dev, "(%d) failed to allocate UVD bo\n", r);
                        return r;
                }
+               adev->uvd.inst[j].gpu_addr =
+                               amdgpu_bo_gpu_offset(adev->uvd.inst[j].vcpu_bo);
+               adev->uvd.inst[j].cpu_addr =
+                               amdgpu_bo_kptr(adev->uvd.inst[j].vcpu_bo);
        }
 
        for (i = 0; i < adev->uvd.max_handles; ++i) {
@@ -339,11 +348,8 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
                adev->uvd.filp[i] = NULL;
        }
 
-       /* from uvd v5.0 HW addressing capacity increased to 64 bits */
-       if (!amdgpu_device_ip_block_version_cmp(adev, AMD_IP_BLOCK_TYPE_UVD, 5, 0))
-               adev->uvd.address_64_bit = true;
-
-       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo);
+       r = amdgpu_uvd_create_msg_bo_helper(adev, 128 << 10, &adev->uvd.ib_bo,
+                       true);
        if (r)
                return r;
 
@@ -1236,7 +1242,7 @@ int amdgpu_uvd_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
        if (direct) {
                bo = adev->uvd.ib_bo;
        } else {
-               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo);
+               r = amdgpu_uvd_create_msg_bo_helper(adev, 4096, &bo, true);
                if (r)
                        return r;
        }
--

2.47.2

maintainer
#17 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 9 lines

Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.

Let me go into a bit of depth about how I arrived at this patch.

First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.

Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.

But the effect is the same. E.g. you have more memory CPU accessible than otherwise.

Quoted 16 lines

Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.

With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.

Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.

Quoted 4 lines

Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.

That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.

Quoted 9 lines

When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.

So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.

Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.

Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.

The problem is that this is basically just coincident. On other systems the base address can be completely different.

See function uvd_v4_2_mc_resume() where the mmUVD_LMI_ADDR_EXT and mmUVD_LMI_EXT40_ADDR register is programmed and try to hack those two register writes and see if they really end up in the HW.

I will try to find a Kaveri system which is still working to reproduce the issue.

Thanks,
Christian.

Quoted 10 lines

Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().

So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.

Thanks,
John

#18 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 52 lines · Christian König
Quoted 9 lines

Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.

Let me go into a bit of depth about how I arrived at this patch.

First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.

Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.

But the effect is the same. E.g. you have more memory CPU accessible than otherwise.

Quoted 16 lines

Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.

With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.

Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.

Quoted 4 lines

Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.

That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.

Quoted 9 lines

When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.

So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.

Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.

Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.

Quoted 9 lines

The problem is that this is basically just coincident. On other systems the base address can be completely different.

See function uvd_v4_2_mc_resume() where the mmUVD_LMI_ADDR_EXT and mmUVD_LMI_EXT40_ADDR register is programmed and try to hack those two register writes and see if they really end up in the HW.

I will try to find a Kaveri system which is still working to reproduce the issue.

Thanks,
Christian.

I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.

The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.

The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.

Anyway, hopefully these extra configuration details help narrow down the problem.

Thanks,
John

Quoted 10 lines
Quoted 10 lines

Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().

So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.

Thanks,
John

#19 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 49 lines · John Olender
Quoted 48 lines · Christian König
Quoted 9 lines

Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.

Let me go into a bit of depth about how I arrived at this patch.

First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.

Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.

But the effect is the same. E.g. you have more memory CPU accessible than otherwise.

Quoted 16 lines

Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.

With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.

Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.

Quoted 4 lines

Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.

That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.

Quoted 9 lines

When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.

So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.

Quoted 7 lines
Quoted 7 lines

Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.

Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.

The problem is that this is basically just coincident. On other systems the base address can be completely different.

See function uvd_v4_2_mc_resume() where the mmUVD_LMI_ADDR_EXT and mmUVD_LMI_EXT40_ADDR register is programmed and try to hack those two register writes and see if they really end up in the HW.

Okay, I did a read and compare after each write.

Both writes seem to go through on both the Kaveri and s9150:

Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote 0x00001001, read 0x00001001 [same] amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote 0x800900F4, read 0x800900F4 [same]

s9150:
amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote 0x0000F00F, read 0x0000F00F [same] amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote 0x800900F7, read 0x800900F7 [same]

Thanks,
John

Quoted 37 lines
Quoted 6 lines

I will try to find a Kaveri system which is still working to reproduce the issue.

Thanks,
Christian.

I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.

The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.

The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.

Anyway, hopefully these extra configuration details help narrow down the problem.

Thanks,
John

Quoted 10 lines
Quoted 10 lines

Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().

So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.

Thanks,
John

#6 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

In reply to Paneer Selvam, Arunpravin
Quoted 31 lines · Arunpravin
Quoted 28 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy.  I think it's fair to consider their style as expected.

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

Quoted 8 lines
Quoted 2 lines

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

I will modify this at drm_buddy to match with drm_mm and send the patch.

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Regards,
Arun.

Quoted 30 lines

Regards,
Arun.

Quoted 26 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
           int r;
 
           lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                   lpfn = man->size;
 

          fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#7 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 37 lines · Arunpravin
Quoted 34 lines · Arunpravin
Quoted 30 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy.  I think it's fair to consider their style as expected.

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

Quoted 8 lines
Quoted 5 lines

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

I will modify this at drm_buddy to match with drm_mm and send the patch.

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Quoted 34 lines

Regards,
Arun.

Quoted 30 lines

Regards,
Arun.

Quoted 26 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
           int r;
 
           lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                   lpfn = man->size;
 

          fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
maintainer
#8 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 44 lines · Arunpravin
Quoted 41 lines · Arunpravin
Quoted 38 lines · Arunpravin
Quoted 25 lines · John Olender
Quoted 11 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy.  I think it's fair to consider their style as expected.

Quoted 5 lines

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I will modify this at drm_buddy to match with drm_mm and send the patch.

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.

Alex most likely won't follow the discussion till the end.

Thanks,
Christian.

Quoted 35 lines
Quoted 34 lines

Regards,
Arun.

Quoted 30 lines

Regards,
Arun.

Quoted 26 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

---
   drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
           int r;
 
           lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                   lpfn = man->size;
 

          fpfn = (u64)place->fpfn << PAGE_SHIFT;
--
2.47.2

#9 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 41 lines · Christian König
Quoted 39 lines · Arunpravin
Quoted 37 lines · Arunpravin
Quoted 35 lines · Arunpravin
Quoted 24 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy.  I think it's fair to consider their style as expected.

Quoted 5 lines

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

I will modify this at drm_buddy to match with drm_mm and send the patch.

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.

Alex most likely won't follow the discussion till the end.

Sure Christian, I will merge this patch into amd-staging-drm-next.

Thanks,
Arun.

Quoted 33 lines

Thanks,
Christian.

Quoted 29 lines
Quoted 29 lines

Regards,
Arun.

Quoted 27 lines

Regards,
Arun.

Quoted 25 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
           int r;
 
           lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                   lpfn = man->size;
 

          fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#10 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 45 lines · Arunpravin
Quoted 42 lines · Christian König
Quoted 40 lines · Arunpravin
Quoted 38 lines · Arunpravin
Quoted 36 lines · Arunpravin
Quoted 24 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.

Quoted 5 lines

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

I will modify this at drm_buddy to match with drm_mm and send the patch.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.

Alex most likely won't follow the discussion till the end.

Sure Christian, I will merge this patch into amd-staging-drm-next.

I see a black screen problem during the amdgpu driver load on the tip of amd-staging-drm-next, once that issue is fixed, I will push this patch into amd-staging-drm-next.

Regards,
Arun.

Quoted 37 lines

Thanks,
Arun.

Quoted 33 lines

Thanks,
Christian.

Quoted 29 lines
Quoted 29 lines

Regards,
Arun.

Quoted 27 lines

Regards,
Arun.

Quoted 25 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

---
    drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

    1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
            int r;
 
            lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                    lpfn = man->size;
 

           fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#11 lore raw

Re: [RFC PATCH 1/2] drm/amdgpu: amdgpu_vram_mgr_new(): Clamp lpfn to total vram

Quoted 51 lines · Arunpravin
Quoted 48 lines · Arunpravin
Quoted 45 lines · Christian König
Quoted 43 lines · Arunpravin
Quoted 41 lines · Arunpravin
Quoted 29 lines · Arunpravin
Quoted 27 lines · John Olender
Quoted 10 lines · Arunpravin
Quoted 9 lines · Alex Deucher

+ Christian

Quoted 5 lines · John Olender

The drm_mm allocator tolerated being passed end > mm->size, but the drm_buddy allocator does not.

Restore the pre-buddy-allocator behavior of allowing such placements.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

This looks correct to me.

I was thinking that we should return an error when lpfn > man->size.

Regards,
Arun.

This patch restores the previous behavior in the spirit of "Do not crash the kernel".  The existing uvd placements are pretty clear in their intent and were accepted until the switch to drm_buddy. I think it's fair to consider their style as expected.

With that in mind, I'm not sure amdgpu_vram_mgr is the place this change really belongs.  That is, I think it's worth asking:

Quoted 7 lines
Quoted 2 lines

1) Why does drm_mm accept end > mm->size without complaint?
2) Why doesn't drm_buddy do the same?

I remember that during the development of DRM buddy , we had a discussion with Intel folks and decided to return an error in DRM buddy when end > mm->size. This was done to ensure that, at the driver level,  lpfn has the correct value.

I will modify this at drm_buddy to match with drm_mm and send the patch.

After giving it some thought, I think it is more effective to implement this tolerance at the VRAM manager level and allow the DRM buddy manager to perform a strict validation, as this is necessary for other graphics drivers (e.g., i915).

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

Ok in that case please pick this patch up and make sure that it land in amd-staging-drm-next Arun.

Alex most likely won't follow the discussion till the end.

Sure Christian, I will merge this patch into amd-staging-drm-next.

I see a black screen problem during the amdgpu driver load on the tip of amd-staging-drm-next, once that issue is fixed, I will push this patch into amd-staging-drm-next.

I have merged this patch into amd-staging-drm-next.

Thanks,
Arun.

Quoted 41 lines

Regards,
Arun.

Quoted 37 lines

Thanks,
Arun.

Quoted 33 lines

Thanks,
Christian.

Quoted 29 lines
Quoted 29 lines

Regards,
Arun.

Quoted 27 lines

Regards,
Arun.

Quoted 25 lines

Thanks,
John

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines

---
    drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 2 +-

    1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ renamed from drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +0 -0

gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 2d7f82e98df9..abdc52b0895a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +1 -1
@@ -463,7 +463,7 @@ static int amdgpu_vram_mgr_new(struct
 ttm_resource_manager *man,
            int r;
 
            lpfn = (u64)place->lpfn << PAGE_SHIFT;
-       if (!lpfn)
+       if (!lpfn || lpfn > man->size)
                    lpfn = man->size;
 

           fpfn = (u64)place->fpfn << PAGE_SHIFT;

Signature · 1 line
2.47.2
#20 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

In reply to John Olender

Ping.

Quoted 62 lines · John Olender
Quoted 49 lines · John Olender
Quoted 48 lines · Christian König
Quoted 9 lines

Simply changing the uvd vcpu bo (and therefore the firmware) to always be allocated in vram does *not* solve #3851.

Let me go into a bit of depth about how I arrived at this patch.

First, what sort of system configuration changes result in the uvd init failure? It looks like having a display connected and changing the BAR size have an impact. Next, which kernel change reliably triggers the issue? The change is the switch to the buddy allocator.

Well that is not a resizable BAR, but rather the "VRAM" is just stolen system memory and we completely bypass the BAR to access it.

But the effect is the same. E.g. you have more memory CPU accessible than otherwise.

Quoted 16 lines

Now that the issue can be reliably triggered, where does the error code, -110 / -ETIMEDOUT, come from? It turns out it's in amdgpu_uvd_ring_test_ib(), specifically a timeout while waiting on the ring's fence.

With that out of the way, what allocator-related change happens when a display is connected at startup? The 'stolen_vga_memory' and related bos are created. Adding a one page dummy bo to the same place in the driver can allow a headless configuration to now pass the uvd ring ib test.

Why does having these extra objects allocated result in a change in behavior? Well, the switch to the buddy allocator drastically changes *where* in vram various objects end up being placed. What about the BAR size change? That ends up influencing where the objects are placed too.

Quoted 4 lines

Which objects related to uvd end up being moved around? The uvd code has a function to force its objects into a specific segment after all. Well, it turns out the vcpu bo doesn't go through this function and is therefore being moved around.

That function is there because independent buffers (the message and the feedback for example) needs to be in the same 256MB segment.

Quoted 9 lines

When the system configuration results in a ring ib timeout, the uvd vcpu bo is pinned *outside* the uvd segment. When uvd init succeeds, the uvd vcpu bo is pinned *inside* the uvd segment.

So, it appears there's a relationship between *where* the vcpu bo ends up and the fence timeout. But why does the issue manifest as a ring fence timeout while testing the ib? Unfortunately, I'm unable to find something like a datasheet or developer's guide containing the finer details of uvd.

Quoted 7 lines
Quoted 7 lines

Mhm, there must be something wrong with programming bits 28-31 of the VCPU BO base address.

Forcing the VCPU into the first 256 segment just makes those bits zero and so makes it work on your system.

The problem is that this is basically just coincident. On other systems the base address can be completely different.

See function uvd_v4_2_mc_resume() where the mmUVD_LMI_ADDR_EXT and mmUVD_LMI_EXT40_ADDR register is programmed and try to hack those two register writes and see if they really end up in the HW.

Okay, I did a read and compare after each write.

Both writes seem to go through on both the Kaveri and s9150:

Quoted 8 lines

Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote 0x00001001, read 0x00001001 [same] amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote 0x800900F4, read 0x800900F4 [same]

s9150:
amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote 0x0000F00F, read 0x0000F00F [same] amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote 0x800900F7, read 0x800900F7 [same]

I've also confirmed the patch works fine when segments other than [0, 256M) are used.

E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.

Quoted 41 lines

Thanks,
John

Quoted 37 lines
Quoted 6 lines

I will try to find a Kaveri system which is still working to reproduce the issue.

Thanks,
Christian.

I first saw this issue with a s9150. I had serious reservations about reporting the issue because, in its default configuration, the s9150 has no display output. I needed to figure out that yes, this is a real issue, I didn't just shoot myself in the foot by enabling broken display hardware.

The issue affects all s9150s in a system, occurs in different slots and numa nodes, still occurs when other hardware is added or removed, and follows the s9150 from x399 to a significantly newer b650 system.

The Kaveri iGPU, while also impacted, mainly serves to show that yes, this issue is happening on more than just some dodgy s9150 setup.

Anyway, hopefully these extra configuration details help narrow down the problem.

Thanks,
John

Quoted 10 lines
Quoted 10 lines

Well, what seems related in the code? Where is the ring fence located? It's placed inside the vcpu bo by amdgpu_fence_driver_start_ring().

So, does this patch provide the correct solution to the problem? Maybe not. But the solution seems plausible enough to at least send in the patch for review.

Thanks,
John

maintainer
#21 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Hi John,

first of all thanks a lot for taking a look into this.

Quoted 13 lines
Quoted 4 lines

Okay, I did a read and compare after each write.

Both writes seem to go through on both the Kaveri and s9150:

Quoted 8 lines

Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote 0x00001001, read 0x00001001 [same] amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote 0x800900F4, read 0x800900F4 [same]

s9150:
amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote 0x0000F00F, read 0x0000F00F [same] amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote 0x800900F7, read 0x800900F7 [same]

Quoted 5 lines

I've also confirmed the patch works fine when segments other than [0, 256M) are used.

E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.

Oh, that's a very interesting find. Could you try to turn around the way the patch works?

E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.

That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.

When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.

Thanks,
Christian.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I will try to find a Kaveri system which is still working to reproduce the issue.

I unfortunately couldn't find a working box of hand. Would need to search in our HW stash for a box which still works and get that shipped to me.

And that is overhead for this issue I would rather like to avoid.

So if you can come up with a simpler patch which works for you I'm happy to take that.

Regards,
Christian.

#22 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 25 lines · Christian König

Hi John,

first of all thanks a lot for taking a look into this.

Quoted 13 lines
Quoted 4 lines

Okay, I did a read and compare after each write.

Both writes seem to go through on both the Kaveri and s9150:

Quoted 8 lines

Kaveri (512MB UMA Buffer):
amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF41FA00000, addr=0x00000001, wrote 0x00001001, read 0x00001001 [same] amdgpu 0000:00:01.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF41FA00000, addr=0x000000F4, wrote 0x800900F4, read 0x800900F4 [same]

s9150:
amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_ADDR_EXT: gpu_addr=0xF7FFA00000, addr=0x0000000F, wrote 0x0000F00F, read 0x0000F00F [same] amdgpu 0000:41:00.0: amdgpu: [drm] uvd_v4_2_mc_resume: mmUVD_LMI_EXT40_ADDR: gpu_addr=0xF7FFA00000, addr=0x000000F7, wrote 0x800900F7, read 0x800900F7 [same]

Quoted 5 lines

I've also confirmed the patch works fine when segments other than [0, 256M) are used.

E.g.: Both init and VA-API playback work fine with a UVD segment of [1792M, 2048M) on Kaveri with a 2G UMA buffer.

Oh, that's a very interesting find. Could you try to turn around the way the patch works?

Quoted 9 lines

E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.

That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.

When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.

Thanks,
Christian.

Okay, yeah, that should make for a significantly simpler fix.

Quoted 10 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

I will try to find a Kaveri system which is still working to reproduce the issue.

I unfortunately couldn't find a working box of hand. Would need to search in our HW stash for a box which still works and get that shipped to me.

And that is overhead for this issue I would rather like to avoid.

So if you can come up with a simpler patch which works for you I'm happy to take that.

Regards,
Christian.

Sounds good. I'll get started on this variant.

Thanks,
John

#23 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 4 lines
Quoted 4 lines

Oh, that's a very interesting find. Could you try to turn around the way the patch works?

E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.

I started implementing this and I realized two main problems with this approach.

First, there's currently no guarantee the UVD FW does not cross a 256MB boundary. Checking for this and providing a fallback is going to make this patch... not really any less complex than the original.

Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.

Quoted 4 lines
Quoted 4 lines

That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.

When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.

During my initial investigation, I found out that the UVD FW got placed in the first segment *because* things were already placed there. This is why adding a 'stolen_vga_memory' substitute was an effective workaround.

So, CIK is *already* forced to deal with an overloaded first segment and, with the inverted approach, will continue to do so for typical use cases. Explicitly placing the UVD FW into the first segment just makes this guaranteed.

I did implement a module parameter for testing that allows designating a specific 256MB segment as the legacy UVD segment. I can polish this up so the user has the option to manually relieve some of the first segment pressure on SI and CIK devices.

I haven't run into a situation where I've needed this during normal use, but I can certainly appreciate it being available.

Thanks,
John

maintainer
#24 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 10 lines · John Olender
Quoted 4 lines
Quoted 4 lines

Oh, that's a very interesting find. Could you try to turn around the way the patch works?

E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.

I started implementing this and I realized two main problems with this approach.

First, there's currently no guarantee the UVD FW does not cross a 256MB boundary.

There actually is a guarantee for that.

FW BOs are always allocated contiguously, and the allocation backend makes sure that for example an 16MiB (rounded up) FW is always aligned to a 16MiB boundary.

So FW memory never crosses a segment boundary larger than it's rounded up size. Without that tons of things in our driver would fall apart, not just UVD :)

Checking for this and providing a fallback is going to make

Quoted 4 lines

this patch... not really any less complex than the original.

Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.

That's fine with me. I'm just concerned about that 1% of people where it potentially worked before and we are now breaking it.

Quoted 15 lines
Quoted 5 lines
Quoted 5 lines

That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.

When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.

During my initial investigation, I found out that the UVD FW got placed in the first segment *because* things were already placed there. This is why adding a 'stolen_vga_memory' substitute was an effective workaround.

So, CIK is *already* forced to deal with an overloaded first segment and, with the inverted approach, will continue to do so for typical use cases. Explicitly placing the UVD FW into the first segment just makes this guaranteed.

Quoted 4 lines

I did implement a module parameter for testing that allows designating a specific 256MB segment as the legacy UVD segment. I can polish this up so the user has the option to manually relieve some of the first segment pressure on SI and CIK devices.

The module parameter is fine for your testing, but that is not something I would accept upstream.

As soon as things are not working automatically any more people will start to complain. That we have a manual work around is only feasible to a minority.

Regards,
Christian.

Quoted 6 lines

I haven't run into a situation where I've needed this during normal use, but I can certainly appreciate it being available.

Thanks,
John

#25 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 11 lines · Christian König
Quoted 8 lines · John Olender
Quoted 4 lines
Quoted 4 lines

Oh, that's a very interesting find. Could you try to turn around the way the patch works?

E.g. instead of forcing the UVD FW into the first segment, change amdgpu_uvd_force_into_uvd_segment() so that the BOs are forced into the same segment as the UVD firmware.

I started implementing this and I realized two main problems with this approach.

Quoted 8 lines
Quoted 2 lines

First, there's currently no guarantee the UVD FW does not cross a 256MB boundary.

There actually is a guarantee for that.

FW BOs are always allocated contiguously, and the allocation backend makes sure that for example an 16MiB (rounded up) FW is always aligned to a 16MiB boundary.

So FW memory never crosses a segment boundary larger than it's rounded up size. Without that tons of things in our driver would fall apart, not just UVD :)

Aha, thanks!

Quoted 14 lines

Checking for this and providing a fallback is going to make

Quoted 4 lines

this patch... not really any less complex than the original.

Second, most of time this is just going to end up selecting the first segment anyway. I'll go more into this below.

That's fine with me. I'm just concerned about that 1% of people where it potentially worked before and we are now breaking it.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

That would resolve my concern that this could overload the first segment. The feedback and message BO are usually rather small (4 or 128k IIRC), but the firmware is a couple of megabytes in size.

When we have other FW and VGA emulation buffers in the first segment as well then that could result into clashing that segment to much.

Okay this makes much more sense. I read this as general concern and put the brakes on, especially after missing the above.

I'll resume work on the simplified patch.

Thanks again,
John

#26 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

In reply to Christian König
Quoted 8 lines · Christian König
Quoted 2 lines

I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM

Yeah completely agree. It's a good catch, but the solution is incorrect.

On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.

So something like AMDGPU_GEM_DOMAIN_VRAM | (adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : 0) would be correct I think.

Quoted 9 lines
Quoted 3 lines

If that fixes it, this should be tagged with:

Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernel

allocations")

And CC stable I think.

Regards,
Christian.

Quoted 1 line

Alex

The simplified patch needs this change to the vcpu bo domain. Would you prefer this change as a separate patch?

Thanks,
John

maintainer
#27 lore raw

Re: [RFC PATCH 2/2] drm/amdgpu/uvd: Ensure vcpu bos are within the uvd segment

Quoted 14 lines · John Olender
Quoted 13 lines · Christian König
Quoted 2 lines

I think we can just make this VRAM only. Or something like:
adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : AMDGPU_GEM_DOMAIN_VRAM

Yeah completely agree. It's a good catch, but the solution is incorrect.

On the older UVD MC interface the FW needs to be in VRAM or the validation fails. If it's inside the window for the message and fence is actually irrelevant.

So something like AMDGPU_GEM_DOMAIN_VRAM | (adev->uvd.address_64_bit ? AMDGPU_GEM_DOMAIN_GTT : 0) would be correct I think.

Quoted 3 lines

If that fixes it, this should be tagged with:

Fixes 58ab2c08d708 ("drm/amdgpu: use VRAM|GTT for a bunch of kernel

allocations")

And CC stable I think.

Quoted 7 lines
Quoted 4 lines

Regards,
Christian.

Quoted 1 line

Alex

The simplified patch needs this change to the vcpu bo domain. Would you prefer this change as a separate patch?

Separate one please.

Just send it as the first one in a series.

Regards,
Christian.

Quoted 3 lines

Thanks,
John