Skip to contentSkip to conversation

drm/amdgpu/ttm: Consider concurrent VM flushes for buffer entities

PATCH Applied Timur Kristóf 3 participants 5 messages Jul 11 – Jul 13

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/3 drm/amdgpu/ttm: Consider concurrent VM flushes for buffer entities

Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.

Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.

Fixes 01c836788b37 ("drm/amdgpu: pass all the sdma scheds to amdgpu_mman")
Fixes e4029f7a9474 ("drm/amdgpu: only use working sdma schedulers for ttm")
Cc Pierre-Eric Pelloux-Prayer <[email protected]>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15 -5
index 74ada995bd8e..9eb55df6e692 100644
@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,
 		return;
 	}
 
-	/* Navi1x's workaround requires us to limit to a single SDMA sched
-	 * for ttm.
-	 */
 	hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
-	adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ?
-		1 : n;
+
+	/*
+	 * Allow using multiple SDMA schedulers only on GPUs where
+	 * we are allowed to do concurrent VM flushes.
+	 * This consideration is necessary because all GART windows
+	 * are mapped in VMID 0 (the kernel VMID) so each buffer
+	 * entity would flush VMID 0 concurrently.
+	 *
+	 * Also consider the SDMA invalidation workaround on
+	 * Navi 1x GPUs, which also prevents us from using
+	 * multiple SDMA engines on VMID 0 at the same time.
+	 */
+	adev->mman.num_buffer_funcs_scheds =
+		(adev->vm_manager.concurrent_flush &&
+		 !hub->sdma_invalidation_workaround) ? n : 1;
 }
 
 #if defined(CONFIG_DEBUG_FS)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 2/3 drm/amdgpu/ttm: Make buffer copy source readonly

When amdgpu is moving a BO from a location in VRAM to another location in VRAM, we map both BOs in the GART and perform the copy operation using the GART addresses.

We are not writing into the first GART window, so let's set the PTE flags to make it readonly. This doesn't fix any known issues but let's do it just to be safe.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +7 -4
index 9eb55df6e692..50725bd2448d 100644
@@ -199,7 +199,8 @@ static int amdgpu_ttm_map_buffer(struct amdgpu_ttm_buffer_entity *entity,
 				 struct ttm_resource *mem,
 				 struct amdgpu_res_cursor *mm_cur,
 				 unsigned int window,
-				 bool tmz, uint64_t *size, uint64_t *addr)
+				 bool tmz, uint64_t *size, uint64_t *addr,
+				 bool readonly)
 {
 	struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev);
 	unsigned int offset, num_pages, num_dw, num_bytes;
@@ -260,6 +261,8 @@ static int amdgpu_ttm_map_buffer(struct amdgpu_ttm_buffer_entity *entity,
 	flags = amdgpu_ttm_tt_pte_flags(adev, bo->ttm, mem);
 	if (tmz)
 		flags |= AMDGPU_PTE_TMZ;
+	if (readonly)
+		flags &= ~AMDGPU_PTE_WRITEABLE;
 
 	cpu_addr = &job->ibs[0].ptr[num_dw];
 
@@ -329,12 +332,12 @@ static int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,
 
 		/* Map src to window 0 and dst to window 1. */
 		r = amdgpu_ttm_map_buffer(entity, src->bo, src->mem, &src_mm,
-					  0, tmz, &cur_size, &from);
+					  0, tmz, &cur_size, &from, true);
 		if (r)
 			goto error;
 
 		r = amdgpu_ttm_map_buffer(entity, dst->bo, dst->mem, &dst_mm,
-					  1, tmz, &cur_size, &to);
+					  1, tmz, &cur_size, &to, false);
 		if (r)
 			goto error;
 
@@ -2593,7 +2596,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_ttm_buffer_entity *entity,
 		cur_size = min(dst.size, 256ULL << 20);
 
 		r = amdgpu_ttm_map_buffer(entity, &bo->tbo, bo->tbo.resource, &dst,
-					  0, false, &cur_size, &to);
+					  0, false, &cur_size, &to, false);
 		if (r)
 			goto error;
 
Signature · 1 line
2.55.0
#2 lore raw

PATCH 3/3 drm/amdgpu/ttm: Use more optimal copy packet sizes for copy and fill

In reply to Timur Kristóf

Currently when amdgpu copies or fills a buffer, it uses the maximum byte count supported by the copy engine (SDMA). This is problematic when the maximum byte count is not aligned to 256 bytes because it then can't use all memory channels optimally and can cause the SDMA to operate in its slower byte mode (as opposed to the faster dword mode).

For example, when copying a 10 MiB buffer on SDMA v2.4, we get 5 packets copying 2097151 bytes and 1 packet copying the remaining 5 bytes. All 6 packets are misaligned and operate in byte mode.
For this example, the optimal solution would be to have 5 packets each copying 2096896 bytes and 1 last packet to copy the remaining 1280 bytes, in which case all 6 packets are aligned to 256 bytes and operate in dword mode.

Let's use the following scheme from now on:

When byte count is dword-aligned and fits a single packet, just emit a single packet.

Otherwise, align the copy packet size down to 256 bytes for optimal use of memory channels and to ensure the HW can use the dword mode.

This assumes that the starting addresses of BOs are always dword aligned, which should be the case for every copy operation in the kernel, because the kernel always copies pages.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 27 +++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +25 -2
index 50725bd2448d..42ed02e7cd85 100644
@@ -2464,6 +2464,27 @@ static int amdgpu_ttm_prepare_job(struct amdgpu_device *adev,
 						   DMA_RESV_USAGE_BOOKKEEP);
 }
 
+static int amdgpu_calc_bytes_per_packet(u32 max_bytes_per_packet,
+					u32 byte_count)
+{
+	/* Byte count is dword-aligned and fits a single packet */
+	if (!(byte_count & 0x3) && byte_count <= max_bytes_per_packet)
+		return max_bytes_per_packet;
+
+	/*
+	 * Align down maximum byte count to 256 bytes so that
+	 * the copy optimally uses all memory channels and
+	 * also to ensure that SDMA can use its dword mode, which
+	 * is faster.
+	 *
+	 * This assumes that the starting addresses of BOs are always
+	 * dword aligned, which should be the case for every copy
+	 * operation in the kernel, because the kernel always copies
+	 * pages.
+	 */
+	return ALIGN_DOWN(max_bytes_per_packet, SZ_256);
+}
+
 int amdgpu_copy_buffer(struct amdgpu_device *adev,
 		       struct amdgpu_ttm_buffer_entity *entity,
 		       uint64_t src_offset,
@@ -2487,7 +2508,8 @@ int amdgpu_copy_buffer(struct amdgpu_device *adev,
 		return -EINVAL;
 	}
 
-	max_bytes = adev->mman.buffer_funcs->copy_max_bytes;
+	max_bytes = amdgpu_calc_bytes_per_packet(adev->mman.buffer_funcs->copy_max_bytes,
+						 byte_count);
 	num_loops = DIV_ROUND_UP(byte_count, max_bytes);
 	num_dw = ALIGN(num_loops * adev->mman.buffer_funcs->copy_num_dw, 8);
 	r = amdgpu_ttm_prepare_job(adev, entity, num_dw,
@@ -2531,7 +2553,8 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_device *adev,
 	unsigned int i;
 	int r;
 
-	max_bytes = adev->mman.buffer_funcs->fill_max_bytes;
+	max_bytes = amdgpu_calc_bytes_per_packet(adev->mman.buffer_funcs->fill_max_bytes,
+						 byte_count);
 	num_loops = DIV_ROUND_UP_ULL(byte_count, max_bytes);
 	num_dw = ALIGN(num_loops * adev->mman.buffer_funcs->fill_num_dw, 8);
 	r = amdgpu_ttm_prepare_job(adev, entity, num_dw, resv,
Signature · 1 line
2.55.0
maintainer Reviewed-by on v1 1/3
#3 lore raw
In reply to Timur Kristóf
Quoted 6 lines · Timur Kristóf

Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.

Quoted 7 lines

Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.

Fixes 01c836788b37 ("drm/amdgpu: pass all the sdma scheds to amdgpu_mman")
Fixes e4029f7a9474 ("drm/amdgpu: only use working sdma schedulers for ttm")
Cc Pierre-Eric Pelloux-Prayer <[email protected]>

I'm not sure if that is actually fixing anything, but better safe than sorry.

Quoted 36 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15 -5
index 74ada995bd8e..9eb55df6e692 100644
@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,
 		return;
 	}
 
-	/* Navi1x's workaround requires us to limit to a single SDMA sched
-	 * for ttm.
-	 */
 	hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
-	adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ?
-		1 : n;
+
+	/*
+	 * Allow using multiple SDMA schedulers only on GPUs where
+	 * we are allowed to do concurrent VM flushes.
+	 * This consideration is necessary because all GART windows
+	 * are mapped in VMID 0 (the kernel VMID) so each buffer
+	 * entity would flush VMID 0 concurrently.
+	 *
+	 * Also consider the SDMA invalidation workaround on
+	 * Navi 1x GPUs, which also prevents us from using
+	 * multiple SDMA engines on VMID 0 at the same time.
+	 */
+	adev->mman.num_buffer_funcs_scheds =
+		(adev->vm_manager.concurrent_flush &&
+		 !hub->sdma_invalidation_workaround) ? n : 1;
 }
 
 #if defined(CONFIG_DEBUG_FS)
Applied
#4 lore raw
In reply to Timur Kristóf

Applied the series. Thanks!

Quoted 53 lines · Timur Kristóf

Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.

Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.

Fixes 01c836788b37 ("drm/amdgpu: pass all the sdma scheds to amdgpu_mman")
Fixes e4029f7a9474 ("drm/amdgpu: only use working sdma schedulers for ttm")
Cc Pierre-Eric Pelloux-Prayer <[email protected]>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15 -6
index 74ada995bd8e..9eb55df6e692 100644
@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,
                return;
        }
 
-       /* Navi1x's workaround requires us to limit to a single SDMA sched
-        * for ttm.
-        */
        hub = &adev->vmhub[AMDGPU_GFXHUB(0)];
-       adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ?
-               1 : n;
+
+       /*
+        * Allow using multiple SDMA schedulers only on GPUs where
+        * we are allowed to do concurrent VM flushes.
+        * This consideration is necessary because all GART windows
+        * are mapped in VMID 0 (the kernel VMID) so each buffer
+        * entity would flush VMID 0 concurrently.
+        *
+        * Also consider the SDMA invalidation workaround on
+        * Navi 1x GPUs, which also prevents us from using
+        * multiple SDMA engines on VMID 0 at the same time.
+        */
+       adev->mman.num_buffer_funcs_scheds =
+               (adev->vm_manager.concurrent_flush &&
+                !hub->sdma_invalidation_workaround) ? n : 1;
 }
 
 #if defined(CONFIG_DEBUG_FS)
--

2.55.0