drm/amdgpu/ttm: Consider concurrent VM flushes for buffer entities
Subsystems (2)
PATCH 1/3 drm/amdgpu/ttm: Consider concurrent VM flushes for buffer entities
Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.
Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15 -5
@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,return ; } -/* Navi1x's workaround requires us to limit to a single SDMA sched -* for ttm. -*/ hub = &adev->vmhub[AMDGPU_GFXHUB (0 )]; - adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ? -1 : n; + +/* +* Allow using multiple SDMA schedulers only on GPUs where +* we are allowed to do concurrent VM flushes. +* This consideration is necessary because all GART windows +* are mapped in VMID 0 (the kernel VMID) so each buffer +* entity would flush VMID 0 concurrently. +* +* Also consider the SDMA invalidation workaround on +* Navi 1x GPUs, which also prevents us from using +* multiple SDMA engines on VMID 0 at the same time. +*/ + adev->mman.num_buffer_funcs_scheds = + (adev->vm_manager.concurrent_flush && + !hub->sdma_invalidation_workaround) ? n :1 ; }#if defined(CONFIG_DEBUG_FS )
Signature · 1 line
2.55.0
PATCH 2/3 drm/amdgpu/ttm: Make buffer copy source readonly
When amdgpu is moving a BO from a location in VRAM to another location in VRAM, we map both BOs in the GART and perform the copy operation using the GART addresses.
We are not writing into the first GART window, so let's set the PTE flags to make it readonly. This doesn't fix any known issues but let's do it just to be safe.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +7 -4
@@ -199,7 +199,8 @@ static int amdgpu_ttm_map_buffer(struct amdgpu_ttm_buffer_entity *entity,struct ttm_resource *mem, struct amdgpu_res_cursor *mm_cur, unsigned int window, - bool tmz, uint64_t *size, uint64_t *addr) + bool tmz, uint64_t *size, uint64_t *addr, + bool readonly) { struct amdgpu_device *adev = amdgpu_ttm_adev(bo->bdev);unsigned int offset, num_pages, num_dw, num_bytes;
@@ -260,6 +261,8 @@ static int amdgpu_ttm_map_buffer(struct amdgpu_ttm_buffer_entity *entity, flags =amdgpu_ttm_tt_pte_flags (adev, bo->ttm, mem);if (tmz) flags |=AMDGPU_PTE_TMZ ; +if (readonly) + flags &= ~AMDGPU_PTE_WRITEABLE ; cpu_addr = &job->ibs[0 ].ptr[num_dw];
@@ -329,12 +332,12 @@ static int amdgpu_ttm_copy_mem_to_mem(struct amdgpu_device *adev,/* Map src to window 0 and dst to window 1. */ r =amdgpu_ttm_map_buffer (entity, src->bo, src->mem, &src_mm, -0 , tmz, &cur_size, &from); +0 , tmz, &cur_size, &from, true);if (r) goto error; r =amdgpu_ttm_map_buffer (entity, dst->bo, dst->mem, &dst_mm, -1 , tmz, &cur_size, &to); +1 , tmz, &cur_size, &to, false);if (r) goto error;
@@ -2593,7 +2596,7 @@ int amdgpu_ttm_clear_buffer(struct amdgpu_ttm_buffer_entity *entity, cur_size =min (dst.size,256ULL <<20 ); r =amdgpu_ttm_map_buffer (entity, &bo->tbo, bo->tbo.resource, &dst, -0 , false, &cur_size, &to); +0 , false, &cur_size, &to, false);if (r) goto error;
Signature · 1 line
2.55.0
PATCH 3/3 drm/amdgpu/ttm: Use more optimal copy packet sizes for copy and fill
In reply to Timur KristófCurrently when amdgpu copies or fills a buffer, it uses the maximum byte count supported by the copy engine (SDMA). This is problematic when the maximum byte count is not aligned to 256 bytes because it then can't use all memory channels optimally and can cause the SDMA to operate in its slower byte mode (as opposed to the faster dword mode).
For example, when copying a 10 MiB buffer on SDMA v2.4, we get 5 packets copying 2097151 bytes and 1 packet copying the remaining 5 bytes. All 6 packets are misaligned and operate in byte mode.
For this example, the optimal solution would be to have 5 packets each copying 2096896 bytes and 1 last packet to copy the remaining 1280 bytes, in which case all 6 packets are aligned to 256 bytes and operate in dword mode.
Let's use the following scheme from now on:
When byte count is dword-aligned and fits a single packet, just emit a single packet.
Otherwise, align the copy packet size down to 256 bytes for optimal use of memory channels and to ensure the HW can use the dword mode.
This assumes that the starting addresses of BOs are always dword aligned, which should be the case for every copy operation in the kernel, because the kernel always copies pages.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 27 +++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +25 -2
@@ -2464,6 +2464,27 @@ static int amdgpu_ttm_prepare_job(struct amdgpu_device *adev, DMA_RESV_USAGE_BOOKKEEP); } +static int amdgpu_calc_bytes_per_packet (u32 max_bytes_per_packet, +u32 byte_count) +{ +/* Byte count is dword-aligned and fits a single packet */ +if (!(byte_count &0x3 ) && byte_count <= max_bytes_per_packet) +return max_bytes_per_packet; + +/* +* Align down maximum byte count to 256 bytes so that +* the copy optimally uses all memory channels and +* also to ensure that SDMA can use its dword mode, which +* is faster. +* +* This assumes that the starting addresses of BOs are always +* dword aligned, which should be the case for every copy +* operation in the kernel, because the kernel always copies +* pages. +*/ +return ALIGN_DOWN (max_bytes_per_packet,SZ_256 ); +} + int amdgpu_copy_buffer(struct amdgpu_device *adev, struct amdgpu_ttm_buffer_entity *entity, uint64_t src_offset,
@@ -2487,7 +2508,8 @@ int amdgpu_copy_buffer(struct amdgpu_device *adev,return -EINVAL ; } - max_bytes = adev->mman.buffer_funcs->copy_max_bytes; + max_bytes =amdgpu_calc_bytes_per_packet (adev->mman.buffer_funcs->copy_max_bytes, + byte_count); num_loops =DIV_ROUND_UP (byte_count, max_bytes); num_dw =ALIGN (num_loops * adev->mman.buffer_funcs->copy_num_dw,8 ); r = amdgpu_ttm_prepare_job(adev, entity, num_dw,
@@ -2531,7 +2553,8 @@ static int amdgpu_ttm_fill_mem(struct amdgpu_device *adev,unsigned int i;int r; - max_bytes = adev->mman.buffer_funcs->fill_max_bytes; + max_bytes =amdgpu_calc_bytes_per_packet (adev->mman.buffer_funcs->fill_max_bytes, + byte_count); num_loops =DIV_ROUND_UP_ULL (byte_count, max_bytes); num_dw =ALIGN (num_loops * adev->mman.buffer_funcs->fill_num_dw,8 ); r = amdgpu_ttm_prepare_job(adev, entity, num_dw, resv,
Signature · 1 line
2.55.0
Quoted 6 lines · Timur Kristóf
Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.
Quoted 7 lines
Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.
Fixes 01c836788b37 ("drm/amdgpu: pass all the sdma scheds to amdgpu_mman")Fixes e4029f7a9474 ("drm/amdgpu: only use working sdma schedulers for ttm")Cc Pierre-Eric Pelloux-Prayer <[email protected]>Signed-off-by Timur Kristóf <[email protected]>
I'm not sure if that is actually fixing anything, but better safe than sorry.
Quoted 36 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15
-5@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,return ; } -/* Navi1x's workaround requires us to limit to a single SDMA sched -* for ttm. -*/ hub = &adev->vmhub[AMDGPU_GFXHUB (0 )]; - adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ? -1 : n; + +/* +* Allow using multiple SDMA schedulers only on GPUs where +* we are allowed to do concurrent VM flushes. +* This consideration is necessary because all GART windows +* are mapped in VMID 0 (the kernel VMID) so each buffer +* entity would flush VMID 0 concurrently. +* +* Also consider the SDMA invalidation workaround on +* Navi 1x GPUs, which also prevents us from using +* multiple SDMA engines on VMID 0 at the same time. +*/ + adev->mman.num_buffer_funcs_scheds = + (adev->vm_manager.concurrent_flush && + !hub->sdma_invalidation_workaround) ? n :1 ; }#if defined(CONFIG_DEBUG_FS )
Alex Deucher
AppliedApplied the series. Thanks!
Quoted 53 lines · Timur Kristóf
Allow using multiple SDMA schedulers only on GPUs where we are allowed to do concurrent VM flushes. This consideration is necessary because all GART windows are mapped in VMID 0 (the kernel VMID) so each buffer entity would flush VMID 0 concurrently.
Practically this means that we can't use multiple SDMA engines for TTM on GFX6-8 and Navi 1x.
Fixes 01c836788b37 ("drm/amdgpu: pass all the sdma scheds to amdgpu_mman")Fixes e4029f7a9474 ("drm/amdgpu: only use working sdma schedulers for ttm")Cc Pierre-Eric Pelloux-Prayer <[email protected]>Signed-off-by Timur Kristóf <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15
-6@@ -2685,12 +2685,22 @@ void amdgpu_sdma_set_buffer_funcs_scheds(struct amdgpu_device *adev,return ; } -/* Navi1x's workaround requires us to limit to a single SDMA sched -* for ttm. -*/ hub = &adev->vmhub[AMDGPU_GFXHUB (0 )]; - adev->mman.num_buffer_funcs_scheds = hub->sdma_invalidation_workaround ? -1 : n; + +/* +* Allow using multiple SDMA schedulers only on GPUs where +* we are allowed to do concurrent VM flushes. +* This consideration is necessary because all GART windows +* are mapped in VMID 0 (the kernel VMID) so each buffer +* entity would flush VMID 0 concurrently. +* +* Also consider the SDMA invalidation workaround on +* Navi 1x GPUs, which also prevents us from using +* multiple SDMA engines on VMID 0 at the same time. +*/ + adev->mman.num_buffer_funcs_scheds = + (adev->vm_manager.concurrent_flush && + !hub->sdma_invalidation_workaround) ? n :1 ; }#if defined(CONFIG_DEBUG_FS ) --2.55.0