drm/amdgpu: Tweaks for high pressure on CPU visible VRAM
Subsystems (2)
cover drm/amdgpu: Tweaks for high pressure on CPU visible VRAM
From: Michel Dänzer <[email protected]>
This series was developed and tested under the following scenario:
Running the PTS dirt-rally benchmark (1920x1080, Ultra) on Tonga with 2G, with CPU visible VRAM artificially restricted to 64 MB.
Without this series, there's a lot of stutter during about the first minute of a benchmark run. During this time there are significant amounts of buffer moves (starting from about 500 MB on the HUD) and evictions, gradually declining until the buffer moves settle around 8 MB on the HUD.
With this series, there's only slight stutter during the first seconds after the car launches, even though the buffer move volume is about the same as without the series. Buffer evictions are eliminated almost completely, except for a few at the beginning. Buffer moves still settle around 8 MB on the HUD, but with less variance than before.
Note that there's only little if any improvement of the average framerate reported, but the minimum framerate as seen on the HUD goes from ~10 fps to ~17.
Patch 1 is a cleanup that I noticed along the way.
Patch 2 makes the main difference for the above scenario.
Patch 3 doesn't make as much difference, I'm fine with it not landing at least for now.
Michel Dänzer (3):
drm/amdgpu: Drop useless loops for placement restrictions drm/amdgpu: Don't evict other BOs from VRAM for page faults drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 42 ++++++++++++--------------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 46 ++++++++++++++++++++---------- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 6 +--- 3 files changed, 51 insertions(+), 43 deletions(-)
Signature · 6 lines
2.11.0 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170518090809.15916-1-michel-otUistvHUpPR7s880joybQ@public.gmane.org
PATCH 1/3 drm/amdgpu: Drop useless loops for placement restrictions
From: Michel Dänzer <[email protected]>
We know how the placements were initialized in these cases, so we can set the restrictions directly without a loop.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 13 +++---------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 19 ++++--------------- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 6 +----- 3 files changed, 8 insertions(+), 30 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +3 -10
@@ -939,8 +939,8 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo) {struct amdgpu_device *adev =amdgpu_ttm_adev (bo->bdev);struct amdgpu_bo *abo; -unsigned long offset, size, lpfn; -int i, r; +unsigned long offset, size; +int r;if (!amdgpu_ttm_bo_is_amdgpu_bo (bo))return 0 ;
@@ -961,14 +961,7 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)/* hurrah the memory is not visible ! */ amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM ); - lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; -for (i =0 ; i < abo->placement.num_placement; i++) { -/* Force into visible VRAM */ -if ((abo->placements[i].flags &TTM_PL_FLAG_VRAM ) && - (!abo->placements[i].lpfn || - abo->placements[i].lpfn > lpfn)) - abo->placements[i].lpfn = lpfn; - } + abo->placements[0 ].lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; r =ttm_bo_validate (bo, &abo->placement, false, false);if (unlikely (r == -ENOMEM )) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT );
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4 -15
@@ -191,7 +191,6 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, .lpfn = 0, .flags = TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM }; -unsigned i;if (!amdgpu_ttm_bo_is_amdgpu_bo (bo)) { placement->placement = &placements;
@@ -209,20 +208,10 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); }else {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT ); -for (i =0 ; i < abo->placement.num_placement; ++i) { -if (!(abo->placements[i].flags & -TTM_PL_FLAG_TT )) -continue ; - -if (abo->placements[i].lpfn) -continue ; - -/* set an upper limit to force directly -* allocating address space for the BO. -*/ - abo->placements[i].lpfn = - adev->mc.gtt_size >>PAGE_SHIFT ; - } +/* Set an upper limit to force directly allocating +* address space for the BO. +*/ + abo->placements[0 ].lpfn = adev->mc.gtt_size >>PAGE_SHIFT ; }break ;case TTM_PL_TT :
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +1 -5
@@ -375,11 +375,7 @@ void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp)static void amdgpu_uvd_force_into_uvd_segment (struct amdgpu_bo *abo) { -int i; -for (i =0 ; i < abo->placement.num_placement; ++i) { - abo->placements[i].fpfn =0 >>PAGE_SHIFT ; - abo->placements[i].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ; - } + abo->placements[0 ].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ; }static u64 amdgpu_uvd_get_addr_from_ctx (struct amdgpu_uvd_cs_ctx *ctx)
Signature · 6 lines
2.11.0 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170518090809.15916-2-michel-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH 1/3] drm/amdgpu: Drop useless loops for placement restrictions
Am 18.05.2017 um 11:08 schrieb Michel Dänzer:
Quoted 85 lines
From: Michel Dänzer <[email protected]>
We know how the placements were initialized in these cases, so we can set the restrictions directly without a loop.
Signed-off-by Michel Dänzer <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 13 +++---------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 19 ++++--------------- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 6 +----- 3 files changed, 8 insertions(+), 30 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +3
-10@@ -939,8 +939,8 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo) {struct amdgpu_device *adev =amdgpu_ttm_adev (bo->bdev);struct amdgpu_bo *abo; -unsigned long offset, size, lpfn; -int i, r; +unsigned long offset, size; +int r;if (!amdgpu_ttm_bo_is_amdgpu_bo (bo))return 0 ;@@ -961,14 +961,7 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)/* hurrah the memory is not visible ! */ amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM ); - lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; -for (i =0 ; i < abo->placement.num_placement; i++) { -/* Force into visible VRAM */ -if ((abo->placements[i].flags &TTM_PL_FLAG_VRAM ) && - (!abo->placements[i].lpfn || - abo->placements[i].lpfn > lpfn)) - abo->placements[i].lpfn = lpfn; - } + abo->placements[0 ].lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; r =ttm_bo_validate (bo, &abo->placement, false, false);if (unlikely (r == -ENOMEM )) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT );drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4
-15@@ -191,7 +191,6 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, .lpfn = 0, .flags = TTM_PL_MASK_CACHING | TTM_PL_FLAG_SYSTEM }; -unsigned i;if (!amdgpu_ttm_bo_is_amdgpu_bo (bo)) { placement->placement = &placements;@@ -209,20 +208,10 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo,amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); }else {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT ); -for (i =0 ; i < abo->placement.num_placement; ++i) { -if (!(abo->placements[i].flags & -TTM_PL_FLAG_TT )) -continue ; - -if (abo->placements[i].lpfn) -continue ; - -/* set an upper limit to force directly -* allocating address space for the BO. -*/ - abo->placements[i].lpfn = - adev->mc.gtt_size >>PAGE_SHIFT ; - } +/* Set an upper limit to force directly allocating +* address space for the BO. +*/ + abo->placements[0 ].lpfn = adev->mc.gtt_size >>PAGE_SHIFT ; }break ;case TTM_PL_TT :drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp) · line 378
Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +1
-5@@ -375,11 +375,7 @@ void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp)static void amdgpu_uvd_force_into_uvd_segment (struct amdgpu_bo *abo) { -int i; -for (i =0 ; i < abo->placement.num_placement; ++i) { - abo->placements[i].fpfn =0 >>PAGE_SHIFT ; - abo->placements[i].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ; - } + abo->placements[0 ].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ;
This is not correct. The restriction applies to all placements, not only the first one.
Christian.
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp) · line 381
Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +0
-0}static u64 amdgpu_uvd_get_addr_from_ctx (struct amdgpu_uvd_cs_ctx *ctx)
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 6c3e7a55-d3b0-fc3f-39f4-e85934119e29-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Re: [PATCH 1/3] drm/amdgpu: Drop useless loops for placement restrictions
Quoted 7 lines · Christian König
Am 18.05.2017 um 11:08 schrieb Michel Dänzer:
Quoted 6 lines
From: Michel Dänzer <[email protected]>
We know how the placements were initialized in these cases, so we can set the restrictions directly without a loop.
Signed-off-by Michel Dänzer <[email protected]>
[...]
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +0
-1@@ -375,11 +375,7 @@ void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp)static void amdgpu_uvd_force_into_uvd_segment (struct amdgpu_bo *abo) { -int i;
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp) · line 378
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +1
-4-for (i =0 ; i < abo->placement.num_placement; ++i) { - abo->placements[i].fpfn =0 >>PAGE_SHIFT ; - abo->placements[i].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ; - } + abo->placements[0 ].lpfn = (256 *1024 *1024 ) >>PAGE_SHIFT ;This is not correct. The restriction applies to all placements, not only the first one.
I see, there can be multiple placements in amdgpu_uvd_cs_pass1, if cmd != 0x0 and != 0x3? I'll drop this hunk then, thanks.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 2/3 drm/amdgpu: Don't evict other BOs from VRAM for page faults
In reply to a message not in the archiveFrom: Michel Dänzer <[email protected]>
If there is no free space in CPU visible VRAM for the faulting BO, move it to GTT instead of evicting other BOs from CPU visible VRAM.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +8 -14
@@ -940,7 +940,6 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)struct amdgpu_device *adev =amdgpu_ttm_adev (bo->bdev);struct amdgpu_bo *abo;unsigned long offset, size; -int r;if (!amdgpu_ttm_bo_is_amdgpu_bo (bo))return 0 ;
@@ -960,22 +959,17 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo)return -EINVAL ;/* hurrah the memory is not visible ! */ -amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM ); +amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); abo->placements[0 ].lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; - r =ttm_bo_validate (bo, &abo->placement, false, false); -if (unlikely (r == -ENOMEM )) { -amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT ); -return ttm_bo_validate (bo, &abo->placement, false, false); - }else if (unlikely (r !=0 )) { -return r; - } - offset = bo->mem.start <<PAGE_SHIFT ; -/* this should never happen */ -if ((offset + size) > adev->mc.visible_vram_size) -return -EINVAL ; +/* Only set GTT as busy placement; if there is no space in CPU visible +* VRAM, move this BO to GTT instead of evicting other BOs +*/ + abo->placement.busy_placement = &abo->placements[1 ]; + abo->placement.num_busy_placement =1 ; -return 0 ; +return ttm_bo_validate (bo, &abo->placement, false, false); }/**
Signature · 6 lines
2.11.0 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 3/3 drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
In reply to a message not in the archiveFrom: Michel Dänzer <[email protected]>
In exchange, move BOs with the AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED flag set to CPU visible VRAM with more force.
For other BOs, this gives another chance to stay in VRAM if they happened to lie in the CPU visible part and another BO needs to go there.
This should allow BOs to stay in VRAM longer in some cases.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 19 ++++++++++++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 27 +++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 5 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +14 -5
@@ -963,11 +963,20 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo) AMDGPU_GEM_DOMAIN_GTT); abo->placements[0 ].lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; -/* Only set GTT as busy placement; if there is no space in CPU visible -* VRAM, move this BO to GTT instead of evicting other BOs -*/ - abo->placement.busy_placement = &abo->placements[1 ]; - abo->placement.num_busy_placement =1 ; +if (abo->flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { +/* Only set VRAM as normal placement; if there is no space in +* CPU visible VRAM, evict other BOs, only fall back to GTT as +* last resort +*/ + abo->placement.num_placement =1 ; + }else { +/* Only set GTT as busy placement; if there is no space in CPU +* visible VRAM, move this BO to GTT instead of evicting other +* BOs +*/ + abo->placement.busy_placement = &abo->placements[1 ]; + abo->placement.num_busy_placement =1 ; + }return ttm_bo_validate (bo, &abo->placement, false, false); }
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +27 -0
@@ -206,7 +206,34 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, adev->mman.buffer_funcs_ring && adev->mman.buffer_funcs_ring->ready == false) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + } else if (adev->mc.visible_vram_size < adev->mc.real_vram_size) { +unsigned fpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; +struct drm_mm_node *node = bo->mem.mm_node; +unsigned long pages_left; + +for (pages_left = bo->mem.num_pages; + pages_left; + pages_left -= node->size, node++) { +if (node->start < fpfn) +break ; + } + +if (!pages_left) + goto gtt; + +/* Try evicting to the CPU inaccessible part of VRAM +* first, but only set GTT as busy placement, so this +* BO will be evicted to GTT rather than causing other +* BOs to be evicted from VRAM +*/ +amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); + abo->placements[0 ].fpfn = fpfn; + abo->placements[0 ].lpfn =0 ; + abo->placement.busy_placement = &abo->placements[1 ]; + abo->placement.num_busy_placement =1 ; } else { +gtt:amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT );/* Set an upper limit to force directly allocating * address space for the BO.
Signature · 6 lines
2.11.0 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170518090809.15916-4-michel-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
Quoted 70 lines · Michel Dänzer
From: Michel Dänzer <[email protected]>
In exchange, move BOs with the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set to CPU visible VRAM with more force.For other BOs, this gives another chance to stay in VRAM if they happened to lie in the CPU visible part and another BO needs to go there.
This should allow BOs to stay in VRAM longer in some cases.
Signed-off-by Michel Dänzer <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 19 ++++++++++++++----- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 27 +++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 5 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +14
-5@@ -963,11 +963,20 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo) AMDGPU_GEM_DOMAIN_GTT); abo->placements[0 ].lpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; -/* Only set GTT as busy placement; if there is no space in CPU visible -* VRAM, move this BO to GTT instead of evicting other BOs -*/ - abo->placement.busy_placement = &abo->placements[1 ]; - abo->placement.num_busy_placement =1 ; +if (abo->flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { +/* Only set VRAM as normal placement; if there is no space in +* CPU visible VRAM, evict other BOs, only fall back to GTT as +* last resort +*/ + abo->placement.num_placement =1 ; + }else { +/* Only set GTT as busy placement; if there is no space in CPU +* visible VRAM, move this BO to GTT instead of evicting other +* BOs +*/ + abo->placement.busy_placement = &abo->placements[1 ]; + abo->placement.num_busy_placement =1 ; + }return ttm_bo_validate (bo, &abo->placement, false, false); }drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +15
-0@@ -206,7 +206,34 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, adev->mman.buffer_funcs_ring && adev->mman.buffer_funcs_ring->ready == false) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + } else if (adev->mc.visible_vram_size < adev->mc.real_vram_size) { +unsigned fpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; +struct drm_mm_node *node = bo->mem.mm_node; +unsigned long pages_left; + +for (pages_left = bo->mem.num_pages; + pages_left; + pages_left -= node->size, node++) { +if (node->start < fpfn) +break ; + } + +if (!pages_left) + goto gtt; +
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 233
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +10
-0+/* Try evicting to the CPU inaccessible part of VRAM +* first, but only set GTT as busy placement, so this +* BO will be evicted to GTT rather than causing other +* BOs to be evicted from VRAM +*/ +amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); + abo->placements[0 ].fpfn = fpfn; + abo->placements[0 ].lpfn =0 ; + abo->placement.busy_placement = &abo->placements[1 ];
Are you sure you want to hardcode the placements index? It'll be dependent on the order set up in amdgpu_ttm_placement_init.
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +2
-0+ abo->placement.num_busy_placement =1 ; }else { +gtt:amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_GTT );/* Set an upper limit to force directly allocating * address space for the BO. Signature · 6 lines
2.11.0 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
--
John Brooks
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170518154346.GA5730-6hIufAJW0g7Gr8qjsLp7YGXnswh1EIUO@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
Quoted 13 lines · John Brooks
Quoted 12 lines · Michel Dänzer
From: Michel Dänzer <[email protected]>
In exchange, move BOs with the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set to CPU visible VRAM with more force.For other BOs, this gives another chance to stay in VRAM if they happened to lie in the CPU visible part and another BO needs to go there.
This should allow BOs to stay in VRAM longer in some cases.
Signed-off-by Michel Dänzer <[email protected]>
[...]
Quoted 28 lines
Quoted 28 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +20
-0@@ -206,7 +206,34 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, adev->mman.buffer_funcs_ring && adev->mman.buffer_funcs_ring->ready == false) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + } else if (adev->mc.visible_vram_size < adev->mc.real_vram_size) { +unsigned fpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; +struct drm_mm_node *node = bo->mem.mm_node; +unsigned long pages_left; + +for (pages_left = bo->mem.num_pages; + pages_left; + pages_left -= node->size, node++) { +if (node->start < fpfn) +break ; + } + +if (!pages_left) + goto gtt; + +/* Try evicting to the CPU inaccessible part of VRAM +* first, but only set GTT as busy placement, so this +* BO will be evicted to GTT rather than causing other +* BOs to be evicted from VRAM +*/
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 233
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0+amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); + abo->placements[0 ].fpfn = fpfn; + abo->placements[0 ].lpfn =0 ; + abo->placement.busy_placement = &abo->placements[1 ];Are you sure you want to hardcode the placements index? It'll be dependent on the order set up in
amdgpu_ttm_placement_init.
Yes, see patch 1. Looping over the placements and testing their contents is silly when we know exactly how they were set up. Or do you mean this code shouldn't call amdgpu_ttm_placement_from_domain at all and just set up the placements itself?
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): b65d8565-47e6-39d7-12f7-f7b60cf9787e-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
Quoted 54 lines · Michel Dänzer
Quoted 13 lines · John Brooks
Quoted 12 lines · Michel Dänzer
From: Michel Dänzer <[email protected]>
In exchange, move BOs with the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set to CPU visible VRAM with more force.For other BOs, this gives another chance to stay in VRAM if they happened to lie in the CPU visible part and another BO needs to go there.
This should allow BOs to stay in VRAM longer in some cases.
Signed-off-by Michel Dänzer <[email protected]>[...]
Quoted 28 lines
Quoted 28 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +20
-0@@ -206,7 +206,34 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, adev->mman.buffer_funcs_ring && adev->mman.buffer_funcs_ring->ready == false) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + } else if (adev->mc.visible_vram_size < adev->mc.real_vram_size) { +unsigned fpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; +struct drm_mm_node *node = bo->mem.mm_node; +unsigned long pages_left; + +for (pages_left = bo->mem.num_pages; + pages_left; + pages_left -= node->size, node++) { +if (node->start < fpfn) +break ; + } + +if (!pages_left) + goto gtt; + +/* Try evicting to the CPU inaccessible part of VRAM +* first, but only set GTT as busy placement, so this +* BO will be evicted to GTT rather than causing other +* BOs to be evicted from VRAM +*/ Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 233
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0+amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); + abo->placements[0 ].fpfn = fpfn; + abo->placements[0 ].lpfn =0 ; + abo->placement.busy_placement = &abo->placements[1 ];Are you sure you want to hardcode the placements index? It'll be dependent on the order set up in
amdgpu_ttm_placement_init.
Quoted 2 lines
Yes, see patch 1. Looping over the placements and testing their contents is silly when we know exactly how they were set up.
Agreed
Quoted 2 lines
Or do you mean this code shouldn't call
amdgpu_ttm_placement_from_domainat all and just set up the placements itself?
Calling amdgpu_ttm_placement_from_domain makes sense. I was just imagining a scenario where code like this that makes assumptions about the ordering of placements in the array would break silently if that order were changed, and you'd have to go about finding the places where integer literals were used to address specific placements.
Quoted 4 lines
Signature · 3 lines
-- Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer
--
John Brooks
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170519014314.GA29857-6hIufAJW0g7Gr8qjsLp7YGXnswh1EIUO@public.gmane.org
Re: [PATCH 3/3] drm/amdgpu: Try evicting from CPU visible to invisible VRAM first
Quoted 60 lines · John Brooks
Quoted 54 lines · Michel Dänzer
Quoted 13 lines · John Brooks
Quoted 12 lines · Michel Dänzer
From: Michel Dänzer <[email protected]>
In exchange, move BOs with the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set to CPU visible VRAM with more force.For other BOs, this gives another chance to stay in VRAM if they happened to lie in the CPU visible part and another BO needs to go there.
This should allow BOs to stay in VRAM longer in some cases.
Signed-off-by Michel Dänzer <[email protected]>[...]
Quoted 28 lines
Quoted 28 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +20
-0@@ -206,7 +206,34 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, adev->mman.buffer_funcs_ring && adev->mman.buffer_funcs_ring->ready == false) {amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_CPU ); + } else if (adev->mc.visible_vram_size < adev->mc.real_vram_size) { +unsigned fpfn = adev->mc.visible_vram_size >>PAGE_SHIFT ; +struct drm_mm_node *node = bo->mem.mm_node; +unsigned long pages_left; + +for (pages_left = bo->mem.num_pages; + pages_left; + pages_left -= node->size, node++) { +if (node->start < fpfn) +break ; + } + +if (!pages_left) + goto gtt; + +/* Try evicting to the CPU inaccessible part of VRAM +* first, but only set GTT as busy placement, so this +* BO will be evicted to GTT rather than causing other +* BOs to be evicted from VRAM +*/ Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c static void amdgpu_evict_flags(struct ttm_buffer_object *bo, · line 233
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +5
-0+amdgpu_ttm_placement_from_domain (abo,AMDGPU_GEM_DOMAIN_VRAM | +AMDGPU_GEM_DOMAIN_GTT ); + abo->placements[0 ].fpfn = fpfn; + abo->placements[0 ].lpfn =0 ; + abo->placement.busy_placement = &abo->placements[1 ];Are you sure you want to hardcode the placements index? It'll be dependent on the order set up in
amdgpu_ttm_placement_init.Quoted 2 lines
Yes, see patch 1. Looping over the placements and testing their contents is silly when we know exactly how they were set up.
Agreed
Quoted 8 lines
Quoted 2 lines
Or do you mean this code shouldn't call
amdgpu_ttm_placement_from_domainat all and just set up the placements itself?Calling
amdgpu_ttm_placement_from_domainmakes sense. I was just imagining a scenario where code like this that makes assumptions about the ordering of placements in the array would break silently if that order were changed, and you'd have to go about finding the places where integer literals were used to address specific placements.
Right, if we make changes to amdgpu_ttm_placement_init, we'll need to audit and possibly update its callers. I think that's reasonable, though others might disagree. :)
Thanks for your feedback.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
I'm glad this is being worked on. However, somewhat to my surprise, this patch series didn't help Dying Light's BO eviction problem. For those who don't know, that game performs very badly in certain areas, and it is correlated with increased TTM eviction rates. Relevant screenshots of gallium HUD and sysprof:
http://www.fastquake.com/images/screen-dlgalliumhud1-20170513-171241.png
http://www.fastquake.com/images/screen-dlsysprof-20170515-225919.png
I noticed last week that adding RADEON_DOMAIN_GTT to the domains in radeonsi (patch: http://www.fastquake.com/files/text/radeon-gtt.txt ) greatly improved performance in these areas, to the tune of about a 30fps increase. Obviously, putting GTT in every buffer's domain is not a proper solution. But it lead me to believe that perhaps the problem wasn't just the swapping of resident BOs, but the creation of new ones that only have VRAM in their domain, and they cause existing BOs to be evicted from visible VRAM unconditionally.
The attached patch assigns GTT as the busy placement for newly created BOs that have the AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED flag, so that they will go to GTT if visible VRAM is full, instead of evicting established BOs. Since there is no way to know what the usage patterns of a new BO will be, we shouldn't evict established BOs (for which we have hypothetically had the opportunity to gather usage data) from visible VRAM for new, unknown BOs.
With this patch I get hugely improved performance in Dying Light just like with the Mesa patch: I observed 30-40fps where I got 14 before, and 60fps where I got 40 before. TTM evictions and bytes moved have dropped to zero where they were exceedingly high before. Buffer evictions no longer dominate the prof trace. Screenshots:
http://www.fastquake.com/images/screen-dl-gtt_busy_only-20170518-192602.png
http://www.fastquake.com/images/screen-dlsysprof-gttpatch-20170518-223200.png
--
John Brooks (Frogging101)
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1495163086-4747-1-git-send-email-john-xq/[email protected]
PATCH 1/1 drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Set GTT as the busy placement for newly created BOs that have the AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED flag, so that they don't cause established BOs to be evicted from visible VRAM.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +18 -0
@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ +if (domain & (AMDGPU_GEM_DOMAIN_VRAM |AMDGPU_GEM_DOMAIN_GTT ) && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { + bo->placement.num_placement =1 ; + bo->placement.num_busy_placement =1 ; + bo->placement.busy_placement = &bo->placement.placement[1 ]; + } +/* Kernel allocation are uninterruptible */ initial_bytes_moved =atomic64_read (&adev->num_bytes_moved);
@@ -484,6 +495,13 @@ int amdgpu_bo_create(struct amdgpu_device *adev,memset (&placements,0 , (AMDGPU_GEM_DOMAIN_MAX +1 ) *sizeof (struct ttm_place )); + +/* New CPU-visible BOs will have GTT set as their busy placement */ +if (domain &AMDGPU_GEM_DOMAIN_VRAM && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { + domain |=AMDGPU_GEM_DOMAIN_GTT ; + } +amdgpu_ttm_placement_init (adev, &placement, placements, domain, flags);
Signature · 6 lines
2.7.4 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1495163086-4747-2-git-send-email-john-xq/[email protected]
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.
You this patch is consistent with my opinion I mentioned in Marek thread (Plan: BO move throttling for visible VRAM evictions).
"I agree on this opinion, from our performance tuning experiment, this case indeed often happen especially under vram memory pressure. redirecting to GTT is better than heavy eviction between VRAM and GTT. But we should get a condition for redirecting (eviction counter?), otherwise, BO have no change back to prefer domain."
You're using busy placement to solve it, then BO is trying its prefer domain first, so it has chance back to prefer domain when memory is free.
Although it cannot prevent evil program occupying memory, it still looks reasonable, since we can gain more from heavy eviction under vram pressure in most cases.
Above all, I vote 'Yes' to your idea, but I also want to listen what others' voice is.
Thanks,
David Zhou
Quoted 32 lines
Signed-off-by John Brooks <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11
-0@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ +if (domain & (AMDGPU_GEM_DOMAIN_VRAM |AMDGPU_GEM_DOMAIN_GTT ) && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { + bo->placement.num_placement =1 ; + bo->placement.num_busy_placement =1 ; + bo->placement.busy_placement = &bo->placement.placement[1 ]; + } +/* Kernel allocation are uninterruptible */ initial_bytes_moved =atomic64_read (&adev->num_bytes_moved);@@ -484,6 +495,13 @@ int amdgpu_bo_create(struct amdgpu_device *adev,memset (&placements,0 , (AMDGPU_GEM_DOMAIN_MAX +1 ) *sizeof (struct ttm_place ));
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 507
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +7
-0+ +/* New CPU-visible BOs will have GTT set as their busy placement */ +if (domain &AMDGPU_GEM_DOMAIN_VRAM && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) { + domain |=AMDGPU_GEM_DOMAIN_GTT ; + } +amdgpu_ttm_placement_init (adev, &placement, placements, domain, flags);
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
In reply to a message not in the archiveQuoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.
This is an interesting idea, but there are some issues with this patch.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 399
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ + if (domain & (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&
Should be something like
if (domain == (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&
otherwise it would also match e.g. BOs with domain == AMDGPU_GEM_DOMAIN_GTT and the AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED flag set (not that this makes sense, but there's nothing to prevent it).
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 404
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0+ flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { + bo->placement.num_placement =1 ; + bo->placement.num_busy_placement =1 ; + bo->placement.busy_placement = &bo->placement.placement[1 ]; + }
The original placements set by amdgpu_fill_placement_to_bo need to be restored before returning from this function, otherwise I suspect such BOs which end up being created in GTT will stay there permanently.
Does the patch still help for Dying Light if you fix this?
The patch as is doesn't seem to make any difference for my dirt-rally test case.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 501
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +4
-0@@ -484,6 +495,13 @@ int amdgpu_bo_create(struct amdgpu_device *adev,memset (&placements,0 , (AMDGPU_GEM_DOMAIN_MAX +1 ) *sizeof (struct ttm_place )); + +/* New CPU-visible BOs will have GTT set as their busy placement */ +if (domain &AMDGPU_GEM_DOMAIN_VRAM && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) {
Make this
if ((domain & AMDGPU_GEM_DOMAIN_VRAM) &&
(flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED)) {to match the coding style.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): b2ecb0f8-cd1b-9a00-6ba6-082674fde776-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 24 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 399
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ + if (domain & (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&Should be something like
Quoted 5 lines
if (domain == (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&otherwise it would also match e.g. BOs with domain ==
AMDGPU_GEM_DOMAIN_GTTand theAMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set (not that this makes sense, but there's nothing to prevent it).
I think it should be like this, which trivially rules out GTT and VRAM|GTT cases:
if (domain == AMDGPU_GEM_DOMAIN_VRAM &&
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A7DjWR8x1WYE=Fgqn7vZ7hyRdQHA3wKo_HcNdZ14KdmkA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 27 lines · Marek Olšák
Quoted 26 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 399
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ + if (domain & (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&Should be something like
if (domain == (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&
Quoted 7 lines
Quoted 3 lines
otherwise it would also match e.g. BOs with domain ==
AMDGPU_GEM_DOMAIN_GTTand theAMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set (not that this makes sense, but there's nothing to prevent it).I think it should be like this, which trivially rules out GTT and VRAM|GTT cases:
if (domain ==AMDGPU_GEM_DOMAIN_VRAM&&
That won't work as is due to the second hunk of the patch, which adds in AMDGPU_GEM_DOMAIN_GTT before calling this function.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
In reply to a message not in the archiveQuoted 31 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 399
Quoted 13 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0@@ -392,6 +392,17 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,#endif amdgpu_fill_placement_to_bo (bo, placement); + +/* This is a new BO that wants to be CPU-visible; set GTT as the busy +* placement so that it does not evict established BOs from visible VRAM. +*/ + if (domain & (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&Should be something like
if (domain == (AMDGPU_GEM_DOMAIN_VRAM | AMDGPU_GEM_DOMAIN_GTT) &&otherwise it would also match e.g. BOs with domain ==
AMDGPU_GEM_DOMAIN_GTTand theAMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag set (not that this makes sense, but there's nothing to prevent it).
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 404
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0+ flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { + bo->placement.num_placement =1 ; + bo->placement.num_busy_placement =1 ; + bo->placement.busy_placement = &bo->placement.placement[1 ]; + }The original placements set by
amdgpu_fill_placement_to_boneed to be restored before returning from this function, otherwise I suspect such BOs which end up being created in GTT will stay there permanently.
I'm curious, what makes you think that the BOs cannot move back to VRAM otherwise? VRAM is still in the placements and prefered/allowed domains, just not in the busy placements.
Quoted 17 lines
Does the patch still help for Dying Light if you fix this?
The patch as is doesn't seem to make any difference for my dirt-rally test case.
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create(struct amdgpu_device *adev, · line 501
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +4
-0@@ -484,6 +495,13 @@ int amdgpu_bo_create(struct amdgpu_device *adev,memset (&placements,0 , (AMDGPU_GEM_DOMAIN_MAX +1 ) *sizeof (struct ttm_place )); + +/* New CPU-visible BOs will have GTT set as their busy placement */ +if (domain &AMDGPU_GEM_DOMAIN_VRAM && + flags &AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED ) {Make this
Quoted 9 lines
if ((domain & AMDGPU_GEM_DOMAIN_VRAM) && (flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED)) {to match the coding style.
Signature · 2 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer
Thanks very much for your feedback. I'll send an updated patch soon.
--
John Brooks
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170519192352.GA21642-6hIufAJW0g7Gr8qjsLp7YGXnswh1EIUO@public.gmane.org
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 7 lines · John Brooks
Quoted 6 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
[...]
Quoted 11 lines
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c int amdgpu_bo_create_restricted(struct amdgpu_device *adev, · line 404
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +5
-0+ flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { + bo->placement.num_placement =1 ; + bo->placement.num_busy_placement =1 ; + bo->placement.busy_placement = &bo->placement.placement[1 ]; + }The original placements set by
amdgpu_fill_placement_to_boneed to be restored before returning from this function, otherwise I suspect such BOs which end up being created in GTT will stay there permanently.
Quoted 3 lines
I'm curious, what makes you think that the BOs cannot move back to VRAM otherwise? VRAM is still in the placements and prefered/allowed domains, just not in the busy placements.
If there is not enough free space when the BO is created, there probably won't be either when it's validated for GPU command streams later.
Quoted 1 line
Quoted 1 line
Does the patch still help for Dying Light if you fix this?
Please test this. The result should tell us whether the problem with Dying Light is really pressure on CPU visible VRAM, or something else.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): d0535946-a10d-3b77-0d94-c52098dce021-otUistvHUpPR7s880joybQ@public.gmane.org
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 26 lines · Michel Dänzer
Quoted 7 lines · John Brooks
Quoted 6 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
[...]
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { + bo->placement.num_placement = 1; + bo->placement.num_busy_placement = 1; + bo->placement.busy_placement = &bo->placement.placement[1]; + }The original placements set by
amdgpu_fill_placement_to_boneed to be restored before returning from this function, otherwise I suspect such BOs which end up being created in GTT will stay there permanently.Quoted 3 lines
I'm curious, what makes you think that the BOs cannot move back to VRAM otherwise? VRAM is still in the placements and prefered/allowed domains, just not in the busy placements.
Quoted 9 lines
If there is not enough free space when the BO is created, there probably won't be either when it's validated for GPU command streams later.
Quoted 1 line
Quoted 1 line
Does the patch still help for Dying Light if you fix this?
Please test this. The result should tell us whether the problem with Dying Light is really pressure on CPU visible VRAM, or something else.
I did some tests. The patch still helps if I restore the old placement values after ttm_bo_init_reserved. But while doing this, I made another observation that throws a wrench into things: It *does* kill performance if I remove AMDGPU_GEM_DOMAIN_GTT from bo->prefered_domains. I think that GTT getting into prefered_domains was what "fixed" the game this whole time and the BO creation was irrelevant. Oh well, it was worth a try.
I'll see what else I can find out.
John
Quoted 4 lines
Signature · 3 lines
-- Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170520032101.GA16371-6hIufAJW0g7Gr8qjsLp7YGXnswh1EIUO@public.gmane.org
Re: [PATCH] drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full
Quoted 37 lines · John Brooks
Quoted 26 lines · Michel Dänzer
Quoted 7 lines · John Brooks
Quoted 6 lines · Michel Dänzer
Quoted 3 lines · John Brooks
Set GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they don't cause established BOs to be evicted from visible VRAM.This is an interesting idea, but there are some issues with this patch.
[...]
Quoted 11 lines
Quoted 10 lines
Review
Quoted 5 lines
(quoted diff) +5
-0+ flags & AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED) { + bo->placement.num_placement = 1; + bo->placement.num_busy_placement = 1; + bo->placement.busy_placement = &bo->placement.placement[1]; + }The original placements set by
amdgpu_fill_placement_to_boneed to be restored before returning from this function, otherwise I suspect such BOs which end up being created in GTT will stay there permanently.Quoted 3 lines
I'm curious, what makes you think that the BOs cannot move back to VRAM otherwise? VRAM is still in the placements and prefered/allowed domains, just not in the busy placements.
Quoted 9 lines
If there is not enough free space when the BO is created, there probably won't be either when it's validated for GPU command streams later.
Quoted 1 line
Quoted 1 line
Does the patch still help for Dying Light if you fix this?
Please test this. The result should tell us whether the problem with Dying Light is really pressure on CPU visible VRAM, or something else.
Quoted 6 lines
I did some tests. The patch still helps if I restore the old placement values after
ttm_bo_init_reserved. But while doing this, I made another observation that throws a wrench into things: It *does* kill performance if I removeAMDGPU_GEM_DOMAIN_GTTfrombo->prefered_domains. I think that GTT getting intoprefered_domainswas what "fixed" the game this whole time and the BO creation was irrelevant.
Ah, right, so the problem I was thinking of was actually caused by the second hunk, not the first one. :) amdgpu_cs_bo_validate re-generates the placements for each command stream according to bo->prefered_domains or bo->allowed_domains.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Where is your "attached" patch?
Marek
Quoted 32 lines · John Brooks
I'm glad this is being worked on. However, somewhat to my surprise, this patch series didn't help Dying Light's BO eviction problem. For those who don't know, that game performs very badly in certain areas, and it is correlated with increased TTM eviction rates. Relevant screenshots of gallium HUD and sysprof:
http://www.fastquake.com/images/screen-dlgalliumhud1-20170513-171241.png
http://www.fastquake.com/images/screen-dlsysprof-20170515-225919.pngI noticed last week that adding
RADEON_DOMAIN_GTTto the domains in radeonsi (patch: http://www.fastquake.com/files/text/radeon-gtt.txt ) greatly improved performance in these areas, to the tune of about a 30fps increase. Obviously, putting GTT in every buffer's domain is not a proper solution. But it lead me to believe that perhaps the problem wasn't just the swapping of resident BOs, but the creation of new ones that only have VRAM in their domain, and they cause existing BOs to be evicted from visible VRAM unconditionally.The attached patch assigns GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they will go to GTT if visible VRAM is full, instead of evicting established BOs. Since there is no way to know what the usage patterns of a new BO will be, we shouldn't evict established BOs (for which we have hypothetically had the opportunity to gather usage data) from visible VRAM for new, unknown BOs.With this patch I get hugely improved performance in Dying Light just like with the Mesa patch: I observed 30-40fps where I got 14 before, and 60fps where I got 40 before. TTM evictions and bytes moved have dropped to zero where they were exceedingly high before. Buffer evictions no longer dominate the prof trace. Screenshots:
http://www.fastquake.com/images/screen-dl-gtt_busy_only-20170518-192602.png
http://www.fastquake.com/images/screen-dlsysprof-gttpatch-20170518-223200.png
Quoted 7 lines
--
John Brooks (Frogging101)_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A6pcm8EiXDRO2kO64TWJRR4BVrzbpiVrLdzhh3uERzTUg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 3 lines · Marek Olšák
Where is your "attached" patch?
Marek
It's actually a reply to my message. Sorry if that was unclear.
Quoted 34 lines · John Brooks
Quoted 32 lines
I'm glad this is being worked on. However, somewhat to my surprise, this patch series didn't help Dying Light's BO eviction problem. For those who don't know, that game performs very badly in certain areas, and it is correlated with increased TTM eviction rates. Relevant screenshots of gallium HUD and sysprof:
http://www.fastquake.com/images/screen-dlgalliumhud1-20170513-171241.png
http://www.fastquake.com/images/screen-dlsysprof-20170515-225919.pngI noticed last week that adding
RADEON_DOMAIN_GTTto the domains in radeonsi (patch: http://www.fastquake.com/files/text/radeon-gtt.txt ) greatly improved performance in these areas, to the tune of about a 30fps increase. Obviously, putting GTT in every buffer's domain is not a proper solution. But it lead me to believe that perhaps the problem wasn't just the swapping of resident BOs, but the creation of new ones that only have VRAM in their domain, and they cause existing BOs to be evicted from visible VRAM unconditionally.The attached patch assigns GTT as the busy placement for newly created BOs that have the
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIREDflag, so that they will go to GTT if visible VRAM is full, instead of evicting established BOs. Since there is no way to know what the usage patterns of a new BO will be, we shouldn't evict established BOs (for which we have hypothetically had the opportunity to gather usage data) from visible VRAM for new, unknown BOs.With this patch I get hugely improved performance in Dying Light just like with the Mesa patch: I observed 30-40fps where I got 14 before, and 60fps where I got 40 before. TTM evictions and bytes moved have dropped to zero where they were exceedingly high before. Buffer evictions no longer dominate the prof trace. Screenshots:
http://www.fastquake.com/images/screen-dl-gtt_busy_only-20170518-192602.png
http://www.fastquake.com/images/screen-dlsysprof-gttpatch-20170518-223200.png
Quoted 7 lines
Quoted 7 lines
--
John Brooks (Frogging101)_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
--
John Brooks
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 20170519152732.GA11484-6hIufAJW0g7Gr8qjsLp7YGXnswh1EIUO@public.gmane.org
Quoted 6 lines · John Brooks
Quoted 3 lines · Marek Olšák
Where is your "attached" patch?
Marek
It's actually a reply to my message. Sorry if that was unclear.
That's OK, but I don't see any patch from you here.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A77Yu9c02Mu-wK3HD_gAKvPbN8q7aEtHcxrcOUhGG4Pfw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 9 lines · Marek Olšák
Quoted 6 lines · John Brooks
Quoted 3 lines · Marek Olšák
Where is your "attached" patch?
Marek
It's actually a reply to my message. Sorry if that was unclear.
That's OK, but I don't see any patch from you here.
Nevermind. I guess it's "drm/amdgpu: Place new CPU-accessbile BOs in GTT if visible VRAM is full", which is a different thread in Gmail.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Hi Michel,
I've applied your series and it doesn't help with low Dirt Rally performance on Fiji. I see TTM buffer moves at 800MB/s and many VRAM page faults.
Marek
Quoted 40 lines · Michel Dänzer
From: Michel Dänzer <[email protected]>
This series was developed and tested under the following scenario:
Running the PTS dirt-rally benchmark (1920x1080, Ultra) on Tonga with 2G, with CPU visible VRAM artificially restricted to 64 MB.
Without this series, there's a lot of stutter during about the first minute of a benchmark run. During this time there are significant amounts of buffer moves (starting from about 500 MB on the HUD) and evictions, gradually declining until the buffer moves settle around 8 MB on the HUD.
With this series, there's only slight stutter during the first seconds after the car launches, even though the buffer move volume is about the same as without the series. Buffer evictions are eliminated almost completely, except for a few at the beginning. Buffer moves still settle around 8 MB on the HUD, but with less variance than before.
Note that there's only little if any improvement of the average framerate reported, but the minimum framerate as seen on the HUD goes from ~10 fps to ~17.
Patch 1 is a cleanup that I noticed along the way.
Patch 2 makes the main difference for the above scenario.
Patch 3 doesn't make as much difference, I'm fine with it not landing at least for now.
Michel Dänzer (3):
drm/amdgpu: Drop useless loops for placement restrictions drm/amdgpu: Don't evict other BOs from VRAM for page faults drm/amdgpu: Try evicting from CPU visible to invisible VRAM firstdrivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 42 ++++++++++++--------------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 46 ++++++++++++++++++++---------- drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c | 6 +--- 3 files changed, 51 insertions(+), 43 deletions(-)
Quoted 7 lines
--
2.11.0_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A65deE+hue7oPMQ+=K5s8TEGZE+bJMNVOdp=LQuq_RnMQ-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 3 lines · Marek Olšák
Hi Michel,
I've applied your series
Thanks for testing it.
Quoted 2 lines
and it doesn't help with low Dirt Rally performance on Fiji. I see TTM buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
Quoted 3 lines
Quoted 3 lines
Note that there's only little if any improvement of the average framerate reported, but the minimum framerate as seen on the HUD goes from ~10 fps to ~17.
I.e. it mostly affects the minimum framerate and smoothness for me as well.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A6ZiNSswhArstqHpYnm30ieoDZj=j_-aFvZz1_ByXLOGA@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAAxE2A4zdgFRh-GDqyDRrXmEWqjSrkfCo=Z41hfo-UNY0E5FhA@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): CAAxE2A4zdgFRh-GDqyDRrXmEWqjSrkfCo=Z41hfo-UNY0E5FhA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected]> wrote:
Quoted 3 lines · Marek Olšák
Hi Michel,
I've applied your series
Thanks for testing it.
Quoted 2 lines
and it doesn't help with low Dirt Rally performance on Fiji. I see TTM buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
Quoted 3 lines
Quoted 3 lines
Note that there's only little if any improvement of the average framerate reported, but the minimum framerate as seen on the HUD goes from ~10 fps to ~17.
I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know. In any case, 30 fps is really bad, so I don't think the series does what you think it does.
Marek
--
Earthling Michel Dänzer | http://www.amd.com
Libre software enthusiast | Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A67T3Mhw68thNewFtcdua3ufSP-FDX_RN153x-aLKUh1Q-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 21 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.
Quoted 8 lines
I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know. In any case, 30 fps is really bad, so I don't think the series does what you think it does.
Ah, yeah, sure. "Doesn't help" sounded like it just didn't improve things. :)
How about with only patches 1 & 2?
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 21 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.
Quoted 6 lines
I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 7649341f-6407-1a12-9674-fb65076125fc-otUistvHUpPR7s880joybQ@public.gmane.org
Quoted 26 lines · Michel Dänzer
Quoted 25 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Quoted 8 lines
Quoted 2 lines
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns. If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown". It's not random with Dying Light.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A446KFb+CiwdZusvdgRSfQS8jLgsCy4K7L0_tc00pvWKA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 44 lines · Marek Olšák
Quoted 26 lines · Michel Dänzer
Quoted 25 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Quoted 8 lines
Quoted 2 lines
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns. If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown". It's not random with Dying Light.
Quoted 1 line
Marek
For what it's worth, the best place to reproduce it in Dying Light is the courtyard where Zere's trailer is. It's the first place you go after leaving the Tower for the first time at the start of the game. The Tower lobby is another good place. Outside of these areas the slowdown may not be apparent at all.
John
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 30 lines · Marek Olšák
Quoted 29 lines · Michel Dänzer
Quoted 27 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Quoted 8 lines
Quoted 5 lines
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns.
No, this thread is about the scenario described in the cover letter of this patch series.
Quoted 4 lines
If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown".
I could say the same thing about you seeing random slowdowns... I've never seen that, I had to artificially limit the size of visible VRAM to 64 MB to make it significantly affect the benchmark result.
How many times do you need to run the benchmark on average to hit a random slowdown? Which desktop environment and other X clients are running during the benchmark? Which tab is active in the Steam window while the benchmark runs?
In my case, it's only xfwm4, xterm and steam on the Dirt Rally page in the library.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): c3dfa73a-e0c1-c6e9-0264-3c52e444c297-otUistvHUpPR7s880joybQ@public.gmane.org
Quoted 53 lines · Michel Dänzer
Quoted 30 lines · Marek Olšák
Quoted 29 lines · Michel Dänzer
Quoted 27 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Quoted 8 lines
Quoted 5 lines
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns.
No, this thread is about the scenario described in the cover letter of this patch series.
Quoted 4 lines
If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown".
I could say the same thing about you seeing random slowdowns... I've never seen that, I had to artificially limit the size of visible VRAM to 64 MB to make it significantly affect the benchmark result.
Quoted 7 lines
How many times do you need to run the benchmark on average to hit a random slowdown? Which desktop environment and other X clients are running during the benchmark? Which tab is active in the Steam window while the benchmark runs?
In my case, it's only xfwm4, xterm and steam on the Dirt Rally page in the library.
Ubuntu Unity, Steam small mode (there are no tabs), Ultra settings in Dirt Rally.
Every single time I run the game with this series, I get 700-1000MB/s of TTM BO moves. There doesn't seem to be any randomness.
It was better without this series. (meaning it was sometimes OK, sometimes bad)
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A7tKvvmZ+qf8f06FJODWNLgHcv-XPiJJgLana-PcXBR_g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 62 lines · Marek Olšák
Quoted 53 lines · Michel Dänzer
Quoted 30 lines · Marek Olšák
Quoted 29 lines · Michel Dänzer
Quoted 27 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Quoted 8 lines
Quoted 5 lines
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns.
No, this thread is about the scenario described in the cover letter of this patch series.
Quoted 4 lines
If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown".
I could say the same thing about you seeing random slowdowns... I've never seen that, I had to artificially limit the size of visible VRAM to 64 MB to make it significantly affect the benchmark result.
Quoted 7 lines
How many times do you need to run the benchmark on average to hit a random slowdown? Which desktop environment and other X clients are running during the benchmark? Which tab is active in the Steam window while the benchmark runs?
In my case, it's only xfwm4, xterm and steam on the Dirt Rally page in the library.
Quoted 7 lines
Ubuntu Unity, Steam small mode (there are no tabs), Ultra settings in Dirt Rally.
Every single time I run the game with this series, I get 700-1000MB/s of TTM BO moves. There doesn't seem to be any randomness.
It was better without this series. (meaning it was sometimes OK, sometimes bad)
Thanks for the additional details. I presume that in the bad case there are some BOs lying around in visible VRAM (e.g. from Unity), which causes some of Dirt Rally's BOs to go back and forth between GTT on CPU page faults and VRAM on GPU usage.
This means at least patch 2 goes out the window. I'll see if I can salvage something out of patch 3.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): f04e8239-e8cb-1c6c-a0ae-581facadb3cf-otUistvHUpPR7s880joybQ@public.gmane.org
Quoted 71 lines · Michel Dänzer
Quoted 62 lines · Marek Olšák
Quoted 53 lines · Michel Dänzer
Quoted 30 lines · Marek Olšák
Quoted 29 lines · Michel Dänzer
Quoted 27 lines · Marek Olšák
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Quoted 8 lines
Quoted 5 lines
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns.
No, this thread is about the scenario described in the cover letter of this patch series.
Quoted 4 lines
If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown".
I could say the same thing about you seeing random slowdowns... I've never seen that, I had to artificially limit the size of visible VRAM to 64 MB to make it significantly affect the benchmark result.
Quoted 7 lines
How many times do you need to run the benchmark on average to hit a random slowdown? Which desktop environment and other X clients are running during the benchmark? Which tab is active in the Steam window while the benchmark runs?
In my case, it's only xfwm4, xterm and steam on the Dirt Rally page in the library.
Quoted 7 lines
Ubuntu Unity, Steam small mode (there are no tabs), Ultra settings in Dirt Rally.
Every single time I run the game with this series, I get 700-1000MB/s of TTM BO moves. There doesn't seem to be any randomness.
It was better without this series. (meaning it was sometimes OK, sometimes bad)
Quoted 7 lines
Thanks for the additional details. I presume that in the bad case there are some BOs lying around in visible VRAM (e.g. from Unity), which causes some of Dirt Rally's BOs to go back and forth between GTT on CPU page faults and VRAM on GPU usage.
This means at least patch 2 goes out the window. I'll see if I can salvage something out of patch 3.
I think the final solution (done in fault_reserve_notify) should be:
if (bo->num_cpu_page_faults++ > 20)
bo->preferred_domain = GTT_WC;
Otherwise I think we'll be just going in circles and not get anywhere.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A5uha-MPx4B=ZeW9kf2qQXOTNcK7sxMJDtEk54S=us47g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Am 24.05.2017 um 13:03 schrieb Marek Olšák:
Quoted 71 lines · Michel Dänzer
Quoted 70 lines · Marek Olšák
Quoted 57 lines · Michel Dänzer
Quoted 49 lines · Marek Olšák
Quoted 29 lines · Michel Dänzer
Quoted 28 lines · Marek Olšák
Quoted 27 lines
On May 20, 2017 3:26 AM, "Michel Dänzer" <[email protected] <mailto:[email protected]>> wrote:
On 20/05/17 01:14 AM, Marek Olšák wrote:
> Hi Michel,
>
> I've applied your seriesThanks for testing it.
> and it doesn't help with low Dirt Rally performance on Fiji. I see TTM > buffer moves at 800MB/s and many VRAM page faults.
Did you see this:
>> Note that there's only little if any improvement of the average framerate
>> reported, but the minimum framerate as seen on the HUD goes from ~10 fps
>> to ~17.I.e. it mostly affects the minimum framerate and smoothness for me as well.
Without the series, I get 70 average fps. With the series, I get 30 average fps. That might just be random bad luck. I don't know.
Quoted 7 lines
Quoted 5 lines
Hmm, yeah, maybe that was just one of the random slowdowns you've been talking about in other threads and on IRC?
I can't reproduce any slowdown with these patches, even leaving visible VRAM size at 256 MB.
The random slowdowns with Dirt Rally are only caused by the pressure on visible VRAM. This whole thread is about those random slowdowns.
No, this thread is about the scenario described in the cover letter of this patch series.
Quoted 4 lines
If you're saying "maybe it was just one of the random slowdowns", you're saying "maybe it was just the visible VRAM pressure". It's only random with Dirt Rally, which makes it difficult to believe statements such as "I can't reproduce any slowdown".
I could say the same thing about you seeing random slowdowns... I've never seen that, I had to artificially limit the size of visible VRAM to 64 MB to make it significantly affect the benchmark result.
Quoted 7 lines
How many times do you need to run the benchmark on average to hit a random slowdown? Which desktop environment and other X clients are running during the benchmark? Which tab is active in the Steam window while the benchmark runs?
In my case, it's only xfwm4, xterm and steam on the Dirt Rally page in the library.
Quoted 7 lines
Ubuntu Unity, Steam small mode (there are no tabs), Ultra settings in Dirt Rally.
Every single time I run the game with this series, I get 700-1000MB/s of TTM BO moves. There doesn't seem to be any randomness.
It was better without this series. (meaning it was sometimes OK, sometimes bad)
Thanks for the additional details. I presume that in the bad case there are some BOs lying around in visible VRAM (e.g. from Unity), which causes some of Dirt Rally's BOs to go back and forth between GTT on CPU page faults and VRAM on GPU usage.
Quoted 5 lines
Quoted 2 lines
This means at least patch 2 goes out the window. I'll see if I can salvage something out of patch 3.
I think the final solution (done in
fault_reserve_notify) should be:
if (bo->num_cpu_page_faults++ > 20)bo->preferred_domain = GTT_WC;
I more or less agree on that, but setting preferred_domain permanently to GTT_WC is what worries me a bit.
E.g. imagine you alt+tab from a game to your browser and back and the game runs way slower now because BOs are never moved back to VRAM.
What we need is a global limit of number of bytes transfered per second for swap operations or something like that.
Or maybe a timeout which says when a BO was moved (either by swapping it out or by a CPU page fault) only move it back after +n jiffies or something like that.
Christian.
Quoted 8 lines
Otherwise I think we'll be just going in circles and not get anywhere.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): eafd4ca6-fbbf-2bb4-be97-16f54fe402dd-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org
Quoted 5 lines · Christian König
Am 24.05.2017 um 13:03 schrieb Marek Olšák:
Quoted 4 lines
Quoted 1 line
I think the final solution (done in
fault_reserve_notify) should be:
if (bo->num_cpu_page_faults++ > 20)bo->preferred_domain = GTT_WC;
I agree something like that will probably be part of the solution, but I doubt it's quite that simple or that it's the only thing that can be improved.
Quoted 5 lines
I more or less agree on that, but setting
preferred_domainpermanently toGTT_WCis what worries me a bit.E.g. imagine you alt+tab from a game to your browser and back and the game runs way slower now because BOs are never moved back to VRAM.
Right, permanently moving a BO to GTT might itself cause performance to drop down a cliff in some cases. It's possible that this is irrelevant compared to excessive buffer migration for CPU access though.
Quoted 6 lines
What we need is a global limit of number of bytes transfered per second for swap operations or something like that.
Or maybe a timeout which says when a BO was moved (either by swapping it out or by a CPU page fault) only move it back after +n jiffies or something like that.
I also feel like something like this will be more useful than the number of CPU page faults per se. But I'm curious what Marek comes up with. :)
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 0535f00f-9095-6209-72e9-e6bb46569a63-otUistvHUpPR7s880joybQ@public.gmane.org
Quoted 26 lines · Michel Dänzer
Quoted 5 lines · Christian König
Am 24.05.2017 um 13:03 schrieb Marek Olšák:
Quoted 4 lines
Quoted 1 line
I think the final solution (done in
fault_reserve_notify) should be:
if (bo->num_cpu_page_faults++ > 20)bo->preferred_domain = GTT_WC;I agree something like that will probably be part of the solution, but I doubt it's quite that simple or that it's the only thing that can be improved.
Quoted 5 lines
I more or less agree on that, but setting
preferred_domainpermanently toGTT_WCis what worries me a bit.E.g. imagine you alt+tab from a game to your browser and back and the game runs way slower now because BOs are never moved back to VRAM.
Right, permanently moving a BO to GTT might itself cause performance to drop down a cliff in some cases. It's possible that this is irrelevant compared to excessive buffer migration for CPU access though.
Quoted 3 lines
What we need is a global limit of number of bytes transfered per second for swap operations or something like that.
Quoted 6 lines
Quoted 3 lines
Or maybe a timeout which says when a BO was moved (either by swapping it out or by a CPU page fault) only move it back after +n jiffies or something like that.
I also feel like something like this will be more useful than the number of CPU page faults per se. But I'm curious what Marek comes up with. :)
I don't have any better idea at the moment. It looks like John Brooks has already solved this issue based on his IRC comments.
Marek
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): CAAxE2A6YFjK1VPuxW=HROqKX+-ejvVJ=YneULp+cDkj033YAbA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Quoted 31 lines · Marek Olšák
Quoted 30 lines · Michel Dänzer
Quoted 5 lines · Christian König
Am 24.05.2017 um 13:03 schrieb Marek Olšák:
Quoted 4 lines
Quoted 1 line
I think the final solution (done in
fault_reserve_notify) should be:
if (bo->num_cpu_page_faults++ > 20)bo->preferred_domain = GTT_WC;I agree something like that will probably be part of the solution, but I doubt it's quite that simple or that it's the only thing that can be improved.
Quoted 5 lines
I more or less agree on that, but setting
preferred_domainpermanently toGTT_WCis what worries me a bit.E.g. imagine you alt+tab from a game to your browser and back and the game runs way slower now because BOs are never moved back to VRAM.
Right, permanently moving a BO to GTT might itself cause performance to drop down a cliff in some cases. It's possible that this is irrelevant compared to excessive buffer migration for CPU access though.
Quoted 6 lines
What we need is a global limit of number of bytes transfered per second for swap operations or something like that.
Or maybe a timeout which says when a BO was moved (either by swapping it out or by a CPU page fault) only move it back after +n jiffies or something like that.
Quoted 5 lines
Quoted 2 lines
I also feel like something like this will be more useful than the number of CPU page faults per se. But I'm curious what Marek comes up with. :)
I don't have any better idea at the moment. It looks like John Brooks has already solved this issue based on his IRC comments.
I don't think there's "the issue" with a single solution. None of John's patches that I've tried so far help for the scenario described in the cover letter of this series.
Signature · 6 lines
Earthling Michel Dänzer | http://www.amd.com Libre software enthusiast | Mesa and X developer _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx