drm/amdgpu: set sched_hw_submission higher for KIQ
Subsystems (2)
PATCH 1/1 drm/amdgpu: set sched_hw_submission higher for KIQ
KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12 -2
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, unsigned irq_type) {int r; +int sched_hw_submission = amdgpu_sched_hw_submission; + +/* Set the hw submission limit higher for KIQ because +* it's used for a number of gfx/compute tasks by both +* KFD and KGD which may have outstanding fences and +* it doesn't really use the gpu scheduler anyway; +* KIQ tasks get submitted directly to the ring. +*/ +if (ring->funcs->type ==AMDGPU_RING_TYPE_KIQ ) + sched_hw_submission *=2 ;if (ring->adev ==NULL ) {if (adev->num_rings >=AMDGPU_MAX_RINGS )
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, ring->idx = adev->num_rings++; adev->rings[ring->idx] = ring; r =amdgpu_fence_driver_init_ring (ring, - amdgpu_sched_hw_submission); + sched_hw_submission);if (r)return r; }
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, } ring->ring_size =roundup_pow_of_two (max_dw *4 * - amdgpu_sched_hw_submission); + sched_hw_submission); ring->buf_mask = (ring->ring_size /4 ) -1 ; ring->ptr_mask = ring->funcs->support_64bit_ptrs ?
Signature · 6 lines
2.5.5 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1503434972-27760-1-git-send-email-alexander.deucher-5C7GfCeVMHo@public.gmane.org
Hi Alex,
See below comment about coding style of 80 characters per line.
Thanks,
Alex Bin Xie
________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <[email protected]>
Sent: Tuesday, August 22, 2017 4:49 PM
To: [email protected]
Cc: Deucher, Alexander
Subject: [PATCH] drm/amdgpu: set sched_hw_submission higher for KIQ
KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +11 -1
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, unsigned irq_type) {int r; +int sched_hw_submission = amdgpu_sched_hw_submission; + +/* Set the hw submission limit higher for KIQ because +* it's used for a number of gfx/compute tasks by both +* KFD and KGD which may have outstanding fences and +* it doesn't really use the gpu scheduler anyway; +* KIQ tasks get submitted directly to the ring. +*/ +if (ring->funcs->type ==AMDGPU_RING_TYPE_KIQ ) + sched_hw_submission *=2 ;if (ring->adev ==NULL ) {if (adev->num_rings >=AMDGPU_MAX_RINGS )
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, ring->idx = adev->num_rings++; adev->rings[ring->idx] = ring; r =amdgpu_fence_driver_init_ring (ring, - amdgpu_sched_hw_submission); + sched_hw_submission);
Alex Bin Xie: With shorter variable name, there is no need to wrap into a new line of code.
if (r)
return r;
}
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
}
ring->ring_size = roundup_pow_of_two(max_dw * 4 *- amdgpu_sched_hw_submission);
+ sched_hw_submission);
Alex Bin Xie: With shorter variable name, there is no need to wrap into a new line of code.
ring->buf_mask = (ring->ring_size / 4) - 1;
ring->ptr_mask = ring->funcs->support_64bit_ptrs ?--
2.5.5
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - freedesktop.org<https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
Subscribing to amd-gfx: Subscribe to amd-gfx by filling out the following form. Use of all freedesktop.org lists is subject to our Code of ...
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Thanks Alex!
Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.
Regards,
Felix
Quoted 44 lines · Alex Deucher
KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However,
amdgpu_sched_hw_submission(which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.Signed-off-by Alex Deucher <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +11
-1@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, unsigned irq_type) {int r; +int sched_hw_submission = amdgpu_sched_hw_submission; + +/* Set the hw submission limit higher for KIQ because +* it's used for a number of gfx/compute tasks by both +* KFD and KGD which may have outstanding fences and +* it doesn't really use the gpu scheduler anyway; +* KIQ tasks get submitted directly to the ring. +*/ +if (ring->funcs->type ==AMDGPU_RING_TYPE_KIQ ) + sched_hw_submission *=2 ;if (ring->adev ==NULL ) {if (adev->num_rings >=AMDGPU_MAX_RINGS )@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, ring->idx = adev->num_rings++; adev->rings[ring->idx] = ring; r =amdgpu_fence_driver_init_ring (ring, - amdgpu_sched_hw_submission); + sched_hw_submission);if (r)return r; }
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +1
-1@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, } ring->ring_size =roundup_pow_of_two (max_dw *4 * - amdgpu_sched_hw_submission); + sched_hw_submission); ring->buf_mask = (ring->ring_size /4 ) -1 ; ring->ptr_mask = ring->funcs->support_64bit_ptrs ?
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 943f41cb-e69d-6355-77b6-a3c992820dcb-5C7GfCeVMHo@public.gmane.org
Quoted 4 lines · Felix Kuehling
Thanks Alex!
Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.
I'm not sure what the best number is. Up to 8 KFD processes is common (beyond that performance drops off due to VMID availability) but I'm not sure how often they would need to submit to KIQ concurrently. If it's not expensive I'd just bump it up to say 16.
The performance problem isn't that bad since all the KIQ requests are serialized but the dmesg spam is not nice. Perhaps lowering the severity of the 'rcu slot is busy' message would address that as well?
Quoted 56 lines
Regards,
FelixQuoted 50 lines · Alex Deucher
KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However,
amdgpu_sched_hw_submission(which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.Signed-off-by Alex Deucher <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12
-2@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, unsigned irq_type) {int r; +int sched_hw_submission = amdgpu_sched_hw_submission; + +/* Set the hw submission limit higher for KIQ because +* it's used for a number of gfx/compute tasks by both +* KFD and KGD which may have outstanding fences and +* it doesn't really use the gpu scheduler anyway; +* KIQ tasks get submitted directly to the ring. +*/ +if (ring->funcs->type ==AMDGPU_RING_TYPE_KIQ ) + sched_hw_submission *=2 ;if (ring->adev ==NULL ) {if (adev->num_rings >=AMDGPU_MAX_RINGS )@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, ring->idx = adev->num_rings++; adev->rings[ring->idx] = ring; r =amdgpu_fence_driver_init_ring (ring, - amdgpu_sched_hw_submission); + sched_hw_submission);if (r)return r; }@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, } ring->ring_size =roundup_pow_of_two (max_dw *4 * - amdgpu_sched_hw_submission); + sched_hw_submission);
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +0
-0ring->buf_mask = (ring->ring_size /4 ) -1 ; ring->ptr_mask = ring->funcs->support_64bit_ptrs ?_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
A message that is not in the archive (sent off-list or not yet received): 1503437658.2195822.1081756264.1E3163C5-2RFepEojUI2N1INw9kWLP6GC3tUn3ZHUQQ4Iyu8u01E@public.gmane.org
Am 22.08.2017 um 23:34 schrieb Jay Cornwall:
Quoted 9 lines · Felix Kuehling
Quoted 4 lines
Thanks Alex!
Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.
I'm not sure what the best number is. Up to 8 KFD processes is common (beyond that performance drops off due to VMID availability) but I'm not sure how often they would need to submit to KIQ concurrently. If it's not expensive I'd just bump it up to say 16.
Well we allocate an array of pointers as ring buffer for the fences.
So I would say lets set this to 256, cause 256*number_of_entries_per_hw submision*number_of_bytes_for_a_pointer=4096.
This way we use up exactly one page for the fence array.
Regards,
Christian.
Quoted 66 lines
The performance problem isn't that bad since all the KIQ requests are serialized but the dmesg spam is not nice. Perhaps lowering the severity of the 'rcu slot is busy' message would address that as well?
Quoted 55 lines
Regards,
FelixQuoted 50 lines · Alex Deucher
KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However,
amdgpu_sched_hw_submission(which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.Signed-off-by Alex Deucher <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12
-2@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, unsigned irq_type) {int r; +int sched_hw_submission = amdgpu_sched_hw_submission; + +/* Set the hw submission limit higher for KIQ because +* it's used for a number of gfx/compute tasks by both +* KFD and KGD which may have outstanding fences and +* it doesn't really use the gpu scheduler anyway; +* KIQ tasks get submitted directly to the ring. +*/ +if (ring->funcs->type ==AMDGPU_RING_TYPE_KIQ ) + sched_hw_submission *=2 ;if (ring->adev ==NULL ) {if (adev->num_rings >=AMDGPU_MAX_RINGS )@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, ring->idx = adev->num_rings++; adev->rings[ring->idx] = ring; r =amdgpu_fence_driver_init_ring (ring, - amdgpu_sched_hw_submission); + sched_hw_submission);if (r)return r; }@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, } ring->ring_size =roundup_pow_of_two (max_dw *4 * - amdgpu_sched_hw_submission); + sched_hw_submission);Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235
Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +0
-0ring->buf_mask = (ring->ring_size /4 ) -1 ; ring->ptr_mask = ring->funcs->support_64bit_ptrs ?_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Quoted 4 lines
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx