Skip to contentSkip to conversation

drm/amdgpu: set sched_hw_submission higher for KIQ

PATCH In review Alex Deucher 5 participants 5 messages Aug 22, 2017 – Aug 23, 2017 3 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: set sched_hw_submission higher for KIQ

KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12 -2
index 6c5646b..f39b851 100644
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		     unsigned irq_type)
 {
 	int r;
+	int sched_hw_submission = amdgpu_sched_hw_submission;
+
+	/* Set the hw submission limit higher for KIQ because
+	 * it's used for a number of gfx/compute tasks by both
+	 * KFD and KGD which may have outstanding fences and
+	 * it doesn't really use the gpu scheduler anyway;
+	 * KIQ tasks get submitted directly to the ring.
+	 */
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		sched_hw_submission *= 2;
 
 	if (ring->adev == NULL) {
 		if (adev->num_rings >= AMDGPU_MAX_RINGS)
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		ring->idx = adev->num_rings++;
 		adev->rings[ring->idx] = ring;
 		r = amdgpu_fence_driver_init_ring(ring,
-			amdgpu_sched_hw_submission);
+						  sched_hw_submission);
 		if (r)
 			return r;
 	}
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 	}
 
 	ring->ring_size = roundup_pow_of_two(max_dw * 4 *
-					     amdgpu_sched_hw_submission);
+					     sched_hw_submission);
 
 	ring->buf_mask = (ring->ring_size / 4) - 1;
 	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?
Signature · 6 lines
2.5.5

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1503434972-27760-1-git-send-email-alexander.deucher-5C7GfCeVMHo@public.gmane.org

#2 lore raw

Hi Alex,

See below comment about coding style of 80 characters per line.

Thanks,

Alex Bin Xie

________________________________
From: amd-gfx <amd-gfx-bounces-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org> on behalf of Alex Deucher <[email protected]>
Sent: Tuesday, August 22, 2017 4:49 PM
To: [email protected]
Cc: Deucher, Alexander
Subject: [PATCH] drm/amdgpu: set sched_hw_submission higher for KIQ

KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +11 -1
index 6c5646b..f39b851 100644
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
                      unsigned irq_type)
 {
         int r;
+       int sched_hw_submission = amdgpu_sched_hw_submission;
+
+       /* Set the hw submission limit higher for KIQ because
+        * it's used for a number of gfx/compute tasks by both
+        * KFD and KGD which may have outstanding fences and
+        * it doesn't really use the gpu scheduler anyway;
+        * KIQ tasks get submitted directly to the ring.
+        */
+       if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+               sched_hw_submission *= 2;
 
         if (ring->adev == NULL) {
                 if (adev->num_rings >= AMDGPU_MAX_RINGS)
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
                 ring->idx = adev->num_rings++;
                 adev->rings[ring->idx] = ring;
                 r = amdgpu_fence_driver_init_ring(ring,
-                       amdgpu_sched_hw_submission);
+                                                 sched_hw_submission);
 

Alex Bin Xie: With shorter variable name, there is no need to wrap into a new line of code.

                 if (r)
                         return r;
         }
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
         }

         ring->ring_size = roundup_pow_of_two(max_dw * 4 *

- amdgpu_sched_hw_submission);
+ sched_hw_submission);

Alex Bin Xie: With shorter variable name, there is no need to wrap into a new line of code.

         ring->buf_mask = (ring->ring_size / 4) - 1;
         ring->ptr_mask = ring->funcs->support_64bit_ptrs ?

--
2.5.5

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
amd-gfx Info Page - freedesktop.org<https://lists.freedesktop.org/mailman/listinfo/amd-gfx>
lists.freedesktop.org
Subscribing to amd-gfx: Subscribe to amd-gfx by filling out the following form. Use of all freedesktop.org lists is subject to our Code of ...

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw
In reply to a message not in the archive

Thanks Alex!

Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.

Regards,
Felix

Quoted 44 lines · Alex Deucher

KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +11 -1
index 6c5646b..f39b851 100644
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		     unsigned irq_type)
 {
 	int r;
+	int sched_hw_submission = amdgpu_sched_hw_submission;
+
+	/* Set the hw submission limit higher for KIQ because
+	 * it's used for a number of gfx/compute tasks by both
+	 * KFD and KGD which may have outstanding fences and
+	 * it doesn't really use the gpu scheduler anyway;
+	 * KIQ tasks get submitted directly to the ring.
+	 */
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		sched_hw_submission *= 2;
 
 	if (ring->adev == NULL) {
 		if (adev->num_rings >= AMDGPU_MAX_RINGS)
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		ring->idx = adev->num_rings++;
 		adev->rings[ring->idx] = ring;
 		r = amdgpu_fence_driver_init_ring(ring,
-			amdgpu_sched_hw_submission);
+						  sched_hw_submission);
 		if (r)
 			return r;
 	}

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +1 -1
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 	}
 
 	ring->ring_size = roundup_pow_of_two(max_dw * 4 *
-					     amdgpu_sched_hw_submission);
+					     sched_hw_submission);
 
 	ring->buf_mask = (ring->ring_size / 4) - 1;
 	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 943f41cb-e69d-6355-77b6-a3c992820dcb-5C7GfCeVMHo@public.gmane.org

#5 lore raw
Quoted 4 lines · Felix Kuehling

Thanks Alex!

Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.

I'm not sure what the best number is. Up to 8 KFD processes is common (beyond that performance drops off due to VMID availability) but I'm not sure how often they would need to submit to KIQ concurrently. If it's not expensive I'd just bump it up to say 16.

The performance problem isn't that bad since all the KIQ requests are serialized but the dmesg spam is not nice. Perhaps lowering the severity of the 'rcu slot is busy' message would address that as well?

Quoted 56 lines

Regards,
Felix

Quoted 50 lines · Alex Deucher

KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12 -2
index 6c5646b..f39b851 100644
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		     unsigned irq_type)
 {
 	int r;
+	int sched_hw_submission = amdgpu_sched_hw_submission;
+
+	/* Set the hw submission limit higher for KIQ because
+	 * it's used for a number of gfx/compute tasks by both
+	 * KFD and KGD which may have outstanding fences and
+	 * it doesn't really use the gpu scheduler anyway;
+	 * KIQ tasks get submitted directly to the ring.
+	 */
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		sched_hw_submission *= 2;
 
 	if (ring->adev == NULL) {
 		if (adev->num_rings >= AMDGPU_MAX_RINGS)
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 		ring->idx = adev->num_rings++;
 		adev->rings[ring->idx] = ring;
 		r = amdgpu_fence_driver_init_ring(ring,
-			amdgpu_sched_hw_submission);
+						  sched_hw_submission);
 		if (r)
 			return r;
 	}
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
 	}
 
 	ring->ring_size = roundup_pow_of_two(max_dw * 4 *
-					     amdgpu_sched_hw_submission);
+					     sched_hw_submission);
 
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +0 -0
 	ring->buf_mask = (ring->ring_size / 4) - 1;
 	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1503437658.2195822.1081756264.1E3163C5-2RFepEojUI2N1INw9kWLP6GC3tUn3ZHUQQ4Iyu8u01E@public.gmane.org

#7 lore raw

Am 22.08.2017 um 23:34 schrieb Jay Cornwall:

Quoted 9 lines · Felix Kuehling
Quoted 4 lines

Thanks Alex!

Jay, do you think this is enough? This bumps the number of concurrent operations on KIQ to 4 by default.

I'm not sure what the best number is. Up to 8 KFD processes is common (beyond that performance drops off due to VMID availability) but I'm not sure how often they would need to submit to KIQ concurrently. If it's not expensive I'd just bump it up to say 16.

Well we allocate an array of pointers as ring buffer for the fences.

So I would say lets set this to 256, cause 256*number_of_entries_per_hw submision*number_of_bytes_for_a_pointer=4096.

This way we use up exactly one page for the fence array.

Regards,
Christian.

Quoted 66 lines

The performance problem isn't that bad since all the KIQ requests are serialized but the dmesg spam is not nice. Perhaps lowering the severity of the 'rcu slot is busy' message would address that as well?

Quoted 55 lines

Regards,
Felix

Quoted 50 lines · Alex Deucher

KIQ doesn't really use the GPU scheduler. The base drivers generally use the KIQ ring directly rather than submitting IBs. However, amdgpu_sched_hw_submission (which defaults to 2) limits the number of outstanding fences to 2. KFD uses the KIQ for TLB flushes and the 2 fence limit hurts performance when there are several KFD processes running.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +12 -2
index 6c5646b..f39b851 100644
@@ -170,6 +170,16 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
  		     unsigned irq_type)
  {
  	int r;
+	int sched_hw_submission = amdgpu_sched_hw_submission;
+
+	/* Set the hw submission limit higher for KIQ because
+	 * it's used for a number of gfx/compute tasks by both
+	 * KFD and KGD which may have outstanding fences and
+	 * it doesn't really use the gpu scheduler anyway;
+	 * KIQ tasks get submitted directly to the ring.
+	 */
+	if (ring->funcs->type == AMDGPU_RING_TYPE_KIQ)
+		sched_hw_submission *= 2;
 
  	if (ring->adev == NULL) {
  		if (adev->num_rings >= AMDGPU_MAX_RINGS)
@@ -179,7 +189,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
  		ring->idx = adev->num_rings++;
  		adev->rings[ring->idx] = ring;
  		r = amdgpu_fence_driver_init_ring(ring,
-			amdgpu_sched_hw_submission);
+						  sched_hw_submission);
  		if (r)
  			return r;
  	}
@@ -219,7 +229,7 @@ int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring,
  	}
 
  	ring->ring_size = roundup_pow_of_two(max_dw * 4 *
-					     amdgpu_sched_hw_submission);
+					     sched_hw_submission);
 
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c int amdgpu_ring_init(struct amdgpu_device *adev, struct amdgpu_ring *ring, · line 235

Quoted 2 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +0 -0
  	ring->buf_mask = (ring->ring_size / 4) - 1;
  	ring->ptr_mask = ring->funcs->support_64bit_ptrs ?

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx