Skip to contentSkip to conversation

drm/amdgpu: Fix KFD oversubscription by tracking queues correctly

PATCH Superseded Jay Cornwall 5 participants 7 messages Jul 12, 2017 – Jul 13, 2017 5 messages not in the archive

Subsystems (3)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu: Fix KFD oversubscription by tracking queues correctly

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
index 7060daf..aa4006a 100644
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
 		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
 		 * nbits is not compile time constant
 		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */
 				* adev->gfx.mec.num_pipe_per_mec
 				* adev->gfx.mec.num_queue_per_pipe;
 		for (i = last_valid_bit; i < KGD_MAX_QUEUES; ++i)
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1499884012-7669-1-git-send-email-Jay.Cornwall-5C7GfCeVMHo@public.gmane.org

#2 lore raw
Quoted 19 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 143

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
  		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
  		 * nbits is not compile time constant
  		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

Otherwise, looks good to me.

Regards,
Andres

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 146

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
  				* adev->gfx.mec.num_pipe_per_mec
  				* adev->gfx.mec.num_queue_per_pipe;
  		for (i = last_valid_bit; i < KGD_MAX_QUEUES; ++i)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 0d68e81c-28dc-a221-e37a-f9f10fd4ee49-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#4 lore raw
Quoted 29 lines · Andres Rodriguez
Quoted 19 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct · line 144

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct 
 amdgpu_device *adev)
          /* According to linux/bitmap.h we shouldn't use bitmap_clear if
           * nbits is not compile time constant
           */
-        last_valid_bit = adev->gfx.mec.num_mec
+        last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Quoted 3 lines

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

Otherwise, looks good to me.

Just re-read my reply and wanted to clarify. I don't really have a strong opining on which side does the resource availability patched. Whether it happens here or on the KFD side it is fine.

I just don't think it is good to keep it in different two places.

Regards,
Andres

Quoted 8 lines

Regards,
Andres

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
                  * adev->gfx.mec.num_pipe_per_mec
                  * adev->gfx.mec.num_queue_per_pipe;

for (i = last_valid_bit; i < KGD_MAX_QUEUES; ++i)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw
In reply to a message not in the archive
Quoted 28 lines · Andres Rodriguez
Quoted 19 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 143

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
  		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
  		 * nbits is not compile time constant
  		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Quoted 3 lines

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

OK. I see shared_resources.num_mec is set to 1 in kgd2kfd_device_init. That's not very clear (the number of MECs doesn't change) and num_mec doesn't appear to be used anywhere except in dead code in kfd_device.c. That code also runs after the queue bitmap setup.

How about I remove that field entirely?
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1499973356.2889480.1040107984.4839CCCF-2RFepEojUI2N1INw9kWLP6GC3tUn3ZHUQQ4Iyu8u01E@public.gmane.org

#7 lore raw
Quoted 31 lines · Jay Cornwall
Quoted 27 lines · Andres Rodriguez
Quoted 19 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 143

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
  		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
  		 * nbits is not compile time constant
  		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Quoted 3 lines

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

Quoted 6 lines

OK. I see shared_resources.num_mec is set to 1 in kgd2kfd_device_init. That's not very clear (the number of MECs doesn't change) and num_mec doesn't appear to be used anywhere except in dead code in kfd_device.c. That code also runs after the queue bitmap setup.

How about I remove that field entirely?

Yeah, that's fine with me.

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7ada7ccb-ed8b-0624-81fe-69cd529ea9c1-5C7GfCeVMHo@public.gmane.org

#9 lore raw
Quoted 32 lines · Felix Kuehling
Quoted 31 lines · Jay Cornwall
Quoted 30 lines · Andres Rodriguez
Quoted 19 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 143

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
   		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
   		 * nbits is not compile time constant
   		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

Quoted 8 lines
Quoted 6 lines

OK. I see shared_resources.num_mec is set to 1 in kgd2kfd_device_init. That's not very clear (the number of MECs doesn't change) and num_mec doesn't appear to be used anywhere except in dead code in kfd_device.c. That code also runs after the queue bitmap setup.

How about I remove that field entirely?

Yeah, that's fine with me.

Good with me as well.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 214a6fc1-d99e-bfe9-6f86-418cefd6283c-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#11 lore raw

There is a function get_mec_num use the field , but seems no one call it , maybe remove it as well.

Regards
Shaoyun.liu

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Andres Rodriguez
Sent: Thursday, July 13, 2017 3:54 PM
To: Kuehling, Felix; Jay Cornwall; [email protected]
Subject: Re: [PATCH] drm/amdgpu: Fix KFD oversubscription by tracking queues correctly

Quoted 33 lines · Felix Kuehling
Quoted 32 lines · Jay Cornwall
Quoted 31 lines · Andres Rodriguez
Quoted 20 lines · Jay Cornwall

The number of compute queues available to the KFD was erroneously calculated as 64. Only the first MEC can execute compute queues and it has 32 queue slots.

This caused the oversubscription limit to be calculated incorrectly, leading to a missing chained runlist command at the end of an oversubscribed runlist.

Change-Id Ic4a139c04b8a6d025fbb831a0a67e98728bfe461
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +0 -0
index 7060daf..aa4006a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) · line 143

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +1 -1
@@ -140,7 +140,7 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev)
   		/* According to linux/bitmap.h we shouldn't use bitmap_clear if
   		 * nbits is not compile time constant
   		 */
-		last_valid_bit = adev->gfx.mec.num_mec
+		last_valid_bit = 1 /* only first MEC can have compute queues */

Hey Jay,

Minor nitpick. We already have some similar resource patching in kgd2kfd_device_init(), and I think it would be good to keep all of these together.

Quoted 8 lines
Quoted 6 lines

OK. I see shared_resources.num_mec is set to 1 in kgd2kfd_device_init. That's not very clear (the number of MECs doesn't change) and num_mec doesn't appear to be used anywhere except in dead code in kfd_device.c. That code also runs after the queue bitmap setup.

How about I remove that field entirely?

Yeah, that's fine with me.

Good with me as well.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx