Skip to contentSkip to conversation

drm/amdgpu: fix ctx init failure for asics without gfx ring

PATCH In review Le Ma 3 participants 7 messages Dec 19, 2019 – Jan 4, 2020

Subsystems (4)
lore
Share & tools
lore

Le Ma

author
#0 lore raw

PATCH 1/1 drm/amdgpu: fix ctx init failure for asics without gfx ring

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +2 -1
index 63f6365..64e2bab 100644
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev,
 
 		switch (i) {
 		case AMDGPU_HW_IP_GFX:
-			scheds = adev->gfx.gfx_sched;
+			sched = &adev->gfx.gfx_ring[0].sched;
+			scheds = &sched;
 			num_scheds = 1;
 			break;
 		case AMDGPU_HW_IP_COMPUTE:
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Reviewed-by Nirmoy Das on v1 1/1
#1 lore raw
Quoted 14 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c static int amdgpu_ctx_init(struct amdgpu_device *adev, · line 134

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +2 -1
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev,
 
  		switch (i) {
  		case AMDGPU_HW_IP_GFX:
-			scheds = adev->gfx.gfx_sched;
+			sched = &adev->gfx.gfx_ring[0].sched;
+			scheds = &sched;
  			num_scheds = 1;
  			break;
  		case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#2 lore raw

Am 19.12.19 um 13:01 schrieb Nirmoy:

Quoted 12 lines
Quoted 9 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 12 lines
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device 
 *adev,
             switch (i) {
Quoted 5 lines

          case AMDGPU_HW_IP_GFX:
-            scheds = adev->gfx.gfx_sched; +            sched = &adev->gfx.gfx_ring[0].sched; +            scheds = &sched;
              num_scheds = 1;

Mhm, we should probably rather fix this here and don't expose a GFX ring when the hardware doesn't have one.

Christian.

Quoted 6 lines
Quoted 2 lines

              break;
          case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#3 lore raw
Quoted 13 lines · Christian König

Am 19.12.19 um 13:01 schrieb Nirmoy:

Quoted 12 lines
Quoted 9 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device 
 *adev,
             switch (i) {

          case AMDGPU_HW_IP_GFX:
-            scheds = adev->gfx.gfx_sched; +            sched = &adev->gfx.gfx_ring[0].sched; +            scheds = &sched;
              num_scheds = 1;

Mhm, we should probably rather fix this here and don't expose a GFX ring when the hardware doesn't have one.

Hi Christian,

Do you mean by not initializing entity for gfx when not available?

Quoted 11 lines

Christian.

Quoted 7 lines
Quoted 2 lines

              break;
          case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cnirmoy.das%40amd.com%7Cbc5fa498efe84fcfb00b08d78eb97049%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637134799421291347&sdata=FxkwaCrbIV28HH18xorgwmzRPCMA1I9IkRkgE1LF80Y%3D&reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#4 lore raw

Am 02.01.20 um 10:47 schrieb Nirmoy:

Quoted 29 lines · Christian König
Quoted 13 lines

Am 19.12.19 um 13:01 schrieb Nirmoy:

Quoted 12 lines
Quoted 9 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 14 lines
Quoted 4 lines
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644
Quoted 8 lines
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device 
 *adev,
             switch (i) {

          case AMDGPU_HW_IP_GFX:
-            scheds = adev->gfx.gfx_sched; +            sched = &adev->gfx.gfx_ring[0].sched; +            scheds = &sched;
              num_scheds = 1;

Mhm, we should probably rather fix this here and don't expose a GFX ring when the hardware doesn't have one.

Quoted 3 lines

Hi Christian,

Do you mean by not initializing entity for gfx when not available?

Well we still initialize it, but with num_scheds=0.

Christian.

Quoted 13 lines
Quoted 13 lines

Christian.

Quoted 7 lines
Quoted 2 lines

              break;
          case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&data=02%7C01%7Cnirmoy.das%40amd.com%7Cbc5fa498efe84fcfb00b08d78eb97049%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637134799421291347&sdata=FxkwaCrbIV28HH18xorgwmzRPCMA1I9IkRkgE1LF80Y%3D&reserved=0

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#5 lore raw
Quoted 30 lines · Christian König

Am 02.01.20 um 10:47 schrieb Nirmoy:

Quoted 29 lines · Christian König
Quoted 27 lines

Am 19.12.19 um 13:01 schrieb Nirmoy:

Quoted 12 lines
Quoted 9 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 12 lines
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct 
 amdgpu_device *adev,
             switch (i) {
Quoted 5 lines

          case AMDGPU_HW_IP_GFX:
-            scheds = adev->gfx.gfx_sched; +            sched = &adev->gfx.gfx_ring[0].sched; +            scheds = &sched;
              num_scheds = 1;

Mhm, we should probably rather fix this here and don't expose a GFX ring when the hardware doesn't have one.

Quoted 5 lines
Quoted 3 lines

Hi Christian,

Do you mean by not initializing entity for gfx when not available?

Well we still initialize it, but with num_scheds=0.

Hi Christian,

Currently drm_sched_entity_init requires a non-NULL sched_list/(old) rq_list. This is forcing us to pass an uninitialized

drm sched with 0 num_scheds to drm_sched_entity_init() for non-existing hw ip. I haven't think about such(missing/failed hw ip) corner case before but

I think we can handle it by:

drivers/gpu/drm/scheduler/sched_entity.c +0 -0
index 2e3a058fc239..563592299817 100644
@@ -56,7 +56,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity,

                          unsigned int num_sched_list,
                          atomic_t *guilty)
 {
-       if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0]))) +       if (!(entity && (num_sched_list == 0 || sched_list[0]))) return -EINVAL;

        memset(entity, 0, sizeof(struct drm_sched_entity));

drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 64e2babbc36e..4c8ad6cb11ee 100644
@@ -127,9 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device *adev,
 

                switch (i) {
                case AMDGPU_HW_IP_GFX:
-                       sched = &adev->gfx.gfx_ring[0].sched;
-                       scheds = &sched;
-                       num_scheds = 1;
+                       scheds = adev->gfx.gfx_sched;
+                       num_scheds = adev->gfx.num_compute_sched; break;
                case AMDGPU_HW_IP_COMPUTE:
                        scheds = adev->gfx.compute_sched;
@@ -622,6 +621,10 @@ void amdgpu_ctx_init_sched(struct amdgpu_device *adev) adev->gfx.num_gfx_sched++;
        }

+       /* Currently there is only one user usable gfx queue */
+       if (adev->gfx.num_gfx_sched > 1)
+               adev->gfx.num_gfx_sched = 1;
+
        for (i = 0; i < adev->gfx.num_compute_rings; i++) { adev->gfx.compute_sched[i] = &adev->gfx.compute_ring[i].sched; adev->gfx.num_compute_sched++;

What do you think?

I have two more questions to clear regarding this

1 Why do we need to create entity for non existing hw IP?

2 What happens when user try push a gfx related job to a GPU without any gfx queue ?

Regards,

Nirmoy

Quoted 21 lines

Christian.

Quoted 18 lines
Quoted 13 lines

Christian.

Quoted 7 lines
Quoted 2 lines

              break;
          case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cnirmoy.das%40amd.com%7C7603e9d4410045c1b63d08d78faf6e41%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637135855948050926&amp;sdata=JOtJwqsXMuXeyq8igxuQ8f4JHCS6MZi6PAJdGWI202g%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cnirmoy.das%40amd.com%7C7603e9d4410045c1b63d08d78faf6e41%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637135855948050926&amp;sdata=JOtJwqsXMuXeyq8igxuQ8f4JHCS6MZi6PAJdGWI202g%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#6 lore raw

Am 03.01.20 um 11:43 schrieb Nirmoy:

Quoted 51 lines · Christian König
Quoted 30 lines

Am 02.01.20 um 10:47 schrieb Nirmoy:

Quoted 29 lines · Christian König
Quoted 27 lines

Am 19.12.19 um 13:01 schrieb Nirmoy:

Quoted 12 lines
Quoted 9 lines · On 12/19/19 12:42 PM, Le Ma wrote:

This workaround does not affect other asics because amdgpu only need expose one gfx sched to user for now.

Change-Id Ica92b8565a89899aebe0eba7b2b5a25159b411d3

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 12 lines
Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 63f6365..64e2bab 100644
@@ -127,7 +127,8 @@ static int amdgpu_ctx_init(struct 
 amdgpu_device *adev,
             switch (i) {
Quoted 5 lines

          case AMDGPU_HW_IP_GFX:
-            scheds = adev->gfx.gfx_sched; +            sched = &adev->gfx.gfx_ring[0].sched; +            scheds = &sched;
              num_scheds = 1;

Mhm, we should probably rather fix this here and don't expose a GFX ring when the hardware doesn't have one.

Quoted 5 lines
Quoted 3 lines

Hi Christian,

Do you mean by not initializing entity for gfx when not available?

Well we still initialize it, but with num_scheds=0.

Hi Christian,

Currently drm_sched_entity_init requires a non-NULL sched_list/(old) rq_list. This is forcing us to pass an uninitialized

drm sched with 0 num_scheds to drm_sched_entity_init() for non-existing hw ip. I haven't think about such(missing/failed hw ip) corner case before but

I think we can handle it by:

drivers/gpu/drm/scheduler/sched_entity.c +0 -0
index 2e3a058fc239..563592299817 100644

drivers/gpu/drm/scheduler/sched_entity.c int drm_sched_entity_init(struct drm_sched_entity · line 61

Quoted 7 lines
drivers/gpu/drm/scheduler/sched_entity.c +1 -1
@@ -56,7 +56,7 @@ int drm_sched_entity_init(struct drm_sched_entity 
 *entity,
                           unsigned int num_sched_list,
                           atomic_t *guilty)
  {
-       if (!(entity && sched_list && (num_sched_list == 0 || 
 sched_list[0])))
+       if (!(entity && (num_sched_list == 0 || sched_list[0])))

I think that change is unnecessary. Even when num_sched_list is 0 we can still pass in the pointer.

Quoted 11 lines
drivers/gpu/drm/scheduler/sched_entity.c +0 -0
 return -EINVAL;

        memset(entity, 0, sizeof(struct drm_sched_entity));

drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +0 -0
index 64e2babbc36e..4c8ad6cb11ee 100644
@@ -127,9 +127,8 @@ static int amdgpu_ctx_init(struct amdgpu_device 
 *adev,
 
                 switch (i) {
Quoted 6 lines

                case AMDGPU_HW_IP_GFX:
-                       sched = &adev->gfx.gfx_ring[0].sched; -                       scheds = &sched;
-                       num_scheds = 1;
+                       scheds = adev->gfx.gfx_sched;
+                       num_scheds = adev->gfx.num_compute_sched;

We need a num_gfx_sched here, because the number of compute scheduler instances is completely unrelated to the number of gfx scheduler instances.

Quoted 17 lines

                        break;
                case AMDGPU_HW_IP_COMPUTE:
                        scheds = adev->gfx.compute_sched;

drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +4 -0
@@ -622,6 +621,10 @@ void amdgpu_ctx_init_sched(struct amdgpu_device 
 *adev)
                 adev->gfx.num_gfx_sched++;
         }
 
+       /* Currently there is only one user usable gfx queue */
+       if (adev->gfx.num_gfx_sched > 1)
+               adev->gfx.num_gfx_sched = 1;
+
         for (i = 0; i < adev->gfx.num_compute_rings; i++) {
                 adev->gfx.compute_sched[i] = &adev->gfx.compute_ring[i].sched;

                adev->gfx.num_compute_sched++;

Quoted 5 lines

What do you think?

I have two more questions to clear regarding this

1 Why do we need to create entity for non existing hw IP?

To cleanly reject command submissions from userspace when they try to access that hw.

E.g. the SW structure is still initialized even if the HW isn't present, we just set the number of schedulers to zero to indicated that this hardware is not present. Otherwise we would work with a uninitialized SW structure.

Alternatively when you are done with your changes we can reject creating the SW structure in the first place, but for this we need those changes first.

Quoted 2 lines

2 What happens when user try push a gfx related job to a GPU without any gfx queue ?

IIRC we just return EINVAL if there is not hw scheduler available to handle a software queue.

Regards,
Christian.

Quoted 27 lines

Regards,

Nirmoy

Quoted 21 lines

Christian.

Quoted 18 lines
Quoted 13 lines

Christian.

Quoted 7 lines
Quoted 2 lines

              break;
          case AMDGPU_HW_IP_COMPUTE:

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cnirmoy.das%40amd.com%7C7603e9d4410045c1b63d08d78faf6e41%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637135855948050926&amp;sdata=JOtJwqsXMuXeyq8igxuQ8f4JHCS6MZi6PAJdGWI202g%3D&amp;reserved=0

_______________________________________________
amd-gfx mailing list
[email protected]
https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=02%7C01%7Cnirmoy.das%40amd.com%7C7603e9d4410045c1b63d08d78faf6e41%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637135855948050926&amp;sdata=JOtJwqsXMuXeyq8igxuQ8f4JHCS6MZi6PAJdGWI202g%3D&amp;reserved=0

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx