Skip to contentSkip to conversation

amdgpu: amdgpu_userq_restore_all return errors on failed bo reservation

PATCH In review Zhu Lingshan 3 participants 7 messages Jun 18 – Jun 19

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 amdgpu: amdgpu_userq_restore_all return errors on failed bo reservation

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +6 -4
index 95b680fc88c5..8b14870afbf5 100644
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
 	unsigned long queue_id;
 	int ret = 0, r;
 
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;
 
 	mutex_lock(&uq_mgr->userq_mutex);
 	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
 		goto put_fence;
 	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
 put_fence:
 	dma_fence_put(ev_fence);
Signature · 1 line
2.54.0
maintainer
#1 lore raw
Quoted 17 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
 	unsigned long queue_id;
 	int ret = 0, r;
 
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
 	mutex_lock(&uq_mgr->userq_mutex);
 	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
 		goto put_fence;
 	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
 put_fence:
 	dma_fence_put(ev_fence);
#2 lore raw
Quoted 21 lines · Christian König
Quoted 20 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
  	unsigned long queue_id;
  	int ret = 0, r;
 
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

I think i probably missed that. There is no return value check for amdgpu_userq_restore_all and this cant fail. We should wait uninterruptible here i.e change false->true and drop the if condition.

Quoted 4 lines

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

we need to maintain order of locking First reserve root bo and then take mutex else we had mutex deadlocks.... this is why we have added root bo locking first before taking mutex in next line.

Regards

Sunil Khatri

Quoted 18 lines

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
  	mutex_lock(&uq_mgr->userq_mutex);
  	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
  		goto put_fence;
  	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
  put_fence:
  	dma_fence_put(ev_fence);
maintainer
#3 lore raw
Quoted 23 lines · Sunil
Quoted 21 lines · Christian König
Quoted 20 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
 	unsigned long queue_id;
 	int ret = 0, r;
 
Quoted 8 lines
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

I think i probably missed that. There is no return value check for amdgpu_userq_restore_all and this cant fail. We should wait uninterruptible here i.e change false->true and drop the if condition.

No, dropping and re-acquiring the lock is a broken approach to begin with.

Quoted 5 lines
Quoted 3 lines

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

we need to maintain order of locking First reserve root bo and then take mutex else we had mutex deadlocks.... this is why we have added root bo locking first before taking mutex in next line.

As long as we do that in amdgpu_userq_vm_validate() that should be unproblematic. The VM root BO and all other BOs are still locked and acquiring the userq_mutex shouldn't be much of a problem.

Regards,
Christian.

Quoted 22 lines

Regards

Sunil Khatri

Quoted 17 lines

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
 	mutex_lock(&uq_mgr->userq_mutex);
 	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
 		goto put_fence;
 	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
 put_fence:
 	dma_fence_put(ev_fence);
#4 lore raw
Quoted 31 lines · Christian König
Quoted 29 lines · Sunil
Quoted 28 lines · Christian König
Quoted 17 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
  	unsigned long queue_id;
  	int ret = 0, r;
 
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

Quoted 8 lines
Quoted 1 line

I think i probably missed that. There is no return value check for amdgpu_userq_restore_all and this cant fail. We should wait uninterruptible here i.e change false->true and drop the if condition.

No, dropping and re-acquiring the lock is a broken approach to begin with.

Quoted 4 lines
Quoted 3 lines

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

we need to maintain order of locking First reserve root bo and then take mutex else we had mutex deadlocks.... this is why we have added root bo locking first before taking mutex in next line.

As long as we do that in amdgpu_userq_vm_validate() that should be unproblematic. The VM root BO and all other BOs are still locked and acquiring the userq_mutex shouldn't be much of a problem.

Are you suggesting to move amdgpu_userq_restore_all inside the amdgpu_userq_vm_validate? Current order is we amdgpu_userq_vm_validate and then amdgpu_userq_restore_all. So if i get it correct, while we have the exec locks just before exec_fini, after amdgpu_evf_mgr_rearm, take mutex and call amdgpu_userq_restore_all?

Regards
Sunil Khatri

Quoted 25 lines

Regards,
Christian.

Quoted 21 lines

Regards

Sunil Khatri

Quoted 17 lines

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
  	mutex_lock(&uq_mgr->userq_mutex);
  	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
  		goto put_fence;
  	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
  put_fence:
  	dma_fence_put(ev_fence);
maintainer
#5 lore raw
Quoted 35 lines · Sunil
Quoted 33 lines · Christian König
Quoted 22 lines · Sunil
Quoted 21 lines · Christian König
Quoted 20 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
 	unsigned long queue_id;
 	int ret = 0, r;
 
Quoted 8 lines
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

I think i probably missed that. There is no return value check for amdgpu_userq_restore_all and this cant fail. We should wait uninterruptible here i.e change false->true and drop the if condition.

No, dropping and re-acquiring the lock is a broken approach to begin with.

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

we need to maintain order of locking First reserve root bo and then take mutex else we had mutex deadlocks.... this is why we have added root bo locking first before taking mutex in next line.

As long as we do that in amdgpu_userq_vm_validate() that should be unproblematic. The VM root BO and all other BOs are still locked and acquiring the userq_mutex shouldn't be much of a problem.

Are you suggesting to move amdgpu_userq_restore_all inside the amdgpu_userq_vm_validate? Current order is we amdgpu_userq_vm_validate and then amdgpu_userq_restore_all. So if i get it correct, while we have the exec locks just before exec_fini, after amdgpu_evf_mgr_rearm, take mutex and call amdgpu_userq_restore_all?

Yes, exactly that.

Christian.

Quoted 28 lines

Regards
Sunil Khatri

Quoted 24 lines

Regards,
Christian.

Quoted 21 lines

Regards

Sunil Khatri

Quoted 17 lines

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
 	mutex_lock(&uq_mgr->userq_mutex);
 	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
 		goto put_fence;
 	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
 put_fence:
 	dma_fence_put(ev_fence);
#6 lore raw
Quoted 35 lines · Christian König
Quoted 34 lines · Sunil
Quoted 33 lines · Christian König
Quoted 22 lines · Sunil
Quoted 21 lines · Christian König
Quoted 20 lines · Zhu Lingshan

In amdgpu_userq_restore_all(), when failed to reserve a bo, it should return a meaningful error code other than "false" that means SUCCESS, which is wrong.

The caller should not ignore the return code of
amdgpu_userq_restore_all as well

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 10 ++++++----
  1 file changed, 6 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 95b680fc88c5..8b14870afbf5 100644
@@ -894,9 +894,9 @@ amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr)
  	unsigned long queue_id;
  	int ret = 0, r;
 
Quoted 8 lines
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_restore_all(struct amdgpu_userq_mgr *uq_mgr) · line 899

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -3
-
-	if (amdgpu_bo_reserve(vm->root.bo, false))
-		return false;
+	r = amdgpu_bo_reserve(vm->root.bo, false);
+	if (r)
+		return r;

Good catch, but that amdgpu_bo_reserve() is called here is a bug in the first place.

I think i probably missed that. There is no return value check for amdgpu_userq_restore_all and this cant fail. We should wait uninterruptible here i.e change false->true and drop the if condition.

No, dropping and re-acquiring the lock is a broken approach to begin with.

Quoted 7 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines

The call to amdgpu_userq_vm_validate() must be moved into amdgpu_userq_vm_validate(), right before we call drm_exec_fini() and the manual call to amdgpu_bo_reserve() here dropped.

Otherwise we have a small windows where we drop the BO locks before starting the queues which could make the VM invalid again and cause all kind of issues.

we need to maintain order of locking First reserve root bo and then take mutex else we had mutex deadlocks.... this is why we have added root bo locking first before taking mutex in next line.

As long as we do that in amdgpu_userq_vm_validate() that should be unproblematic. The VM root BO and all other BOs are still locked and acquiring the userq_mutex shouldn't be much of a problem.

Are you suggesting to move amdgpu_userq_restore_all inside the amdgpu_userq_vm_validate? Current order is we amdgpu_userq_vm_validate and then amdgpu_userq_restore_all. So if i get it correct, while we have the exec locks just before exec_fini, after amdgpu_evf_mgr_rearm, take mutex and call amdgpu_userq_restore_all?

Yes, exactly that.

Let me work on that and push another patch set on this.

Regards

Sunil Khatri

Quoted 30 lines

Christian.

Quoted 27 lines

Regards
Sunil Khatri

Quoted 24 lines

Regards,
Christian.

Quoted 21 lines

Regards

Sunil Khatri

Quoted 17 lines

Regards,
Christian.

Quoted 14 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +3 -1
  	mutex_lock(&uq_mgr->userq_mutex);
  	/* Resume all the queues for this process */
@@ -1133,7 +1133,9 @@ static void amdgpu_userq_restore_worker(struct work_struct *work)
  		goto put_fence;
  	}
 
-	amdgpu_userq_restore_all(uq_mgr);
+	ret = amdgpu_userq_restore_all(uq_mgr);
+	if (ret)
+		drm_file_err(uq_mgr->file, "Failed to restore user queues, ret=%d\n", ret);
 
  put_fence:
  	dma_fence_put(ev_fence);