Skip to contentSkip to conversation

drm/amdgpu/userq: serialize queue map against GPU reset

PATCH In review Jesse Zhang 5 participants 5 messages Aug 3 – Aug 5

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/amdgpu/userq: serialize queue map against GPU reset

Creating a user queue can race with a GPU reset. While recovery holds reset_domain->sem for write, MES is unresponsive, so the ADD_QUEUE from amdgpu_userq_map_helper() times out (-110) and an otherwise valid queue create fails:

amdgpu: MES(0) failed to respond to msg=ADD_QUEUE [drm:mes_userq_map [amdgpu]] *ERROR* Failed to map queue in HW, err (-110)
amdgpu: [drm] *ERROR* ... Failed to map Queue
amdgpu: [drm] *ERROR* ... Failed to create usermode queue

Take reset_domain->sem for read around the map so it runs only once MES is back up. This mirrors amdgpu_userq_cleanup() and honors the userq_mutex -> reset_domain->sem order; the reset path never takes userq_mutex, so there is no deadlock.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +5 -0
index 652599f08990..770635ab5298 100644
@@ -745,7 +745,12 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
 	if (!adev->userq_halt_for_enforce_isolation ||
 	    ((queue->queue_type != AMDGPU_HW_IP_GFX) &&
 	     (queue->queue_type != AMDGPU_HW_IP_COMPUTE))) {
+		/* Serialize the map against an in-progress GPU reset (MES is
+		 * unresponsive during recovery), matching amdgpu_userq_cleanup().
+		 */
+		down_read(&adev->reset_domain->sem);
 		r = amdgpu_userq_map_helper(queue);
+		up_read(&adev->reset_domain->sem);
 		if (r) {
 			drm_file_err(uq_mgr->file, "Failed to map Queue\n");
 			trace_amdgpu_userq_create_end(queue, r);
Signature · 1 line
2.49.0
#1 lore raw

AMD General

We may need to ensure some other MES HW accessing without the GPU reset is kicked in concurrently. I'm working on the userq eviction case and figure out how to handle such issue in a general way.

Regards,
Prike

Quoted 50 lines

-----Original Message-----
From: amd-gfx <[email protected]> On Behalf Of Jesse Zhang
Sent: Monday, August 3, 2026 5:36 PM
To: [email protected]
Cc: Deucher, Alexander <[email protected]>; Koenig, Christian <[email protected]>; Zhang, Jesse(Jie) <[email protected]>
Subject: [PATCH] drm/amdgpu/userq: serialize queue map against GPU reset

Creating a user queue can race with a GPU reset. While recovery holds reset_domain->sem for write, MES is unresponsive, so the ADD_QUEUE from amdgpu_userq_map_helper() times out (-110) and an otherwise valid queue create fails:

amdgpu: MES(0) failed to respond to msg=ADD_QUEUE [drm:mes_userq_map [amdgpu]] *ERROR* Failed to map queue in HW, err (-110)
amdgpu: [drm] *ERROR* ... Failed to map Queue
amdgpu: [drm] *ERROR* ... Failed to create usermode queue

Take reset_domain->sem for read around the map so it runs only once MES is back up. This mirrors amdgpu_userq_cleanup() and honors the userq_mutex -> reset_domain->sem order; the reset path never takes userq_mutex, so there is no deadlock.

---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +5 -0
index 652599f08990..770635ab5298 100644
@@ -745,7 +745,12 @@ amdgpu_userq_create(struct drm_file *filp, union
 drm_amdgpu_userq *args)
      if (!adev->userq_halt_for_enforce_isolation ||
          ((queue->queue_type != AMDGPU_HW_IP_GFX) &&
           (queue->queue_type != AMDGPU_HW_IP_COMPUTE))) {
+             /* Serialize the map against an in-progress GPU reset (MES is
+              * unresponsive during recovery), matching amdgpu_userq_cleanup().
+              */
+             down_read(&adev->reset_domain->sem);
              r = amdgpu_userq_map_helper(queue);
+             up_read(&adev->reset_domain->sem);
              if (r) {
                      drm_file_err(uq_mgr->file, "Failed to map Queue\n");
                      trace_amdgpu_userq_create_end(queue, r);

--
2.49.0

Reviewed-by on v1 1/1
#2 lore raw
In reply to Jesse Zhang
Quoted 11 lines · Jesse Zhang

Creating a user queue can race with a GPU reset. While recovery holds reset_domain->sem for write, MES is unresponsive, so the ADD_QUEUE from amdgpu_userq_map_helper() times out (-110) and an otherwise valid queue create fails:

amdgpu: MES(0) failed to respond to msg=ADD_QUEUE [drm:mes_userq_map [amdgpu]] *ERROR* Failed to map queue in HW, err (-110)
amdgpu: [drm] *ERROR* ... Failed to map Queue
amdgpu: [drm] *ERROR* ... Failed to create usermode queue

Quoted 4 lines

Take reset_domain->sem for read around the map so it runs only once MES is back up. This mirrors amdgpu_userq_cleanup() and honors the userq_mutex -> reset_domain->sem order; the reset path never takes userq_mutex, so there is no deadlock.

Quoted 26 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +5 -1
index 652599f08990..770635ab5298 100644
@@ -745,7 +745,12 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
        if (!adev->userq_halt_for_enforce_isolation ||
            ((queue->queue_type != AMDGPU_HW_IP_GFX) &&
             (queue->queue_type != AMDGPU_HW_IP_COMPUTE))) {
+               /* Serialize the map against an in-progress GPU reset (MES is
+                * unresponsive during recovery), matching amdgpu_userq_cleanup().
+                */
+               down_read(&adev->reset_domain->sem);
                r = amdgpu_userq_map_helper(queue);
+               up_read(&adev->reset_domain->sem);
                if (r) {
                        drm_file_err(uq_mgr->file, "Failed to map Queue\n");
                        trace_amdgpu_userq_create_end(queue, r);
--

2.49.0

maintainer Acked-by on v1 1/1
#3 lore raw
In reply to Jesse Zhang
Quoted 10 lines · Jesse Zhang

Creating a user queue can race with a GPU reset. While recovery holds reset_domain->sem for write, MES is unresponsive, so the ADD_QUEUE from amdgpu_userq_map_helper() times out (-110) and an otherwise valid queue create fails:

amdgpu: MES(0) failed to respond to msg=ADD_QUEUE [drm:mes_userq_map [amdgpu]] *ERROR* Failed to map queue in HW, err (-110)
amdgpu: [drm] *ERROR* ... Failed to map Queue
amdgpu: [drm] *ERROR* ... Failed to create usermode queue

Quoted 6 lines

Take reset_domain->sem for read around the map so it runs only once MES is back up. This mirrors amdgpu_userq_cleanup() and honors the userq_mutex -> reset_domain->sem order; the reset path never takes userq_mutex, so there is no deadlock.

Please also loop in Sunil on such patches.

Thanks,
Christian.

Quoted 21 lines
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +5 -0
index 652599f08990..770635ab5298 100644
@@ -745,7 +745,12 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
 	if (!adev->userq_halt_for_enforce_isolation ||
 	    ((queue->queue_type != AMDGPU_HW_IP_GFX) &&
 	     (queue->queue_type != AMDGPU_HW_IP_COMPUTE))) {
+		/* Serialize the map against an in-progress GPU reset (MES is
+		 * unresponsive during recovery), matching amdgpu_userq_cleanup().
+		 */
+		down_read(&adev->reset_domain->sem);
 		r = amdgpu_userq_map_helper(queue);
+		up_read(&adev->reset_domain->sem);
 		if (r) {
 			drm_file_err(uq_mgr->file, "Failed to map Queue\n");
 			trace_amdgpu_userq_create_end(queue, r);
#4 lore raw
In reply to Jesse Zhang
Quoted 25 lines · Jesse Zhang

Creating a user queue can race with a GPU reset. While recovery holds reset_domain->sem for write, MES is unresponsive, so the ADD_QUEUE from amdgpu_userq_map_helper() times out (-110) and an otherwise valid queue create fails:

amdgpu: MES(0) failed to respond to msg=ADD_QUEUE [drm:mes_userq_map [amdgpu]] *ERROR* Failed to map queue in HW, err (-110)
amdgpu: [drm] *ERROR* ... Failed to map Queue
amdgpu: [drm] *ERROR* ... Failed to create usermode queue

Take reset_domain->sem for read around the map so it runs only once MES is back up. This mirrors amdgpu_userq_cleanup() and honors the userq_mutex -> reset_domain->sem order; the reset path never takes userq_mutex, so there is no deadlock.

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 5 +++++
  1 file changed, 5 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +0 -0
index 652599f08990..770635ab5298 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args) · line 751

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +4 -0
@@ -745,7 +745,12 @@ amdgpu_userq_create(struct drm_file *filp, union drm_amdgpu_userq *args)
  	if (!adev->userq_halt_for_enforce_isolation ||
  	    ((queue->queue_type != AMDGPU_HW_IP_GFX) &&
  	     (queue->queue_type != AMDGPU_HW_IP_COMPUTE))) {
+		/* Serialize the map against an in-progress GPU reset (MES is
+		 * unresponsive during recovery), matching amdgpu_userq_cleanup().
+		 */
+		down_read(&adev->reset_domain->sem);

If a reset is ongoing and a userq operation comes in the middle, there is no need to wait till reset semaphore is available. The ongoing process i going to terminate sooner or later after the reset.

This should only try to acquire the lock and return an error if not able to.

Thanks,
Lijo

Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +1 -0
  		r = amdgpu_userq_map_helper(queue);
+		up_read(&adev->reset_domain->sem);
  		if (r) {
  			drm_file_err(uq_mgr->file, "Failed to map Queue\n");
  			trace_amdgpu_userq_create_end(queue, r);