drm/amdgpu/userq: fix indefinite fence wait during GPU reset
Subsystems (2)
PATCH 1/1 drm/amdgpu/userq: fix indefinite fence wait during GPU reset
pre_reset only force-completes fences of MAPPED queues. A queue in any other state (e.g. mid-eviction) keeps its last_fence pending; after a GPU reset that fence never signals, so the eviction/suspend worker and process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and wedge the machine:
INFO: task kworker/6:28 blocked for more than 120 seconds. Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu] Call Trace: dma_fence_wait_timeout+0x7e/0x130 amdgpu_userq_evict+0x67/0x140 [amdgpu] amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu] process_scheduled_works+0xa6/0x420
Force-complete every queue's fence regardless of state. The unmap and mark-hung step stays gated on MAPPED, since unmapping a queue that is not mapped is invalid.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +++++++++++++---------- 1 file changed, 14 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +14 -11
@@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev)/* TODO: We probably need a new lock for the queue state */ xa_for_each (&adev->userq_doorbell_xa, queue_id, queue) { -if (queue->state !=AMDGPU_USERQ_STATE_MAPPED ) -continue ; - -trace_amdgpu_userq_state_start (queue); - userq_funcs = adev->userq_funcs[queue->queue_type]; - userq_funcs->unmap (queue); -/* just mark all queues as hung at this point. -* if unmap succeeds, we could map again -* in amdgpu_userq_post_reset() if vram is not lost +if (queue->state ==AMDGPU_USERQ_STATE_MAPPED ) { +trace_amdgpu_userq_state_start (queue); + userq_funcs = adev->userq_funcs[queue->queue_type]; + userq_funcs->unmap (queue); +/* just mark all queues as hung at this point. +* if unmap succeeds, we could map again +* in amdgpu_userq_post_reset() if vram is not lost +*/ +trace_amdgpu_userq_state_changed (queue,AMDGPU_USERQ_STATE_HUNG ); + queue->state =AMDGPU_USERQ_STATE_HUNG ; + } +/* Force-complete any pending fence regardless of queue state so +* that eviction/suspend and queue teardown waiters don't block +* forever on a fence that will never signal after the reset. */ -trace_amdgpu_userq_state_changed (queue,AMDGPU_USERQ_STATE_HUNG ); - queue->state =AMDGPU_USERQ_STATE_HUNG ;amdgpu_userq_fence_driver_force_completion (queue); } }
Signature · 1 line
2.49.0
Quoted 14 lines · Jesse Zhang
pre_resetonly force-completes fences of MAPPED queues. A queue in any other state (e.g. mid-eviction) keeps itslast_fencepending; after a GPU reset that fence never signals, so the eviction/suspend worker and process teardown (amdgpu_evf_mgr_flush_suspend) wait on it forever and wedge the machine:INFO: task kworker/6:28 blocked for more than 120 seconds. Workqueue: events amdgpu_eviction_fence_suspend_worker [amdgpu] Call Trace: dma_fence_wait_timeout+0x7e/0x130 amdgpu_userq_evict+0x67/0x140 [amdgpu] amdgpu_eviction_fence_suspend_worker+0xd8/0x160 [amdgpu] process_scheduled_works+0xa6/0x420
Quoted 6 lines
Force-complete every queue's fence regardless of state. The unmap and mark-hung step stays gated on MAPPED, since unmapping a queue that is not mapped is invalid.
Fixes 290f46cf5726 ("drm/amdgpu: Implement user queue reset functionality")Signed-off-by Jesse Zhang <[email protected]>
Quoted 41 lines
--- drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c | 25 +++++++++++++---------- 1 file changed, 14 insertions(+), 11 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_userq.c +14
-11@@ -1424,18 +1424,21 @@ void amdgpu_userq_pre_reset(struct amdgpu_device *adev)/* TODO: We probably need a new lock for the queue state */ xa_for_each (&adev->userq_doorbell_xa, queue_id, queue) { -if (queue->state !=AMDGPU_USERQ_STATE_MAPPED ) -continue ; - -trace_amdgpu_userq_state_start (queue); - userq_funcs = adev->userq_funcs[queue->queue_type]; - userq_funcs->unmap (queue); -/* just mark all queues as hung at this point. -* if unmap succeeds, we could map again -* in amdgpu_userq_post_reset() if vram is not lost +if (queue->state ==AMDGPU_USERQ_STATE_MAPPED ) { +trace_amdgpu_userq_state_start (queue); + userq_funcs = adev->userq_funcs[queue->queue_type]; + userq_funcs->unmap (queue); +/* just mark all queues as hung at this point. +* if unmap succeeds, we could map again +* in amdgpu_userq_post_reset() if vram is not lost +*/ +trace_amdgpu_userq_state_changed (queue,AMDGPU_USERQ_STATE_HUNG ); + queue->state =AMDGPU_USERQ_STATE_HUNG ; + } +/* Force-complete any pending fence regardless of queue state so +* that eviction/suspend and queue teardown waiters don't block +* forever on a fence that will never signal after the reset. */ -trace_amdgpu_userq_state_changed (queue,AMDGPU_USERQ_STATE_HUNG ); - queue->state =AMDGPU_USERQ_STATE_HUNG ;amdgpu_userq_fence_driver_force_completion (queue); } }