Skip to contentSkip to conversation

tests/amdgpu/amd_queue_reset: Fix random hang during queue reset stress

PATCH In review Jesse Zhang 3 participants 3 messages Jul 9

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/amdgpu/amd_queue_reset: Fix random hang during queue reset stress

A concurrent "good" background job can be collateral-cancelled by the queue reset and return an error the good-job path did not tolerate, tripping igt_assert_eq() and aborting the background process. Its death breaks the fixed NUM_CHILD_PROCESSES barrier, so the survivors block forever in sem_wait().

Tolerate every reset-induced error on good jobs: unify the two asymmetric allow-lists in amdgpu_test_exec_cs_helper() and add -EHWPOISON in run_background(), so the process never aborts and the barrier cannot deadlock.

---
 lib/amdgpu/amd_command_submission.c | 14 ++++++++++----
 tests/amdgpu/amd_queue_reset.c      | 10 +++++++++-
 2 files changed, 19 insertions(+), 5 deletions(-)
lib/amdgpu/amd_command_submission.c +10 -4
index a0777e5ae..0334c2ded 100644
@@ -119,8 +119,11 @@ int amdgpu_test_exec_cs_helper(amdgpu_device_handle device, unsigned int ip_type
 		if (expect_failure)
 			igt_info("amdgpu_cs_submit %d PID %d\n", r, getpid());
 		else {
-			/* we allow ECANCELED, ENODATA or -EHWPOISON for good jobs temporally */
-			if (r != -ECANCELED && r != -ENODATA && r != -EHWPOISON)
+			/* a good job may be collateral-cancelled by a concurrent
+			 * queue reset; tolerate every reset-induced error
+			 */
+			if (r != -ECANCELED && r != -ENODATA && r != -EHWPOISON &&
+			    r != -ETIME)
 				igt_assert_eq(r, 0);
 		}
 
@@ -142,8 +145,11 @@ int amdgpu_test_exec_cs_helper(amdgpu_device_handle device, unsigned int ip_type
 			igt_info("EXPECT FAILURE amdgpu_cs_query_fence_status %d\n"
 				 "expired %d PID %d\n", r, expired, getpid());
 		} else {
-			/* we allow ECANCELED or ENODATA for good jobs temporally */
-			if (r != -ECANCELED && r != -ENODATA && r != -ETIME)
+			/* a good job may be collateral-cancelled by a concurrent
+			 * queue reset; tolerate every reset-induced error
+			 */
+			if (r != -ECANCELED && r != -ENODATA && r != -ETIME &&
+			    r != -EHWPOISON)
 				igt_assert_eq(r, 0);
 		}
 	}
tests/amdgpu/amd_queue_reset.c +9 -1
index 546845d29..ebd7c8682 100644
@@ -840,7 +840,15 @@ run_background(amdgpu_device_handle device, struct shmbuf *sh_mem,
 				//if entire gpu reset then stop back ground jobs
 				break;
 			}
-			if (r != -ECANCELED && r != -ETIME && r != -ENODATA)
+			/*
+			 * A "good" background job can be collateral-cancelled by
+			 * the concurrent queue reset triggered by the bad job.
+			 * Tolerate every reset-induced error so the helper never
+			 * aborts this process - its death would break the
+			 * NUM_CHILD_PROCESSES barrier and hang the whole test.
+			 */
+			if (r != -ECANCELED && r != -ETIME && r != -ENODATA &&
+			    r != -EHWPOISON)
 				igt_assert_eq(r, 0);
 			//igt_assert_eq(err_codes.err_code_wait_for_fence, job.reset_err_result);
 			counter++;
Signature · 1 line
2.49.0
#1 lore raw

LGTM

Reviewed-by Vitaly Prosyak <[email protected]>

Quoted 69 lines · Jesse Zhang

A concurrent "good" background job can be collateral-cancelled by the queue reset and return an error the good-job path did not tolerate, tripping igt_assert_eq() and aborting the background process. Its death breaks the fixed NUM_CHILD_PROCESSES barrier, so the survivors block forever in sem_wait().

Tolerate every reset-induced error on good jobs: unify the two asymmetric allow-lists in amdgpu_test_exec_cs_helper() and add -EHWPOISON in run_background(), so the process never aborts and the barrier cannot deadlock.

---
 lib/amdgpu/amd_command_submission.c | 14 ++++++++++----
 tests/amdgpu/amd_queue_reset.c      | 10 +++++++++-
 2 files changed, 19 insertions(+), 5 deletions(-)
lib/amdgpu/amd_command_submission.c +10 -4
index a0777e5ae..0334c2ded 100644
@@ -119,8 +119,11 @@ int amdgpu_test_exec_cs_helper(amdgpu_device_handle device, unsigned int ip_type
 		if (expect_failure)
 			igt_info("amdgpu_cs_submit %d PID %d\n", r, getpid());
 		else {
-			/* we allow ECANCELED, ENODATA or -EHWPOISON for good jobs temporally */
-			if (r != -ECANCELED && r != -ENODATA && r != -EHWPOISON)
+			/* a good job may be collateral-cancelled by a concurrent
+			 * queue reset; tolerate every reset-induced error
+			 */
+			if (r != -ECANCELED && r != -ENODATA && r != -EHWPOISON &&
+			    r != -ETIME)
 				igt_assert_eq(r, 0);
 		}
 
@@ -142,8 +145,11 @@ int amdgpu_test_exec_cs_helper(amdgpu_device_handle device, unsigned int ip_type
 			igt_info("EXPECT FAILURE amdgpu_cs_query_fence_status %d\n"
 				 "expired %d PID %d\n", r, expired, getpid());
 		} else {
-			/* we allow ECANCELED or ENODATA for good jobs temporally */
-			if (r != -ECANCELED && r != -ENODATA && r != -ETIME)
+			/* a good job may be collateral-cancelled by a concurrent
+			 * queue reset; tolerate every reset-induced error
+			 */
+			if (r != -ECANCELED && r != -ENODATA && r != -ETIME &&
+			    r != -EHWPOISON)
 				igt_assert_eq(r, 0);
 		}
 	}
tests/amdgpu/amd_queue_reset.c +9 -1
index 546845d29..ebd7c8682 100644
@@ -840,7 +840,15 @@ run_background(amdgpu_device_handle device, struct shmbuf *sh_mem,
 				//if entire gpu reset then stop back ground jobs
 				break;
 			}
-			if (r != -ECANCELED && r != -ETIME && r != -ENODATA)
+			/*
+			 * A "good" background job can be collateral-cancelled by
+			 * the concurrent queue reset triggered by the bad job.
+			 * Tolerate every reset-induced error so the helper never
+			 * aborts this process - its death would break the
+			 * NUM_CHILD_PROCESSES barrier and hang the whole test.
+			 */
+			if (r != -ECANCELED && r != -ETIME && r != -ENODATA &&
+			    r != -EHWPOISON)
 				igt_assert_eq(r, 0);
 			//igt_assert_eq(err_codes.err_code_wait_for_fence, job.reset_err_result);
 			counter++;