Skip to contentSkip to conversation

drm/amd/sched: rename amd_sched_entity_pop_job

PATCH In review Nicolai Hähnle 11 participants 36 messages Sep 28, 2017 – Oct 13, 2017 26 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/5 drm/amd/sched: rename amd_sched_entity_pop_job

From: Nicolai Hähnle <[email protected]>

The function does not actually remove the job from the FIFO, so "peek" describes it better.

---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -2
index 97c94f9683fa..742d724cd720 100644
@@ -301,21 +301,21 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
 
 	if (!dma_fence_add_callback(entity->dependency, &entity->cb,
 				    amd_sched_entity_wakeup))
 		return true;
 
 	dma_fence_put(entity->dependency);
 	return false;
 }
 
 static struct amd_sched_job *
-amd_sched_entity_pop_job(struct amd_sched_entity *entity)
+amd_sched_entity_peek_job(struct amd_sched_entity *entity)
 {
 	struct amd_gpu_scheduler *sched = entity->sched;
 	struct amd_sched_job *sched_job;
 
 	if (!kfifo_out_peek(&entity->job_queue, &sched_job, sizeof(sched_job)))
 		return NULL;
 
 	while ((entity->dependency = sched->ops->dependency(sched_job)))
 		if (amd_sched_entity_add_dependency_cb(entity))
 			return NULL;
@@ -593,21 +593,21 @@ static int amd_sched_main(void *param)
 		struct dma_fence *fence;
 
 		wait_event_interruptible(sched->wake_up_worker,
 					 (!amd_sched_blocked(sched) &&
 					  (entity = amd_sched_select_entity(sched))) ||
 					 kthread_should_stop());
 
 		if (!entity)
 			continue;
 
-		sched_job = amd_sched_entity_pop_job(entity);
+		sched_job = amd_sched_entity_peek_job(entity);
 		if (!sched_job)
 			continue;
 
 		s_fence = sched_job->s_fence;
 
 		atomic_inc(&sched->hw_rq_count);
 		amd_sched_job_begin(sched_job);
 
 		fence = sched->ops->run_job(sched_job);
 		amd_sched_fence_scheduled(s_fence);
Signature · 6 lines
2.11.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 20170928145530.12844-1-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#2 lore raw

PATCH 2/5 drm/amd/sched: fix an outdated comment

From: Nicolai Hähnle <[email protected]>

---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -2
index 742d724cd720..6e899c593b7e 100644
@@ -347,22 +347,21 @@ static bool amd_sched_entity_in(struct amd_sched_job *sched_job)
 
 	/* first job wakes up scheduler */
 	if (first) {
 		/* Add the entity to the run queue */
 		amd_sched_rq_add_entity(entity->rq, entity);
 		amd_sched_wakeup(sched);
 	}
 	return added;
 }
 
-/* job_finish is called after hw fence signaled, and
- * the job had already been deleted from ring_mirror_list
+/* job_finish is called after hw fence signaled
  */
 static void amd_sched_job_finish(struct work_struct *work)
 {
 	struct amd_sched_job *s_job = container_of(work, struct amd_sched_job,
 						   finish_work);
 	struct amd_gpu_scheduler *sched = s_job->sched;
 
 	/* remove job from ring_mirror_list */
 	spin_lock(&sched->job_list_lock);
 	list_del_init(&s_job->node);
Signature · 6 lines
2.11.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#3 lore raw

PATCH 3/5 drm/amd/sched: move adding finish callback to amd_sched_job_begin

In reply to a message not in the archive

From: Nicolai Hähnle <[email protected]>

The finish callback is responsible for removing the job from the ring mirror list, among other things. It makes sense to add it as callback in the place where the job is added to the ring mirror list.

---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -2
index 6e899c593b7e..e793312e351c 100644
@@ -388,20 +388,23 @@ static void amd_sched_job_finish_cb(struct dma_fence *f,
 {
 	struct amd_sched_job *job = container_of(cb, struct amd_sched_job,
 						 finish_cb);
 	schedule_work(&job->finish_work);
 }
 
 static void amd_sched_job_begin(struct amd_sched_job *s_job)
 {
 	struct amd_gpu_scheduler *sched = s_job->sched;
 
+	dma_fence_add_callback(&s_job->s_fence->finished, &s_job->finish_cb,
+			       amd_sched_job_finish_cb);
+
 	spin_lock(&sched->job_list_lock);
 	list_add_tail(&s_job->node, &sched->ring_mirror_list);
 	if (sched->timeout != MAX_SCHEDULE_TIMEOUT &&
 	    list_first_entry_or_null(&sched->ring_mirror_list,
 				     struct amd_sched_job, node) == s_job)
 		schedule_delayed_work(&s_job->work_tdr, sched->timeout);
 	spin_unlock(&sched->job_list_lock);
 }
 
 static void amd_sched_job_timedout(struct work_struct *work)
@@ -480,22 +483,20 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  *
  * @sched_job		The pointer to job required to submit
  *
  * Returns 0 for success, negative error code otherwise.
  */
 void amd_sched_entity_push_job(struct amd_sched_job *sched_job)
 {
 	struct amd_sched_entity *entity = sched_job->s_entity;
 
 	trace_amd_sched_job(sched_job);
-	dma_fence_add_callback(&sched_job->s_fence->finished, &sched_job->finish_cb,
-			       amd_sched_job_finish_cb);
 	wait_event(entity->sched->job_scheduled,
 		   amd_sched_entity_in(sched_job));
 }
 
 /* init a sched_job with basic field */
 int amd_sched_job_init(struct amd_sched_job *job,
 		       struct amd_gpu_scheduler *sched,
 		       struct amd_sched_entity *entity,
 		       void *owner)
 {
Signature · 6 lines
2.11.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#4 lore raw

PATCH 4/5 drm/amd/sched: NULL out the s_fence field after run_job

In reply to a message not in the archive

From: Nicolai Hähnle <[email protected]>

amd_sched_process_job drops the fence reference, so NULL out the s_fence field before adding it as a callback to guard against accidentally using s_fence after it may have be freed.

---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
index e793312e351c..54eb77cffd9b 100644
@@ -604,20 +604,23 @@ static int amd_sched_main(void *param)
 		if (!sched_job)
 			continue;
 
 		s_fence = sched_job->s_fence;
 
 		atomic_inc(&sched->hw_rq_count);
 		amd_sched_job_begin(sched_job);
 
 		fence = sched->ops->run_job(sched_job);
 		amd_sched_fence_scheduled(s_fence);
+
+		sched_job->s_fence = NULL;
+
 		if (fence) {
 			s_fence->parent = dma_fence_get(fence);
 			r = dma_fence_add_callback(fence, &s_fence->cb,
 						   amd_sched_process_job);
 			if (r == -ENOENT)
 				amd_sched_process_job(fence, &s_fence->cb);
 			else if (r)
 				DRM_ERROR("fence add callback failed (%d)\n",
 					  r);
 			dma_fence_put(fence);
Signature · 6 lines
2.11.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 20170928145530.12844-4-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

Reviewed-by
#6 lore raw

Re: [PATCH 4/5] drm/amd/sched: NULL out the s_fence field after run_job

Quoted 19 lines · Nicolai Hähnle

From: Nicolai Hähnle <[email protected]>

amd_sched_process_job drops the fence reference, so NULL out the s_fence field before adding it as a callback to guard against accidentally using s_fence after it may have be freed.

---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index e793312e351c..54eb77cffd9b 100644
@@ -604,20 +604,23 @@ static int amd_sched_main(void *param)
  		if (!sched_job)
  			continue;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static int amd_sched_main(void *param) · line 615

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
 
  		s_fence = sched_job->s_fence;
 
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
  		fence = sched->ops->run_job(sched_job);
  		amd_sched_fence_scheduled(s_fence);
+
+		sched_job->s_fence = NULL;

Minor optional nitpick here. Could this be moved somewhere closer to where the fence reference is actually dropped? Alternatively, could a comment be added to specify which function call results in the reference ownership transfer?

Whether a change is made or not, this series is

Currently running piglit to check if this fixes the occasional soft hangs I was getting where all tests complete except one.

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static int amd_sched_main(void *param) · line 626

Quoted 12 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -0
+
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence);
  			r = dma_fence_add_callback(fence, &s_fence->cb,
  						   amd_sched_process_job);
  			if (r == -ENOENT)
  				amd_sched_process_job(fence, &s_fence->cb);
  			else if (r)
  				DRM_ERROR("fence add callback failed (%d)\n",
  					  r);
  			dma_fence_put(fence);

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7064b408-60db-2817-0ae7-af6b2c56580b-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#8 lore raw

Re: [PATCH 4/5] drm/amd/sched: NULL out the s_fence field after run_job

Quoted 15 lines · Andres Rodriguez
Quoted 12 lines · Nicolai Hähnle

From: Nicolai Hähnle <[email protected]>

amd_sched_process_job drops the fence reference, so NULL out the s_fence field before adding it as a callback to guard against accidentally using s_fence after it may have be freed.

---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 3 +++

  1 file changed, 3 insertions(+)
Quoted 16 lines
Quoted 5 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index e793312e351c..54eb77cffd9b 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static int amd_sched_main(void *param) · line 612

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
@@ -604,20 +604,23 @@ static int amd_sched_main(void *param)
           if (!sched_job)
               continue;
           s_fence = sched_job->s_fence;
           atomic_inc(&sched->hw_rq_count);
           amd_sched_job_begin(sched_job);
           fence = sched->ops->run_job(sched_job);
           amd_sched_fence_scheduled(s_fence);
+
+        sched_job->s_fence = NULL;

Minor optional nitpick here. Could this be moved somewhere closer to where the fence reference is actually dropped? Alternatively, could a comment be added to specify which function call results in the reference ownership transfer?

Sure, I can add a comment. (It's amd_sched_process_job, which is called directly or indirectly in all the branches of the following if-statement.)

Quoted 2 lines

Whether a change is made or not, this series is

Thanks.

Quoted 1 line

Currently running piglit to check if this fixes the occasional soft hangs I was getting where all tests complete except one.

You may be running into this Mesa issue:

https://patchwork.freedesktop.org/patch/179535/

Cheers,
Nicolai

Quoted 13 lines

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static int amd_sched_main(void *param)

Quoted 13 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -0
+
           if (fence) {
               s_fence->parent = dma_fence_get(fence);
               r = dma_fence_add_callback(fence, &s_fence->cb,
                              amd_sched_process_job);
               if (r == -ENOENT)
                   amd_sched_process_job(fence, &s_fence->cb);
               else if (r)
                   DRM_ERROR("fence add callback failed (%d)\n",
                         r);
               dma_fence_put(fence);
 

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#9 lore raw

PATCH 5/5 drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

    [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
    [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
    [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
    [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
    [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
    [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
    [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
    [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
    [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
    [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
    [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
    [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
    [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
    [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
    [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
    [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
    [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
    [<ffffffffffffffff>] 0xffffffffffffffff
---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -1
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
 					amd_sched_entity_is_idle(entity));
 	amd_sched_rq_remove_entity(rq, entity);
 	if (r) {
 		struct amd_sched_job *job;
 
 		/* Park the kernel for a moment to make sure it isn't processing
 		 * our enity.
 		 */
 		kthread_park(sched->thread);
 		kthread_unpark(sched->thread);
-		while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+		while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+			struct amd_sched_fence *s_fence = job->s_fence;
+			amd_sched_fence_scheduled(s_fence);
+			amd_sched_fence_finished(s_fence);
+			dma_fence_put(&s_fence->finished);
 			sched->ops->free_job(job);
+		}
 
 	}
 	kfifo_free(&entity->job_queue);
 }
 
 static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
 {
 	struct amd_sched_entity *entity =
 		container_of(cb, struct amd_sched_entity, cb);
 	entity->dependency = NULL;
Signature · 6 lines
2.11.0

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 20170928145530.12844-5-nhaehnle-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#11 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 45 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
     [<ffffffffffffffff>] 0xffffffffffffffff
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
  					amd_sched_entity_is_idle(entity));
  	amd_sched_rq_remove_entity(rq, entity);
  	if (r) {
  		struct amd_sched_job *job;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 232

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
 
  		/* Park the kernel for a moment to make sure it isn't processing
  		 * our enity.
  		 */
  		kthread_park(sched->thread);
  		kthread_unpark(sched->thread);
-		while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+		while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+			struct amd_sched_fence *s_fence = job->s_fence;
+			amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
+			amd_sched_fence_finished(s_fence);
+			dma_fence_put(&s_fence->finished);
  			sched->ops->free_job(job);
+		}

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 246

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
 
  	}
  	kfifo_free(&entity->job_queue);
  }
 
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
  {
  	struct amd_sched_entity *entity =
  		container_of(cb, struct amd_sched_entity, cb);
  	entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 3032bef3-4829-8cae-199a-11353b38c49a-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#13 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Hi Nicolai,

Ping? :-)

Tom

Quoted 80 lines · Christian König

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 45 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

     [<ffffffffffffffff>] 0xffffffffffffffff

---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct 
 amd_gpu_scheduler *sched,
                       amd_sched_entity_is_idle(entity));
       amd_sched_rq_remove_entity(rq, entity);
       if (r) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct · line 232

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
           struct amd_sched_job *job;
           /* Park the kernel for a moment to make sure it isn't processing
            * our enity.
            */
           kthread_park(sched->thread);
           kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
+            amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
               sched->ops->free_job(job);
+        }
       }
       kfifo_free(&entity->job_queue);
   }
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
   {
Quoted 3 lines

      struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#14 lore raw

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive

Christian

Quoted 1 line

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 46 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
     [<ffffffffffffffff>] 0xffffffffffffffff
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
  					amd_sched_entity_is_idle(entity));
  	amd_sched_rq_remove_entity(rq, entity);
  	if (r) {
  		struct amd_sched_job *job;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 232

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
 
  		/* Park the kernel for a moment to make sure it isn't processing
  		 * our enity.
  		 */
  		kthread_park(sched->thread);
  		kthread_unpark(sched->thread);
-		while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+		while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+			struct amd_sched_fence *s_fence = job->s_fence;
+			amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
+			amd_sched_fence_finished(s_fence);
+			dma_fence_put(&s_fence->finished);
  			sched->ops->free_job(job);
+		}

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 246

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
 
  	}
  	kfifo_free(&entity->job_queue);
  }
 
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
  {
  	struct amd_sched_entity *entity =
  		container_of(cb, struct amd_sched_entity, cb);
  	entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB044904A26E01C265C49042E484740-7LeqcoF/[email protected]

#16 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 1 line

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 95 lines

Christian

Quoted 1 line

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 46 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

      [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
      [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
      [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
      [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
      [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
      [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
      [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
      [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
      [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
      [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
      [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
      [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
      [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
      [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
      [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
      [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
      [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
      [<ffffffffffffffff>] 0xffffffffffffffff
---
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
   1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
   					amd_sched_entity_is_idle(entity));
   	amd_sched_rq_remove_entity(rq, entity);
   	if (r) {
   		struct amd_sched_job *job;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 232

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
 
   		/* Park the kernel for a moment to make sure it isn't processing
   		 * our enity.
   		 */
   		kthread_park(sched->thread);
   		kthread_unpark(sched->thread);
-		while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+		while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+			struct amd_sched_fence *s_fence = job->s_fence;
+			amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
+			amd_sched_fence_finished(s_fence);
+			dma_fence_put(&s_fence->finished);
   			sched->ops->free_job(job);
+		}
 
   	}
   	kfifo_free(&entity->job_queue);
   }
 
Quoted 14 lines

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 246

Quoted 5 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
   {
   	struct amd_sched_entity *entity =
   		container_of(cb, struct amd_sched_entity, cb);
   	entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 11f21e54-16b8-68e4-c63e-d791ef8bbffa-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#18 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 4 lines · Christian König
Quoted 1 line

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 114 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 107 lines

Christian

Quoted 1 line

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 46 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

      [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
      [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
      [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
      [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
      [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
      [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
      [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
      [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
      [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
      [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
      [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
      [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
      [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
      [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
      [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
      [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
      [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

      [<ffffffffffffffff>] 0xffffffffffffffff

---
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

   1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct 
 amd_gpu_scheduler *sched,
                        amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct · line 232

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't processing
             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -0
+            amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
    {
Quoted 3 lines

       struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): d0f66c04-fbcd-09a2-6e4c-9de9ca7a93ff-5C7GfCeVMHo@public.gmane.org

#20 lore raw

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 134 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 127 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 55 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

      [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
      [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
      [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
      [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
      [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
      [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
      [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
      [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
      [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
      [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
      [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 
[amdgpu]
      [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 
[amdgpu]
      [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
      [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
      [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
      [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
      [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

      [<ffffffffffffffff>] 0xffffffffffffffff

---
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

   1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct 
 amd_gpu_scheduler *sched,
                        amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't 

processing

Quoted 8 lines

            * our enity.
            */
           kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) { +            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+            amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
               sched->ops->free_job(job);
+        }
       }

Quoted 8 lines

       kfifo_free(&entity->job_queue);
   }
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
   {
       struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB044925932C8D956F93CAF93E84740-7LeqcoF/[email protected]

#22 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 47 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 7 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

Quoted 143 lines
Quoted 1 line

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 134 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 127 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 55 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
                        amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

Quoted 8 lines

            * our enity.
            */
           kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) { +            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+            amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
               sched->ops->free_job(job);
+        }
       }

Quoted 8 lines

       kfifo_free(&entity->job_queue);
   }
   static void amd_sched_entity_wakeup(struct dma_fence *f, struct
dma_fence_cb *cb)
   {
       struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7e338e23-540c-4e2e-982f-f0eb623c75b1-5C7GfCeVMHo@public.gmane.org

#24 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Hi Nicolai & Monk,

well that sounds like exactly what I wanted to hear.

When Mesa already handles ECANCELED gracefully as return code for a lost context then I think we can move forward with that. Monk, would that be ok with you?

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 201 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 192 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 134 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 127 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 59 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
                        amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 9 lines

+ amd_sched_fence_finished(s_fence);

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
+            dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct

dma_fence_cb *cb)

Quoted 4 lines
    {
        struct amd_sched_entity *entity =
            container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 760c1434-0739-81ff-82c3-a5210c5575d3-5C7GfCeVMHo@public.gmane.org

#26 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 1 line · Christian König

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 209 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 201 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 192 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED ,  we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 134 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 127 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 55 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

       [<ffffffffffffffff>] 0xffffffffffffffff

---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
                         amd_sched_entity_is_idle(entity));
         amd_sched_rq_remove_entity(rq, entity);
         if (r) {
             struct amd_sched_job *job;
             /* Park the kernel for a moment to make sure it isn't

processing

Quoted 8 lines

             * our enity.
             */
            kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) { +            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+ amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }

Quoted 8 lines

        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): de5e2c7c-b6cd-1c24-4d8e-7ae3cdfad0bd-5C7GfCeVMHo@public.gmane.org

#28 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 4 lines
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

How hard would it be to handle that correctly?

Quoted 1 line

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

I thought of that on top of the -ENODEV handling.

In other words when we see -ENODEV we call an IOCTL to let the kernel know we noticed that something is wrong and then reinit all shared resources in userspace.

All existing context will still see -ECANCELED when we drop their command submission, but new contexts would at least not cause a new lockup immediately because their shader binaries are corrupted.

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 222 lines · Christian König
Quoted 1 line

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 210 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 202 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 193 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 135 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 128 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 60 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -1
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 9 lines

+ amd_sched_fence_finished(s_fence);

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
+            dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct

dma_fence_cb *cb)

Quoted 4 lines
    {
        struct amd_sched_entity *entity =
            container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 9619ebd2-f218-7568-3b24-0a9d2b008a6a-5C7GfCeVMHo@public.gmane.org

#30 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

Quoted 7 lines
Quoted 1 line

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

I thought of that on top of the -ENODEV handling.

In other words when we see -ENODEV we call an IOCTL to let the kernel know we noticed that something is wrong and then reinit all shared resources in userspace.

All existing context will still see -ECANCELED when we drop their command submission, but new contexts would at least not cause a new lockup immediately because their shader binaries are corrupted.

I don't think we need -ENODEV for this. We just need -ECANCELED to be returned when a submission is rejected due to reset (hang or VRAM loss).

Mesa would keep a shadow of the VRAM loss counter in pipe_screen and pipe_context, and query the kernel's counter when it encounters -ECANCELED. Each context would then know to drop the CS it's built up so far and restart based on comparing the VRAM loss counter of pipe_screen to that of pipe_context, and similarly we could keep a copy of the VRAM loss counter for important buffer objects like shader binaries, descriptors, etc.

This seems more robust to me than relying only on an ENODEV. We'd most likely keep some kind of VRAM loss counter in Mesa *anyway* (we don't maintain a list of all shaders, for example, and we can't nuke important per-context across threads), and synthesizing such a counter from ENODEVs is not particularly robust (what if multiple ENODEVs occur for the same loss event?).

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Cheers,
Nicolai

Quoted 227 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 221 lines · Christian König
Quoted 1 line

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 209 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 201 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 192 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED ,  we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 134 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 127 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 55 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

       [<ffffffffffffffff>] 0xffffffffffffffff

---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
         amd_sched_rq_remove_entity(rq, entity);
         if (r) {
             struct amd_sched_job *job;
             /* Park the kernel for a moment to make sure it isn't

processing

Quoted 8 lines

             * our enity.
             */
            kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) { +            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+ amd_sched_fence_finished(s_fence);
+            dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }

Quoted 8 lines

        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): de68c0ca-f36e-3adb-2c42-83a5176f07d8-5C7GfCeVMHo@public.gmane.org

#32 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

Quoted 6 lines · Christian König
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Quoted 7 lines
Quoted 3 lines

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. E.g. you could have an application which uploads shaders, a GPU reset happens and VRAM is lost and then the application creates a new context and makes submission with broken shader binaries.

So I would still vote for a separate IOCTL to reset the VRAM lost state which is called *before" user space starts to reupload shader/descriptors etc...

This way you also catch the case when another reset happens while you reupload things.

Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

I thought of that on top of the -ENODEV handling.

In other words when we see -ENODEV we call an IOCTL to let the kernel know we noticed that something is wrong and then reinit all shared resources in userspace.

All existing context will still see -ECANCELED when we drop their command submission, but new contexts would at least not cause a new lockup immediately because their shader binaries are corrupted.

I don't think we need -ENODEV for this. We just need -ECANCELED to be returned when a submission is rejected due to reset (hang or VRAM loss).

Quoted 7 lines

Mesa would keep a shadow of the VRAM loss counter in pipe_screen and pipe_context, and query the kernel's counter when it encounters
-ECANCELED. Each context would then know to drop the CS it's built up
so far and restart based on comparing the VRAM loss counter of pipe_screen to that of pipe_context, and similarly we could keep a copy of the VRAM loss counter for important buffer objects like shader binaries, descriptors, etc.

This seems more robust to me than relying only on an ENODEV. We'd most likely keep some kind of VRAM loss counter in Mesa *anyway* (we don't maintain a list of all shaders, for example, and we can't nuke important per-context across threads), and synthesizing such a counter from ENODEVs is not particularly robust (what if multiple ENODEVs occur for the same loss event?).

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still somehow note the fatality of loosing VRAM to userspace.

How about ENODATA or EBADFD?

Regards,
Christian.

Quoted 237 lines

Cheers,
Nicolai

Quoted 231 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 225 lines · Christian König
Quoted 1 line

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 213 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 205 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 196 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 139 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 132 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 65 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 
[ttm]
       [<ffffffffc042f523>] 
amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -1
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, 

sizeof(job))) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

Quoted 2 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 14 lines

+ amd_sched_fence_finished(s_fence);

+ dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, 
struct
dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity =
            container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 2f113fd3-ab4a-58b8-31d8-dc0a23751513-5C7GfCeVMHo@public.gmane.org

#34 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 9 lines · Christian König

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

Quoted 8 lines · Christian König
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. E.g. you could have an application which uploads shaders, a GPU reset happens and VRAM is lost and then the application creates a new context and makes submission with broken shader binaries.

Hmm. Here's how I imagined we'd be using a VRAM lost counter:

int si_shader_binary_upload(...)
{
...
shader->bo_vram_lost_counter = sscreen->vram_lost_counter; shader->bo = pipe_buffer_create(...);
ptr = sscreen->b.ws->buffer_map(shader->bo->buf, ...); ... copies ... sscreen->b.ws->buffer_unmap(shader->bo->buf);
}

int si_shader_select(...)
{
...
r = si_shader_select_with_key(ctx->sscreen, state, ...); if (r) return r;

if (state->current->bo_vram_lost_counter !=

       ctx->sscreen->vram_lost_counter) {
      ... re-upload sequence ...

}
}

(Not shown: logic that compares ctx->vram_lost_counter with sscreen->vram_lost_counter and forces a re-validation of all state including shaders.)

That should cover this scenario, shouldn't it?

Oh... I see one problem. But it should be easy to fix: when creating a new amdgpu context, Mesa needs to query the vram lost counter. So then the sequence of events would be either:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Mesa creates a new amdgpu context, queries the VRAM lost counter --> 1
- si_screen::vram_lost_counter is updated to 1
- Draw happens on the new context --> si_shader_select will catch the VRAM loss

Or:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Mesa creates a new amdgpu context, VRAM lost counter still 0
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Draw happens on the new context and proceeds normally
...
- Mesa flushes the CS, and the kernel will return an error code because the device VRAM lost counter is different from the amdgpu context VRAM lost counter

Quoted 1 line

So I would still vote for a separate IOCTL to reset the VRAM lost state which is called *before" user space starts to reupload shader/descriptors etc...

The question is: is that separate IOCTL per-context or per-fd? If it's per-fd, then it's not compatible with multiple drivers. If it's per-context, then I don't see how it helps. Perhaps you could explain?

Quoted 2 lines

This way you also catch the case when another reset happens while you re-upload things.

My assumption would be that the re-upload happens *after* the new amdgpu context is created. Then the repeat reset should be caught by the kernel when we try to submit a CS on the new context (this is assuming that the kernel context's vram lost counter is initialized properly when the context is created):

- Mesa prepares upload, sets shader->bo_vram_lost_counter to 0
- Mesa uploads a shader binary
- While doing this, a GPU reset happens[0], kernel increments device VRAM lost counter to 1
- Draw happens with the new shader, Mesa proceeds normally
...
- On flush / CS submit, the kernel detects the VRAM lost state and returns an error to Mesa

[0] Out of curiosity: What happens on the CPU side if the PCI / full ASIC reset method is used? Is there a time window where we could get a SEGV?

[snip]

Quoted 5 lines
Quoted 1 line

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still somehow note the fatality of loosing VRAM to userspace.

How about ENODATA or EBADFD?

According to the manpage, EBADFD is "File descriptor in bad state.". Sounds fitting :)

Cheers,
Nicolai

Quoted 238 lines

Regards,
Christian.

Quoted 233 lines

Cheers,
Nicolai

Quoted 227 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 221 lines · Christian König
Quoted 1 line

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 209 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 201 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 192 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED ,  we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 135 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 128 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 56 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

       [<ffffffffffffffff>] 0xffffffffffffffff

---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
         amd_sched_rq_remove_entity(rq, entity);
         if (r) {
             struct amd_sched_job *job;
             /* Park the kernel for a moment to make sure it isn't

processing
             * our enity.

Quoted 8 lines

             */
            kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+ amd_sched_fence_finished(s_fence);
+ dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }

Quoted 8 lines

        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1a79e19c-a654-f5c7-84d9-ce4cce76243f-5C7GfCeVMHo@public.gmane.org

#36 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Mesa does not handle -ECANCELED. It only returns -ECANCELED from the Mesa winsys layer if the CS ioctl wasn't called (because the context is already lost and so the winsys doesn't submit further CS ioctls).

When the CS ioctl fails for the first time, the kernel error is returned and the context is marked as "lost".

The next command submission is automatically dropped by the winsys and it returns -ECANCELED.

Marek

________________________________
From: Haehnle, Nicolai
Sent: Monday, October 9, 2017 2:58:02 PM
To: Koenig, Christian; Liu, Monk; Nicolai Hähnle; [email protected]; Olsak, Marek
Cc: Li, Bingley
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 17 lines · Christian König

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

Quoted 16 lines · Christian König
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Quoted 8 lines
Quoted 2 lines

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

How hard would it be to handle that correctly?

Okay, that makes more sense :)

Quoted 8 lines
Quoted 3 lines

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. E.g. you could have an application which uploads shaders, a GPU reset happens and VRAM is lost and then the application creates a new context and makes submission with broken shader binaries.

Hmm. Here's how I imagined we'd be using a VRAM lost counter:

int si_shader_binary_upload(...)

{
    ...
    shader->bo_vram_lost_counter = sscreen->vram_lost_counter;
    shader->bo = pipe_buffer_create(...);
    ptr = sscreen->b.ws->buffer_map(shader->bo->buf, ...);
    ... copies ...
    sscreen->b.ws->buffer_unmap(shader->bo->buf);
}

int si_shader_select(...)

{
    ...
    r = si_shader_select_with_key(ctx->sscreen, state, ...);
    if (r) return r;

    if (state->current->bo_vram_lost_counter !=
        ctx->sscreen->vram_lost_counter) {
       ... re-upload sequence ...
    }
}

(Not shown: logic that compares ctx->vram_lost_counter with sscreen->vram_lost_counter and forces a re-validation of all state including shaders.)

That should cover this scenario, shouldn't it?

Oh... I see one problem. But it should be easy to fix: when creating a new amdgpu context, Mesa needs to query the vram lost counter. So then the sequence of events would be either:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Mesa creates a new amdgpu context, queries the VRAM lost counter --> 1
- si_screen::vram_lost_counter is updated to 1
- Draw happens on the new context --> si_shader_select will catch the VRAM loss

Or:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Mesa creates a new amdgpu context, VRAM lost counter still 0
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Draw happens on the new context and proceeds normally
...
- Mesa flushes the CS, and the kernel will return an error code because the device VRAM lost counter is different from the amdgpu context VRAM lost counter

Quoted 3 lines

So I would still vote for a separate IOCTL to reset the VRAM lost state which is called *before" user space starts to reupload shader/descriptors etc...

The question is: is that separate IOCTL per-context or per-fd? If it's per-fd, then it's not compatible with multiple drivers. If it's per-context, then I don't see how it helps. Perhaps you could explain?

Quoted 2 lines

This way you also catch the case when another reset happens while you re-upload things.

My assumption would be that the re-upload happens *after* the new amdgpu context is created. Then the repeat reset should be caught by the kernel when we try to submit a CS on the new context (this is assuming that the kernel context's vram lost counter is initialized properly when the context is created):

- Mesa prepares upload, sets shader->bo_vram_lost_counter to 0
- Mesa uploads a shader binary
- While doing this, a GPU reset happens[0], kernel increments device VRAM lost counter to 1
- Draw happens with the new shader, Mesa proceeds normally
...
- On flush / CS submit, the kernel detects the VRAM lost state and returns an error to Mesa

[0] Out of curiosity: What happens on the CPU side if the PCI / full ASIC reset method is used? Is there a time window where we could get a SEGV?

[snip]

Quoted 8 lines
Quoted 3 lines

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still somehow note the fatality of loosing VRAM to userspace.

How about ENODATA or EBADFD?

According to the manpage, EBADFD is "File descriptor in bad state.". Sounds fitting :)

Cheers,
Nicolai

Quoted 279 lines

Regards,
Christian.

Quoted 274 lines

Cheers,
Nicolai

Quoted 268 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 262 lines · Christian König
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 246 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 236 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 225 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty, also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver,
which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 7 lines · Christian König
Quoted 7 lines

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Quoted 5 lines

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences?
That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 150 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 143 lines

Christian

Quoted 4 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel
return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL)
founded as error, that way UMD would run into robust extension path
and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being
killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 68 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>]
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420
[ttm]
       [<ffffffffc042f523>]
amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -1
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job,

sizeof(job))) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

Quoted 2 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that
this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 14 lines

+ amd_sched_fence_finished(s_fence);

+ dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f,
struct
dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity =
            container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): CY1PR12MB0885AF7148CD8ECE929E96D2F9740-1s8aH8ViOEfCYw/[email protected]

#38 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

It depends on what you mean by "handle". If amdgpu_cs_submit_raw were to return ECANCELED, the correct error message would be printed.

We don't do any of the "trying to continue" business because back when we last discussed that we said that it wasn't such a great idea, and to be honest, it really isn't a great idea for normal applications. For the X server / compositor it could be valuable though.

Cheers,
Nicolai

Quoted 123 lines · Marek

Mesa does not handle -ECANCELED. It only returns -ECANCELED from the Mesa winsys layer if the CS ioctl wasn't called (because the context is already lost and so the winsys doesn't submit further CS ioctls).

When the CS ioctl fails for the first time, the kernel error is returned and the context is marked as "lost".

The next command submission is automatically dropped by the winsys and it returns -ECANCELED.

Marek

------------------------------------------------------------------------
*From:* Haehnle, Nicolai
*Sent:* Monday, October 9, 2017 2:58:02 PM
*To:* Koenig, Christian; Liu, Monk; Nicolai Hähnle; [email protected]; Olsak, Marek *Cc:* Li, Bingley
*Subject:* Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 9 lines · Christian König

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

Quoted 8 lines · Christian König
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either. E.g. you could have an application which uploads shaders, a GPU reset happens and VRAM is lost and then the application creates a new context and makes submission with broken shader binaries.

Hmm. Here's how I imagined we'd be using a VRAM lost counter:

int si_shader_binary_upload(...)
{
    ...
    shader->bo_vram_lost_counter = sscreen->vram_lost_counter; shader->bo = pipe_buffer_create(...);
    ptr = sscreen->b.ws->buffer_map(shader->bo->buf, ...); ... copies ... sscreen->b.ws->buffer_unmap(shader->bo->buf);
}

int si_shader_select(...)
{
    ...
    r = si_shader_select_with_key(ctx->sscreen, state, ...); if (r) return r;

    if (state->current->bo_vram_lost_counter !=
        ctx->sscreen->vram_lost_counter) {
       ... re-upload sequence ...
    }
}

(Not shown: logic that compares ctx->vram_lost_counter with sscreen->vram_lost_counter and forces a re-validation of all state including shaders.)

That should cover this scenario, shouldn't it?

Oh... I see one problem. But it should be easy to fix: when creating a new amdgpu context, Mesa needs to query the vram lost counter. So then the sequence of events would be either:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Mesa creates a new amdgpu context, queries the VRAM lost counter --> 1
- si_screen::vram_lost_counter is updated to 1
- Draw happens on the new context --> si_shader_select will catch the VRAM loss

Or:

- VRAM lost counter starts at 0
- Mesa uploads a shader binary
- Mesa creates a new amdgpu context, VRAM lost counter still 0
- Unrelated GPU reset happens, kernel increments VRAM lost counter to 1
- Draw happens on the new context and proceeds normally
...
- Mesa flushes the CS, and the kernel will return an error code because the device VRAM lost counter is different from the amdgpu context VRAM lost counter

Quoted 1 line

So I would still vote for a separate IOCTL to reset the VRAM lost state which is called *before" user space starts to reupload shader/descriptors etc...

The question is: is that separate IOCTL per-context or per-fd? If it's per-fd, then it's not compatible with multiple drivers. If it's per-context, then I don't see how it helps. Perhaps you could explain?

 > This way you also catch the case when another reset happens while you > re-upload things.

My assumption would be that the re-upload happens *after* the new amdgpu context is created. Then the repeat reset should be caught by the kernel when we try to submit a CS on the new context (this is assuming that the kernel context's vram lost counter is initialized properly when the context is created):

- Mesa prepares upload, sets shader->bo_vram_lost_counter to 0
- Mesa uploads a shader binary
- While doing this, a GPU reset happens[0], kernel increments device VRAM lost counter to 1
- Draw happens with the new shader, Mesa proceeds normally
...
- On flush / CS submit, the kernel detects the VRAM lost state and returns an error to Mesa

[0] Out of curiosity: What happens on the CPU side if the PCI / full ASIC reset method is used? Is there a time window where we could get a SEGV?

[snip]

Quoted 4 lines
Quoted 1 line

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still somehow note the fatality of loosing VRAM to userspace.

Quoted 248 lines
Quoted 1 line

How about ENODATA or EBADFD?

According to the manpage, EBADFD is "File descriptor in bad state.". Sounds fitting :)

Cheers,
Nicolai

Quoted 238 lines

Regards,
Christian.

Quoted 233 lines

Cheers,
Nicolai

Quoted 227 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 221 lines · Christian König
Quoted 1 line

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 209 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 201 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 192 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty,
also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME,
also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED ,  we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König
Quoted 4 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences? That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 135 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 128 lines

Christian

Quoted 3 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences IOCTL) founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 56 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
       [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad

       [<ffffffffffffffff>] 0xffffffffffffffff

---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-

    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
         amd_sched_rq_remove_entity(rq, entity);
         if (r) {
             struct amd_sched_job *job;
             /* Park the kernel for a moment to make sure it isn't

processing
             * our enity.

Quoted 8 lines

             */
            kthread_park(sched->thread); kthread_unpark(sched->thread);
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) +        while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+            struct amd_sched_fence *s_fence = job->s_fence; +            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 5 lines

+ amd_sched_fence_finished(s_fence);
+ dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }

Quoted 8 lines

        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity = container_of(cb, struct amd_sched_entity, cb); entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#39 lore raw

RE: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive

What about below plan?

1, ECANCELED is used for kernel to return an error upon guilty context submitting jobs, In order to prevent guilty context from submitting anymore ... or prevent all contexts from submitting if VRAM lost happens 2, ENODEV is used for kernel to return error in cs_wait/fences_wait IOCTL, so UMD can be notified that the fence they are waiting on will never accomplished but just fake signaled due to GPU hang
If you guys are picky on ENODEV, maybe ETIME/EBADFD ??

3, ENODEV is used for kernel to return error in amdgpu_info_ioctl so UMD can call info_ioctl to acknowledge that the GPU is not work anymore, and after GPU reset done, info_ioctl can work as expected
Use info_ioctl there is because it doesn't need context as input parameter, because for VK UMD sometimes it want to query GPU healthy but no context on hand, only with FD/DEVICE

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月9日 20:34
To: Haehnle, Nicolai <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]; Olsak, Marek <[email protected]>
Cc: Li, Bingley <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 09.10.2017 um 13:27 schrieb Nicolai Hähnle:

Quoted 12 lines · Christian König
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

Ok, let me refine the question: I assume there are resources "shared" between contexts like binary shader code for example which needs to be reuploaded when VRAM is lost.

Quoted 7 lines
Quoted 1 line

How hard would it be to handle that correctly?

Okay, that makes more sense :)

With the current interface it's still pretty difficult, but if we could get a new per-device query ioctl which returns a "VRAM loss counter", it would be reasonably straight-forward.

The problem with the VRAM lost counter is that this isn't save either.
E.g. you could have an application which uploads shaders, a GPU reset happens and VRAM is lost and then the application creates a new context and makes submission with broken shader binaries.

So I would still vote for a separate IOCTL to reset the VRAM lost state which is called *before" user space starts to reupload shader/descriptors etc...

This way you also catch the case when another reset happens while you reupload things.

Quoted 33 lines
Quoted 7 lines
Quoted 4 lines

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

I thought of that on top of the -ENODEV handling.

Quoted 7 lines

In other words when we see -ENODEV we call an IOCTL to let the kernel know we noticed that something is wrong and then reinit all shared resources in userspace.

All existing context will still see -ECANCELED when we drop their command submission, but new contexts would at least not cause a new lockup immediately because their shader binaries are corrupted.

I don't think we need -ENODEV for this. We just need -ECANCELED to be returned when a submission is rejected due to reset (hang or VRAM loss).

Mesa would keep a shadow of the VRAM loss counter in pipe_screen and pipe_context, and query the kernel's counter when it encounters -ECANCELED. Each context would then know to drop the CS it's built up so far and restart based on comparing the VRAM loss counter of pipe_screen to that of pipe_context, and similarly we could keep a copy of the VRAM loss counter for important buffer objects like shader binaries, descriptors, etc.

This seems more robust to me than relying only on an ENODEV. We'd most likely keep some kind of VRAM loss counter in Mesa *anyway* (we don't maintain a list of all shaders, for example, and we can't nuke important per-context across threads), and synthesizing such a counter from ENODEVs is not particularly robust (what if multiple ENODEVs occur for the same loss event?).

Quoted 3 lines

BTW, I still don't like ENODEV. It seems more like the kind of error code you'd return with hot-pluggable GPUs where the device can physically disappear...

Yeah, ECANCELED sounds like a better alternative. But I think we should still somehow note the fatality of loosing VRAM to userspace.

How about ENODATA or EBADFD?

Regards,
Christian.

Quoted 269 lines

Cheers,
Nicolai

Quoted 263 lines

Regards,
Christian.

Am 09.10.2017 um 13:04 schrieb Nicolai Hähnle:

Quoted 257 lines · Christian König
Quoted 2 lines

Nicolai, how hard would it be to handle ENODEV as failure for all currently existing contexts?

Impossible? "All currently existing contexts" is not a well-defined concept when multiple drivers co-exist in the same process.

And what would be the purpose of this? If it's to support VRAM loss, having a per-context VRAM loss counter would enable each context to signal ECANCELED separately.

Cheers,
Nicolai

Quoted 241 lines

Monk, would it be ok with you when we return ENODEV only for existing context when VRAM is lost and/or we have a strict mode GPU reset? E.g. newly created contexts would continue work as they should.

Regards,
Christian.

Am 09.10.2017 um 12:49 schrieb Nicolai Hähnle:

Quoted 231 lines

Hi Monk,

Yes, you're right, we're only using ECANCELED internally. But as a consequence, Mesa would already handle a kernel error of ECANCELED on context loss correctly :)

Cheers,
Nicolai

Quoted 220 lines · Monk

Hi Christian

You reject some of my patches that returns -ENODEV, with the cause that MESA doesn't do the handling on -ENODEV

But if Nicolai can confirm that MESA do have a handling on -ECANCELED, then we need to overall align our error code, on detail below IOCTL can return error code:

Amdgpu_cs_ioctl
Amdgpu_cs_wait_ioctl
Amdgpu_cs_wait_fences_ioctl
Amdgpu_info_ioctl

My patches is:
return -ENODEV on cs_ioctl if the context is detected guilty, also return -ENODEV on cs_wait|cs_wait_fences if the fence is signaled but with error -ETIME, also return -ENODEV on info_ioctl so UMD can query if gpu reset happened after the process created (because for strict mode we block process instead of context)

according to Nicolai:

amdgpu_cs_ioctl *can* return -ECANCELED, but to be frankly speaking, kernel part doesn't have any place with "-ECANCELED" so this solution on MESA side doesn't align with *current* amdgpu driver, which only return 0 on success or -EINVALID on other error but definitely no "-ECANCELED" error code,

so if we talking about community rules we shouldn't let MESA handle -ECANCELED , we should have a unified error code

+ Marek

BR Monk

-----Original Message-----
From: Haehnle, Nicolai
Sent: 2017年10月9日 18:14
To: Koenig, Christian <[email protected]>; Liu, Monk <[email protected]>; Nicolai Hähnle <[email protected]>; [email protected]
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 11 lines · Christian König
Quoted 6 lines

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences
IOCTL)
founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Well that is only closed source behavior which is completely irrelevant for upstream development.

As far as I know we haven't pushed the change to return -ENODEV upstream.

FWIW, radeonsi currently expects -ECANCELED on CS submissions and treats those as context lost. Perhaps we could use the same error on fences?
That makes more sense to me than -ENODEV.

Cheers,
Nicolai

Quoted 148 lines

Regards,
Christian.

Am 09.10.2017 um 08:42 schrieb Liu, Monk:

Quoted 141 lines

Christian

Quoted 4 lines

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error()
for this.

For gpu reset patches (already submitted to pub) I would make kernel return -ENODEV if the waiting fence (in cs_wait or wait_fences
IOCTL)
founded as error, that way UMD would run into robust extension path and considering the GPU hang occurred,

Don't know if this is expected for the case of normal process being killed or crashed like Nicolai hit ... since there is no gpu hang hit

BR Monk

-----Original Message-----
From: amd-gfx [mailto:[email protected]] On Behalf Of Christian K?nig
Sent: 2017年9月28日 23:01
To: Nicolai Hähnle <[email protected]>; [email protected]
Cc: Haehnle, Nicolai <[email protected]>
Subject: Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 28.09.2017 um 16:55 schrieb Nicolai Hähnle:

Quoted 68 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a
pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the
job's fences were never signaled, the buffer object was effectively
leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

       [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
       [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
       [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
       [<ffffffffc03ce56f>] 
ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
       [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
       [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
       [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
       [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 
[ttm]
       [<ffffffffc042f523>] 
amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
       [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
       [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140
[amdgpu]
       [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120
[amdgpu]
       [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
       [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
       [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
       [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
       [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
       [<ffffffffffffffff>] 0xffffffffffffffff
---
    drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
    1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct
 amd_gpu_scheduler *sched,
 amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
            struct amd_sched_job *job;
            /* Park the kernel for a moment to make sure it isn't

processing

             * our enity.
             */
            kthread_park(sched->thread);
            kthread_unpark(sched->thread);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -1
-        while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+        while (kfifo_out(&entity->job_queue, &job, 

sizeof(job))) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c

Quoted 2 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
+            struct amd_sched_fence *s_fence = job->s_fence;
+            amd_sched_fence_scheduled(s_fence);

It would be really nice to have an error code set on s_fence->finished before it is signaled, use dma_fence_set_error() for this.

Additional to that it would be nice to note in the subject line that
this is a rather important bug fix.

With that fixed the whole series is Reviewed-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 14 lines

+ amd_sched_fence_finished(s_fence);

+ dma_fence_put(&s_fence->finished);
                sched->ops->free_job(job);
+        }
        }
        kfifo_free(&entity->job_queue);
    }
    static void amd_sched_entity_wakeup(struct dma_fence *f, 
struct
dma_fence_cb *cb)
    {
        struct amd_sched_entity *entity =
            container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#40 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive

Thanks for this series. I can finally finish piglit on VI.

Marek

Quoted 72 lines · Nicolai Hähnle

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

    [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
    [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
    [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
    [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
    [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
    [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
    [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
    [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
    [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
    [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
    [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
    [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
    [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
    [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
    [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
    [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
    [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
    [<ffffffffffffffff>] 0xffffffffffffffff
---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -2
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
                                        amd_sched_entity_is_idle(entity));
        amd_sched_rq_remove_entity(rq, entity);
        if (r) {
                struct amd_sched_job *job;
 
                /* Park the kernel for a moment to make sure it isn't processing
                 * our enity.
                 */
                kthread_park(sched->thread);
                kthread_unpark(sched->thread);
-               while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+               while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+                       struct amd_sched_fence *s_fence = job->s_fence;
+                       amd_sched_fence_scheduled(s_fence);
+                       amd_sched_fence_finished(s_fence);
+                       dma_fence_put(&s_fence->finished);
                        sched->ops->free_job(job);
+               }
 
        }
        kfifo_free(&entity->job_queue);
 }
 
 static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
 {
        struct amd_sched_entity *entity =
                container_of(cb, struct amd_sched_entity, cb);
        entity->dependency = NULL;
--

2.11.0

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#41 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive
Quoted 7 lines · Nicolai Hähnle

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

Indeed good catch.

Cheers,
David Zhou

Quoted 52 lines

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
     [<ffffffffffffffff>] 0xffffffffffffffff
---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -1
index 54eb77cffd9b..32a99e980d78 100644
@@ -220,22 +220,27 @@ void amd_sched_entity_fini(struct amd_gpu_scheduler *sched,
  					amd_sched_entity_is_idle(entity));
  	amd_sched_rq_remove_entity(rq, entity);
  	if (r) {
  		struct amd_sched_job *job;
 
  		/* Park the kernel for a moment to make sure it isn't processing
  		 * our enity.
  		 */
  		kthread_park(sched->thread);
  		kthread_unpark(sched->thread);
-		while (kfifo_out(&entity->job_queue, &job, sizeof(job)))
+		while (kfifo_out(&entity->job_queue, &job, sizeof(job))) {
+			struct amd_sched_fence *s_fence = job->s_fence;
+			amd_sched_fence_scheduled(s_fence);
+			amd_sched_fence_finished(s_fence);
+			dma_fence_put(&s_fence->finished);
  			sched->ops->free_job(job);
+		}

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_entity_fini(struct amd_gpu_scheduler *sched, · line 246

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
 
  	}
  	kfifo_free(&entity->job_queue);
  }
 
  static void amd_sched_entity_wakeup(struct dma_fence *f, struct dma_fence_cb *cb)
  {
  	struct amd_sched_entity *entity =
  		container_of(cb, struct amd_sched_entity, cb);
  	entity->dependency = NULL;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#42 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive
Quoted 30 lines · Nicolai Hähnle

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

    [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
    [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
    [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
    [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
    [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
    [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
    [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
    [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
    [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
    [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
    [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
    [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
    [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
    [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
    [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
    [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
    [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
    [<ffffffffffffffff>] 0xffffffffffffffff
Quoted 2 lines

Since Christian's commit which introduced the problem (6af0883ed977 "drm/amdgpu: discard commands of killed processes") is in 4.14, we need a solution for that. Should we backport Nicolai's five commits fixing the problem, or revert 6af0883ed977?

While looking into this, I noticed that the following commits by Christian in 4.14 each also cause hangs for me when running the piglit gpu profile on Tonga:

457e0fee04b0 "drm/amdgpu: remove the GART copy hack" 1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"

Are there fixes for these that can be backported to 4.14, or do they need to be reverted there?

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 7cb63e4c-9b65-b9b9-14dc-26368ca7126a-otUistvHUpPR7s880joybQ@public.gmane.org

#44 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 11.10.2017 um 18:30 schrieb Michel Dänzer:

Quoted 43 lines · Nicolai Hähnle
Quoted 30 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>] reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0 [ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470 [amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
     [<ffffffffffffffff>] 0xffffffffffffffff
Quoted 2 lines

Since Christian's commit which introduced the problem (6af0883ed977 "drm/amdgpu: discard commands of killed processes") is in 4.14, we need a solution for that. Should we backport Nicolai's five commits fixing the problem, or revert 6af0883ed977?

While looking into this, I noticed that the following commits by Christian in 4.14 each also cause hangs for me when running the piglit gpu profile on Tonga:

Quoted 5 lines

457e0fee04b0 "drm/amdgpu: remove the GART copy hack" 1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"

Are there fixes for these that can be backported to 4.14, or do they need to be reverted there?

Well I'm not aware that any of those two can cause problems.

For "drm/amdgpu: remove the GART copy hack" I also don't have the slightest idea how that could be an issue. It just removes an unused code path.

Is amd-staging-drm-next stable for you?

Thanks,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): c67d1bd8-81a0-4133-c3df-dd2a1b1a8c11-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

#46 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 43 lines · Christian König

Am 11.10.2017 um 18:30 schrieb Michel Dänzer:

Quoted 42 lines · Nicolai Hähnle
Quoted 33 lines

From: Nicolai Hähnle <[email protected]>

Highly concurrent Piglit runs can trigger a race condition where a pending
SDMA job on a buffer object is never executed because the corresponding process is killed (perhaps due to a crash). Since the job's fences were never signaled, the buffer object was effectively leaked. Worse, the buffer was stuck wherever it happened to be at the time, possibly in VRAM.

The symptom was user space processes stuck in interruptible waits with kernel stacks like:

     [<ffffffffbc5e6722>] dma_fence_default_wait+0x112/0x250
     [<ffffffffbc5e6399>] dma_fence_wait_timeout+0x39/0xf0
     [<ffffffffbc5e82d2>]
reservation_object_wait_timeout_rcu+0x1c2/0x300
     [<ffffffffc03ce56f>] ttm_bo_cleanup_refs_and_unlock+0xff/0x1a0
[ttm]
     [<ffffffffc03cf1ea>] ttm_mem_evict_first+0xba/0x1a0 [ttm]
     [<ffffffffc03cf611>] ttm_bo_mem_space+0x341/0x4c0 [ttm]
     [<ffffffffc03cfc54>] ttm_bo_validate+0xd4/0x150 [ttm]
     [<ffffffffc03cffbd>] ttm_bo_init_reserved+0x2ed/0x420 [ttm]
     [<ffffffffc042f523>] amdgpu_bo_create_restricted+0x1f3/0x470
[amdgpu]
     [<ffffffffc042f9fa>] amdgpu_bo_create+0xda/0x220 [amdgpu]
     [<ffffffffc04349ea>] amdgpu_gem_object_create+0xaa/0x140 [amdgpu]
     [<ffffffffc0434f97>] amdgpu_gem_create_ioctl+0x97/0x120 [amdgpu]
     [<ffffffffc037ddba>] drm_ioctl+0x1fa/0x480 [drm]
     [<ffffffffc041904f>] amdgpu_drm_ioctl+0x4f/0x90 [amdgpu]
     [<ffffffffbc23db33>] do_vfs_ioctl+0xa3/0x5f0
     [<ffffffffbc23e0f9>] SyS_ioctl+0x79/0x90
     [<ffffffffbc864ffb>] entry_SYSCALL_64_fastpath+0x1e/0xad
Quoted 4 lines

     [<ffffffffffffffff>] 0xffffffffffffffff

Since Christian's commit which introduced the problem (6af0883ed977 "drm/amdgpu: discard commands of killed processes") is in 4.14, we need a solution for that. Should we backport Nicolai's five commits fixing the problem, or revert 6af0883ed977?

BTW, any preference for this Christian or Nicolai?

Quoted 7 lines
Quoted 7 lines

While looking into this, I noticed that the following commits by Christian in 4.14 each also cause hangs for me when running the piglit gpu profile on Tonga:

457e0fee04b0 "drm/amdgpu: remove the GART copy hack" 1d00402b4da2 "drm/amdgpu: fix amdgpu_ttm_bind"

Quoted 7 lines
Quoted 2 lines

Are there fixes for these that can be backported to 4.14, or do they need to be reverted there?

Well I'm not aware that any of those two can cause problems.

For "drm/amdgpu: remove the GART copy hack" I also don't have the slightest idea how that could be an issue. It just removes an unused code path.

I also thought it's weird, and indeed I can no longer reproduce a hang with only 457e0fee04b0; but I still can with only 1d00402b4da2. I guess one of my bisections went wrong and incorrectly identified 457e0fee04b0 instead of 1d00402b4da2.

Quoted 1 line

Is amd-staging-drm-next stable for you?

It seemed stable before the changes you pushed this morning. :) As of cfb6dee86711 "drm/ttm: add transparent huge page support for cached allocations v2", I get a flood of

[TTM] Erroneous page count. Leaking pages.

in dmesg while running piglit, and it eventually hangs[0].

Anyway, unless anyone knows which commits from amd-staging-drm-next are needed to make 1d00402b4da2 stable in 4.14, the safe course of action seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

[0] I also got this, but I don't know yet if it's related:

 BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
 IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 PGD 0 
 P4D 0 
 
 Oops: 0000 [#1] SMP
 Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd soundcore rng_core sg wmi parport_pc parport i2c_designware_platform i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
  jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore shpchp gpio_amdpt gpio_generic
 CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: G        W  O    4.13.0-rc5+ #28
 Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
 task: ffff9d2982c75a00 task.stack: ffffb2744e9bc000
 RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 RSP: 0018:ffffb2744e9bf6e8 EFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff9d2848642820 RCX: ffff9d28c77fdae0
 RDX: 0000000000000001 RSI: ffff9d28c77fd800 RDI: ffff9d288f286008
 RBP: ffffb2744e9bf728 R08: 000000ffffffffff R09: 0000000000000000
 R10: 0000000000000078 R11: ffff9d298ba170a0 R12: ffff9d28c77fd800
 R13: 0000000000000001 R14: ffff9d288f286000 R15: ffff9d2848642800
 FS:  00007f809fc5c300(0000) GS:ffff9d298e940000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000220 CR3: 000000030e05a000 CR4: 00000000003406e0
 Call Trace:
  amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
  ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
  ? ttm_bo_mem_space+0x391/0x580 [ttm]
  ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
  ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
  ttm_bo_mem_space+0x306/0x580 [ttm]
  ttm_bo_validate+0xd4/0x150 [ttm]
  ttm_bo_init_reserved+0x22e/0x440 [ttm]
  amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
  ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
  amdgpu_bo_create+0x50/0x2b0 [amdgpu]
  amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
  amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
  ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
  drm_ioctl_kernel+0x5d/0xf0 [drm]
  drm_ioctl+0x32a/0x630 [drm]
  ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
  ? lru_cache_add_active_or_unevictable+0x36/0xb0
  ? __handle_mm_fault+0x90d/0xff0
  amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
  do_vfs_ioctl+0xa5/0x600
  ? handle_mm_fault+0xd8/0x230
  ? __do_page_fault+0x267/0x4c0
  SyS_ioctl+0x79/0x90
  entry_SYSCALL_64_fastpath+0x1e/0xa9
 RIP: 0033:0x7f809c8f3dc7
 RSP: 002b:00007ffcc8c485f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
 RAX: ffffffffffffffda RBX: 00007f809cbaab00 RCX: 00007f809c8f3dc7
 RDX: 00007ffcc8c48640 RSI: 00000000c0206440 RDI: 0000000000000006
 RBP: 0000000040000010 R08: 00007f809cbaabe8 R09: 0000000000000060
 R10: 0000000000000004 R11: 0000000000000246 R12: 0000000040001000
 R13: 00007f809cbaab58 R14: 0000000000001000 R15: 00007f809cbaab00
 Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 84 24 20 02 00 00 0f 84 ab 00 00 00 
 RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: ffffb2744e9bf6e8
 CR2: 0000000000000220
Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 51ec8d88-32eb-ef4a-b34b-d2fd8e23281e-otUistvHUpPR7s880joybQ@public.gmane.org

#48 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 12.10.2017 um 13:00 schrieb Michel Dänzer:

Quoted 10 lines · Christian König

[SNIP]

Quoted 1 line

Is amd-staging-drm-next stable for you?

It seemed stable before the changes you pushed this morning. :) As of cfb6dee86711 "drm/ttm: add transparent huge page support for cached allocations v2", I get a flood of

[TTM] Erroneous page count. Leaking pages.

in dmesg while running piglit, and it eventually hangs[0].

Great, going to take a look.

Quoted 3 lines

Anyway, unless anyone knows which commits from amd-staging-drm-next are needed to make 1d00402b4da2 stable in 4.14, the safe course of action seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix placement flags in amdgpu_ttm_bind". But I've assumed they went both into 4.14.

Thanks for the bugreport,
Christian.

Quoted 53 lines

[0] I also got this, but I don't know yet if it's related:

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
  IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  PGD 0
  P4D 0
  
  Oops: 0000 [#1] SMP
  Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd soundcore rng_core sg wmi parport_pc parport i2c_designware_platform i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
   jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore shpchp gpio_amdpt gpio_generic
  CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: G        W  O    4.13.0-rc5+ #28
  Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
  task: ffff9d2982c75a00 task.stack: ffffb2744e9bc000
  RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  RSP: 0018:ffffb2744e9bf6e8 EFLAGS: 00010202
  RAX: 0000000000000000 RBX: ffff9d2848642820 RCX: ffff9d28c77fdae0
  RDX: 0000000000000001 RSI: ffff9d28c77fd800 RDI: ffff9d288f286008
  RBP: ffffb2744e9bf728 R08: 000000ffffffffff R09: 0000000000000000
  R10: 0000000000000078 R11: ffff9d298ba170a0 R12: ffff9d28c77fd800
  R13: 0000000000000001 R14: ffff9d288f286000 R15: ffff9d2848642800
  FS:  00007f809fc5c300(0000) GS:ffff9d298e940000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000220 CR3: 000000030e05a000 CR4: 00000000003406e0
  Call Trace:
   amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
   ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
   ? ttm_bo_mem_space+0x391/0x580 [ttm]
   ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
   ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
   ttm_bo_mem_space+0x306/0x580 [ttm]
   ttm_bo_validate+0xd4/0x150 [ttm]
   ttm_bo_init_reserved+0x22e/0x440 [ttm]
   amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
   ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
   amdgpu_bo_create+0x50/0x2b0 [amdgpu]
   amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
   amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   drm_ioctl_kernel+0x5d/0xf0 [drm]
   drm_ioctl+0x32a/0x630 [drm]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   ? lru_cache_add_active_or_unevictable+0x36/0xb0
   ? __handle_mm_fault+0x90d/0xff0
   amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
   do_vfs_ioctl+0xa5/0x600
   ? handle_mm_fault+0xd8/0x230
   ? __do_page_fault+0x267/0x4c0
   SyS_ioctl+0x79/0x90
   entry_SYSCALL_64_fastpath+0x1e/0xa9
  RIP: 0033:0x7f809c8f3dc7
  RSP: 002b:00007ffcc8c485f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Quoted 10 lines
  RAX: ffffffffffffffda RBX: 00007f809cbaab00 RCX: 00007f809c8f3dc7
  RDX: 00007ffcc8c48640 RSI: 00000000c0206440 RDI: 0000000000000006
  RBP: 0000000040000010 R08: 00007f809cbaabe8 R09: 0000000000000060
  R10: 0000000000000004 R11: 0000000000000246 R12: 0000000040001000
  R13: 00007f809cbaab58 R14: 0000000000001000 R15: 00007f809cbaab00
  Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 84 24 20 02 00 00 0f 84 ab 00 00 00
  RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: ffffb2744e9bf6e8
  CR2: 0000000000000220

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 4c750ed5-98be-eafa-e684-940ecb2787f0-5C7GfCeVMHo@public.gmane.org

#50 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König

Am 12.10.2017 um 13:00 schrieb Michel Dänzer:

Quoted 3 lines

Anyway, unless anyone knows which commits from amd-staging-drm-next are needed to make 1d00402b4da2 stable in 4.14, the safe course of action seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix placement flags in amdgpu_ttm_bind".

Indeed, that fixes it for me.

Quoted 1 line

But I've assumed they went both into 4.14.

Unfortunately, it looks like only 1d00402b4da2 made it into 4.14. Alex, please send a fixes pull for 4.14 with a backport of 70a9c6b.

For the other issue, do we want to backport Nicolai's commits 6b37d03280a4..318d85de9c20 or revert 6af0883ed977?

Christian, can you check that there are no other fixes missing from 4.14?

BTW, this raises an issue: Since we push both fixes and new development work to the same internal branch, sometimes it isn't clear which changes should go upstream via -fixes or -next. Any ideas for mitigating the risk of missing an important fix?

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): bc0e87da-a632-07ce-6934-86aee099b916-otUistvHUpPR7s880joybQ@public.gmane.org

#52 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 12.10.2017 um 15:42 schrieb Michel Dänzer:

Quoted 10 lines · Christian König
Quoted 7 lines

Am 12.10.2017 um 13:00 schrieb Michel Dänzer:

Quoted 3 lines

Anyway, unless anyone knows which commits from amd-staging-drm-next are needed to make 1d00402b4da2 stable in 4.14, the safe course of action seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix placement flags in amdgpu_ttm_bind".

Indeed, that fixes it for me.

Quoted 8 lines
Quoted 1 line

But I've assumed they went both into 4.14.

Unfortunately, it looks like only 1d00402b4da2 made it into 4.14. Alex, please send a fixes pull for 4.14 with a backport of 70a9c6b.

For the other issue, do we want to backport Nicolai's commits 6b37d03280a4..318d85de9c20 or revert 6af0883ed977?

Christian, can you check that there are no other fixes missing from 4.14?

Not that I know of, and manually checking is nearly impossible since I don't know what Alex pushed to 4.14 and what not.

Manually checking what went in and what not would take quite some time.

Quoted 4 lines

BTW, this raises an issue: Since we push both fixes and new development work to the same internal branch, sometimes it isn't clear which changes should go upstream via -fixes or -next. Any ideas for mitigating the risk of missing an important fix?

Well we would need to drop that amd-staging-* model and revert back to something amd-drm-next-*/amd-drm-fixes-* based.

But that is not something we were able to sell to our internal teams so far.

Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 609e2516-d783-597c-d771-21dc89091043-5C7GfCeVMHo@public.gmane.org

#54 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 18 lines · Christian König

Am 12.10.2017 um 15:42 schrieb Michel Dänzer:

Quoted 17 lines · Christian König
Quoted 7 lines

Am 12.10.2017 um 13:00 schrieb Michel Dänzer:

Quoted 3 lines

Anyway, unless anyone knows which commits from amd-staging-drm-next are needed to make 1d00402b4da2 stable in 4.14, the safe course of action seems to be reverting it (and ac7afe6b3cf3, which depends on it)?

The amdgpu_ttm_bind change should be fixed by "70a9c6b drm/amdgpu: fix placement flags in amdgpu_ttm_bind".

Indeed, that fixes it for me.

Quoted 1 line

But I've assumed they went both into 4.14.

Unfortunately, it looks like only 1d00402b4da2 made it into 4.14. Alex, please send a fixes pull for 4.14 with a backport of 70a9c6b.

For the other issue, do we want to backport Nicolai's commits 6b37d03280a4..318d85de9c20 or revert 6af0883ed977?

Quoted 6 lines
Quoted 1 line

Christian, can you check that there are no other fixes missing from 4.14?

Not that I know of, and manually checking is nearly impossible since I don't know what Alex pushed to 4.14 and what not.

Manually checking what went in and what not would take quite some time.

Looking at commits which are in drm-next-4.15 / amd-staging-drm-next but not in Linus' tree should be a good start. Something like:

gitk v4.14-rc4..origin/amd-staging-drm-next drivers/gpu/drm/{ttm,amd} (in an internal tree)

gitk v4.14-rc4..<remote for Alex's tree>/drm-next-4.15 drivers/gpu/drm/{ttm,amd}

Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#55 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

In reply to a message not in the archive
Quoted 2 lines · Michel Dänzer

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the huge page related changes reverted. Unfortunately, it doesn't seem to happen reliably on every piglit run.

Even before your changes this morning, there's another hang which doesn't happen every time, without any corresponding dmesg output.

Lots of "fun" in amd-staging-drm-next...

Quoted 59 lines
 BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
 IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 PGD 0 
 P4D 0 
 
 Oops: 0000 [#1] SMP
 Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd soundcore rng_core sg wmi parport_pc parport i2c_designware_platform i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
  jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore shpchp gpio_amdpt gpio_generic
 CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: G        W  O    4.13.0-rc5+ #28
 Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
 task: ffff9d2982c75a00 task.stack: ffffb2744e9bc000
 RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
 RSP: 0018:ffffb2744e9bf6e8 EFLAGS: 00010202
 RAX: 0000000000000000 RBX: ffff9d2848642820 RCX: ffff9d28c77fdae0
 RDX: 0000000000000001 RSI: ffff9d28c77fd800 RDI: ffff9d288f286008
 RBP: ffffb2744e9bf728 R08: 000000ffffffffff R09: 0000000000000000
 R10: 0000000000000078 R11: ffff9d298ba170a0 R12: ffff9d28c77fd800
 R13: 0000000000000001 R14: ffff9d288f286000 R15: ffff9d2848642800
 FS:  00007f809fc5c300(0000) GS:ffff9d298e940000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000000220 CR3: 000000030e05a000 CR4: 00000000003406e0
 Call Trace:
  amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
  ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
  ? ttm_bo_mem_space+0x391/0x580 [ttm]
  ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
  ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
  ttm_bo_mem_space+0x306/0x580 [ttm]
  ttm_bo_validate+0xd4/0x150 [ttm]
  ttm_bo_init_reserved+0x22e/0x440 [ttm]
  amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
  ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
  amdgpu_bo_create+0x50/0x2b0 [amdgpu]
  amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
  amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
  ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
  drm_ioctl_kernel+0x5d/0xf0 [drm]
  drm_ioctl+0x32a/0x630 [drm]
  ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
  ? lru_cache_add_active_or_unevictable+0x36/0xb0
  ? __handle_mm_fault+0x90d/0xff0
  amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
  do_vfs_ioctl+0xa5/0x600
  ? handle_mm_fault+0xd8/0x230
  ? __do_page_fault+0x267/0x4c0
  SyS_ioctl+0x79/0x90
  entry_SYSCALL_64_fastpath+0x1e/0xa9
 RIP: 0033:0x7f809c8f3dc7
 RSP: 002b:00007ffcc8c485f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
 RAX: ffffffffffffffda RBX: 00007f809cbaab00 RCX: 00007f809c8f3dc7
 RDX: 00007ffcc8c48640 RSI: 00000000c0206440 RDI: 0000000000000006
 RBP: 0000000040000010 R08: 00007f809cbaabe8 R09: 0000000000000060
 R10: 0000000000000004 R11: 0000000000000246 R12: 0000000040001000
 R13: 00007f809cbaab58 R14: 0000000000001000 R15: 00007f809cbaab00
 Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 84 24 20 02 00 00 0f 84 ab 00 00 00 
 RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: ffffb2744e9bf6e8
 CR2: 0000000000000220
Signature · 6 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 6b509b43-a6e9-175b-7d64-87e38c5ea4e2-otUistvHUpPR7s880joybQ@public.gmane.org

#57 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 12.10.2017 um 18:49 schrieb Michel Dänzer:

Quoted 5 lines · Michel Dänzer
Quoted 1 line

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the huge page related changes reverted. Unfortunately, it doesn't seem to happen reliably on every piglit run.

Can you enable KASAN in your kernel, and please look up at which line number amdgpu_vm_bo_invalidate+0x88 is.

Quoted 4 lines

Even before your changes this morning, there's another hang which doesn't happen every time, without any corresponding dmesg output.

Lots of "fun" in amd-staging-drm-next...

Yeah, way to much stuff on my TODO list and not enough time/resources for extensive testing :(

Thanks for the reports,
Christian.

Quoted 53 lines
Quoted 53 lines

  BUG: unable to handle kernel NULL pointer dereference at 0000000000000220
  IP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  PGD 0
  P4D 0
  
  Oops: 0000 [#1] SMP
  Modules linked in: cpufreq_powersave cpufreq_userspace cpufreq_conservative amdkfd(O) edac_mce_amd kvm amdgpu(O) irqbypass crct10dif_pclmul crc32_pclmul chash snd_hda_codec_realtek ghash_clmulni_intel snd_hda_codec_generic snd_hda_codec_hdmi pcbc binfmt_misc ttm(O) efi_pstore snd_hda_intel drm_kms_helper(O) snd_hda_codec nls_ascii drm(O) snd_hda_core nls_cp437 i2c_algo_bit aesni_intel snd_hwdep fb_sys_fops aes_x86_64 crypto_simd vfat syscopyarea glue_helper sysfillrect snd_pcm fat sysimgblt sp5100_tco wmi_bmof ppdev r8169 snd_timer cryptd pcspkr efivars mfd_core mii ccp i2c_piix4 snd soundcore rng_core sg wmi parport_pc parport i2c_designware_platform i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq sunrpc nct6775 hwmon_vid efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache
   jbd2 fscrypto raid10 raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev hid_generic usbhid hid crc32c_intel ahci libahci xhci_pci libata xhci_hcd scsi_mod usbcore shpchp gpio_amdpt gpio_generic
  CPU: 13 PID: 1075 Comm: max-texture-siz Tainted: G        W  O    4.13.0-rc5+ #28
  Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
  task: ffff9d2982c75a00 task.stack: ffffb2744e9bc000
  RIP: 0010:amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu]
  RSP: 0018:ffffb2744e9bf6e8 EFLAGS: 00010202
  RAX: 0000000000000000 RBX: ffff9d2848642820 RCX: ffff9d28c77fdae0
  RDX: 0000000000000001 RSI: ffff9d28c77fd800 RDI: ffff9d288f286008
  RBP: ffffb2744e9bf728 R08: 000000ffffffffff R09: 0000000000000000
  R10: 0000000000000078 R11: ffff9d298ba170a0 R12: ffff9d28c77fd800
  R13: 0000000000000001 R14: ffff9d288f286000 R15: ffff9d2848642800
  FS:  00007f809fc5c300(0000) GS:ffff9d298e940000(0000) knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 0000000000000220 CR3: 000000030e05a000 CR4: 00000000003406e0
  Call Trace:
   amdgpu_bo_move_notify+0x42/0xd0 [amdgpu]
   ttm_bo_unmap_virtual_locked+0x298/0xac0 [ttm]
   ? ttm_bo_mem_space+0x391/0x580 [ttm]
   ttm_bo_unmap_virtual_locked+0x737/0xac0 [ttm]
   ttm_bo_unmap_virtual_locked+0xa6f/0xac0 [ttm]
   ttm_bo_mem_space+0x306/0x580 [ttm]
   ttm_bo_validate+0xd4/0x150 [ttm]
   ttm_bo_init_reserved+0x22e/0x440 [ttm]
   amdgpu_ttm_placement_from_domain+0x33c/0x580 [amdgpu]
   ? amdgpu_fill_buffer+0x300/0x420 [amdgpu]
   amdgpu_bo_create+0x50/0x2b0 [amdgpu]
   amdgpu_gem_object_create+0x9f/0x110 [amdgpu]
   amdgpu_gem_create_ioctl+0x12f/0x270 [amdgpu]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   drm_ioctl_kernel+0x5d/0xf0 [drm]
   drm_ioctl+0x32a/0x630 [drm]
   ? amdgpu_gem_object_close+0x210/0x210 [amdgpu]
   ? lru_cache_add_active_or_unevictable+0x36/0xb0
   ? __handle_mm_fault+0x90d/0xff0
   amdgpu_drm_ioctl+0x4f/0x1c20 [amdgpu]
   do_vfs_ioctl+0xa5/0x600
   ? handle_mm_fault+0xd8/0x230
   ? __do_page_fault+0x267/0x4c0
   SyS_ioctl+0x79/0x90
   entry_SYSCALL_64_fastpath+0x1e/0xa9
  RIP: 0033:0x7f809c8f3dc7
  RSP: 002b:00007ffcc8c485f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
  RAX: ffffffffffffffda RBX: 00007f809cbaab00 RCX: 00007f809c8f3dc7
  RDX: 00007ffcc8c48640 RSI: 00000000c0206440 RDI: 0000000000000006
Quoted 9 lines
Quoted 8 lines

RBP: 0000000040000010 R08: 00007f809cbaabe8 R09: 0000000000000060
R10: 0000000000000004 R11: 0000000000000246 R12: 0000000040001000
R13: 00007f809cbaab58 R14: 0000000000001000 R15: 00007f809cbaab00

  Code: 49 8b 47 10 48 39 45 d0 4c 8d 78 f0 0f 84 87 00 00 00 4d 8b 37 45 84 ed 41 c6 47 30 01 49 8d 5f 20 49 8d 7e 08 74 19 49 8b 46 58 <48> 8b 80 20 02 00 00 49 39 84 24 20 02 00 00 0f 84 ab 00 00 00
  RIP: amdgpu_vm_bo_invalidate+0x88/0x210 [amdgpu] RSP: ffffb2744e9bf6e8
  CR2: 0000000000000220

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): fcb5f430-5912-0feb-a586-eaf710433d8d-5C7GfCeVMHo@public.gmane.org

#59 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Quoted 8 lines · Christian König

Am 12.10.2017 um 18:49 schrieb Michel Dänzer:

Quoted 5 lines · Michel Dänzer
Quoted 1 line

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the huge page related changes reverted. Unfortunately, it doesn't seem to happen reliably on every piglit run.

Can you enable KASAN in your kernel,

KASAN caught something else at the beginning of piglit, see the attached dmesg excerpt. Not sure it's related though.

amdgpu_job_free_cb+0x13d/0x160 decodes to:

amd_sched_get_job_priority at .../drivers/gpu/drm/amd/amdgpu/../scheduler/gpu_scheduler.h:182

static inline enum amd_sched_priority amd_sched_get_job_priority(struct amd_sched_job *job)

{
	return (job->s_entity->rq - job->sched->sched_rq); <===
}

(inlined by) amdgpu_job_free_cb at .../drivers/gpu/drm/amd/amdgpu/amdgpu_job.c:107

	amdgpu_ring_priority_put(job->ring, amd_sched_get_job_priority(s_job));
Quoted 2 lines

and please look up at which line number amdgpu_vm_bo_invalidate+0x88 is.

Looks like it's this line:

		if (evicted && bo->tbo.resv == vm->root.base.bo->tbo.resv) {

Maybe vm or vm->root.base.bo is NULL?

Signature · 2 lines
Earthling Michel Dänzer               |               http://www.amd.com
Libre software enthusiast             |             Mesa and X developer
[   89.594368] ==================================================================
[   89.594440] BUG: KASAN: use-after-free in amdgpu_job_free_cb+0x13d/0x160 [amdgpu]
[   89.594444] Read of size 8 at addr ffff880367cc22c0 by task kworker/8:1/142
[   89.594449] CPU: 8 PID: 142 Comm: kworker/8:1 Tainted: G        W       4.13.0-rc5+ #29
[   89.594451] Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
[   89.594516] Workqueue: events amd_sched_job_finish [amdgpu]
[   89.594517] Call Trace:
[   89.594522]  dump_stack+0xb8/0x152
[   89.594524]  ? dma_virt_map_sg+0x1fe/0x1fe
[   89.594527]  ? show_regs_print_info+0x62/0x62
[   89.594531]  print_address_description+0x6f/0x280
[   89.594533]  kasan_report+0x27a/0x370
[   89.594596]  ? amdgpu_job_free_cb+0x13d/0x160 [amdgpu]
[   89.594599]  __asan_report_load8_noabort+0x19/0x20
[   89.594662]  amdgpu_job_free_cb+0x13d/0x160 [amdgpu]
[   89.594726]  amd_sched_job_finish+0x36e/0x630 [amdgpu]
[   89.594790]  ? trace_event_raw_event_amd_sched_process_job+0x180/0x180 [amdgpu]
[   89.594792]  ? pick_next_task_fair+0x435/0x15c0
[   89.594795]  ? pwq_dec_nr_in_flight+0x1c2/0x4d0
[   89.594797]  ? cpu_load_update_active+0x330/0x330
[   89.594800]  ? __switch_to+0x685/0xda0
[   89.594801]  ? load_balance+0x3490/0x3490
[   89.594803]  process_one_work+0x8a5/0x1a30
[   89.594805]  ? wq_worker_sleeping+0x86/0x310
[   89.594808]  ? create_worker+0x590/0x590
[   89.594810]  ? __schedule+0x83b/0x1c80
[   89.594813]  ? schedule+0x10e/0x450
[   89.594815]  ? __schedule+0x1c80/0x1c80
[   89.594817]  ? alloc_worker+0x360/0x360
[   89.594819]  ? update_stack_state+0x402/0x780
[   89.594820]  ? update_stack_state+0x402/0x780
[   89.594822]  ? tsc_resume+0x10/0x10
[   89.594824]  worker_thread+0x21f/0x1920
[   89.594825]  ? sched_clock+0x9/0x10
[   89.594826]  ? sched_clock+0x9/0x10
[   89.594828]  ? sched_clock_local+0x43/0x130
[   89.594831]  ? process_one_work+0x1a30/0x1a30
[   89.594832]  ? pick_next_task_fair+0xcd3/0x15c0
[   89.594833]  ? cpu_load_update_active+0x330/0x330
[   89.594835]  ? __switch_to+0x685/0xda0
[   89.594836]  ? load_balance+0x3490/0x3490
[   89.594838]  ? compat_start_thread+0x80/0x80
[   89.594839]  ? sched_clock+0x9/0x10
[   89.594840]  ? sched_clock_local+0x43/0x130
[   89.594843]  ? set_rq_online.part.79+0x130/0x130
[   89.594844]  ? put_prev_entity+0x4e/0x370
[   89.594846]  ? __schedule+0x83b/0x1c80
[   89.594847]  ? kasan_kmalloc+0xad/0xe0
[   89.594849]  ? kmem_cache_alloc_trace+0xe9/0x1f0
[   89.594851]  ? firmware_map_remove+0x80/0x80
[   89.594852]  ? migrate_swap_stop+0x660/0x660
[   89.594855]  ? __schedule+0x1c80/0x1c80
[   89.594856]  ? default_wake_function+0x35/0x50
[   89.594858]  ? __wake_up_common+0xb9/0x150
[   89.594859]  ? print_dl_stats+0x80/0x80
[   89.594861]  kthread+0x310/0x3d0
[   89.594863]  ? process_one_work+0x1a30/0x1a30
[   89.594864]  ? kthread_create_on_node+0xc0/0xc0
[   89.594866]  ret_from_fork+0x25/0x30
[   89.594869] Allocated by task 1701:
[   89.594872]  save_stack_trace+0x1b/0x20
[   89.594873]  save_stack+0x43/0xd0
[   89.594874]  kasan_kmalloc+0xad/0xe0
[   89.594875]  kmem_cache_alloc_trace+0xe9/0x1f0
[   89.594913]  amdgpu_driver_open_kms+0xec/0x3f0 [amdgpu]
[   89.594925]  drm_open+0x7ea/0x13a0 [drm]
[   89.594936]  drm_stub_open+0x2a7/0x420 [drm]
[   89.594939]  chrdev_open+0x24d/0x6f0
[   89.594940]  do_dentry_open+0x5b1/0xd30
[   89.594941]  vfs_open+0xf1/0x260
[   89.594942]  path_openat+0x130a/0x5240
[   89.594944]  do_filp_open+0x23e/0x3c0
[   89.594945]  do_sys_open+0x47a/0x800
[   89.594946]  SyS_open+0x1e/0x20
[   89.594947]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[   89.594949] Freed by task 1737:
[   89.594951]  save_stack_trace+0x1b/0x20
[   89.594952]  save_stack+0x43/0xd0
[   89.594953]  kasan_slab_free+0x72/0xc0
[   89.594954]  kfree+0x94/0x1a0
[   89.594992]  amdgpu_driver_postclose_kms+0x495/0x830 [amdgpu]
[   89.595002]  drm_release+0x9bf/0x1350 [drm]
[   89.595004]  __fput+0x306/0x900
[   89.595005]  ____fput+0xe/0x10
[   89.595006]  task_work_run+0x14d/0x230
[   89.595008]  exit_to_usermode_loop+0x1f5/0x230
[   89.595009]  syscall_return_slowpath+0x1d8/0x240
[   89.595011]  entry_SYSCALL_64_fastpath+0xa7/0xa9
[   89.595013] The buggy address belongs to the object at ffff880367cc2200
                which belongs to the cache kmalloc-2048 of size 2048
[   89.595015] The buggy address is located 192 bytes inside of
                2048-byte region [ffff880367cc2200, ffff880367cc2a00)
[   89.595017] The buggy address belongs to the page:
[   89.595019] page:ffffea000d9f3000 count:1 mapcount:0 mapping:          (null) index:0x0 compound_mapcount: 0
[   89.595022] flags: 0x17fffc000008100(slab|head)
[   89.595026] raw: 017fffc000008100 0000000000000000 0000000000000000 00000001800f000f
[   89.595028] raw: dead000000000100 dead000000000200 ffff88038e00ea00 0000000000000000
[   89.595029] page dumped because: kasan: bad access detected
[   89.595031] Memory state around the buggy address:
[   89.595032]  ffff880367cc2180: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[   89.595034]  ffff880367cc2200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   89.595036] >ffff880367cc2280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   89.595037]                                            ^
[   89.595038]  ffff880367cc2300: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   89.595040]  ffff880367cc2380: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[   89.595041] ==================================================================
[   89.595042] Disabling lock debugging due to kernel taint
[  239.309507] [drm:amdgpu_gem_object_create [amdgpu]] *ERROR* Failed to allocate GEM object (4294967296, 2, 4096, -12)
[  239.340689] [drm:amdgpu_gem_object_create [amdgpu]] *ERROR* Failed to allocate GEM object (4294967296, 2, 4096, -12)
[  323.869197] [drm:amdgpu_gem_object_create [amdgpu]] *ERROR* Failed to allocate GEM object (2147483648, 2, 4096, -12)
[  323.869377] [drm:amdgpu_gem_object_create [amdgpu]] *ERROR* Failed to allocate GEM object (2147483648, 2, 4096, -12)
[  349.580299] kasan: CONFIG_KASAN_INLINE enabled
[  349.580305] kasan: GPF could be caused by NULL-ptr deref or user memory access
[  349.580311] general protection fault: 0000 [#1] SMP KASAN
[  349.580313] Modules linked in: snd_hda_codec_realtek snd_hda_codec_generic cpufreq_powersave cpufreq_userspace cpufreq_conservative binfmt_misc nls_ascii nls_cp437 vfat fat edac_mce_amd kvm amdkfd irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel amdgpu pcbc snd_hda_codec_hdmi efi_pstore chash ttm snd_hda_intel snd_hda_codec drm_kms_helper snd_hda_core snd_hwdep wmi_bmof drm snd_pcm aesni_intel i2c_algo_bit snd_timer aes_x86_64 crypto_simd glue_helper sp5100_tco snd fb_sys_fops ccp syscopyarea r8169 ppdev sysfillrect cryptd pcspkr efivars sysimgblt i2c_piix4 mii sg rng_core mfd_core soundcore wmi parport_pc parport i2c_designware_platform i2c_designware_core button acpi_cpufreq tcp_bbr sch_fq nct6775 hwmon_vid sunrpc efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 fscrypto raid10
[  349.580366]  raid1 raid0 multipath linear md_mod dm_mod sd_mod evdev hid_generic usbhid hid ahci crc32c_intel libahci xhci_pci libata xhci_hcd usbcore scsi_mod shpchp gpio_amdpt gpio_generic
[  349.580385] CPU: 5 PID: 529 Comm: max-texture-siz Tainted: G    B   W       4.13.0-rc5+ #29
[  349.580388] Hardware name: Micro-Star International Co., Ltd. MS-7A34/B350 TOMAHAWK (MS-7A34), BIOS 1.80 09/13/2017
[  349.580392] task: ffff88036d258000 task.stack: ffff880274260000
[  349.580466] RIP: 0010:amdgpu_vm_bo_invalidate+0x277/0xd60 [amdgpu]
[  349.580469] RSP: 0018:ffff880274266810 EFLAGS: 00010216
[  349.580472] RAX: 0000000000000000 RBX: ffff8801d9ac2d00 RCX: 0000000000000000
[  349.580474] RDX: ffff880389c74408 RSI: ffff880389c73da8 RDI: 0000000000000220
[  349.580476] RBP: ffff880274266aa8 R08: 0000000000000010 R09: 0000000000000044
[  349.580479] R10: ffff880274266d88 R11: 0000000000000010 R12: dffffc0000000000
[  349.580481] R13: ffff8801d9ac2d20 R14: ffff880259586600 R15: ffff880389c74400
[  349.580484] FS:  00007ff180510300(0000) GS:ffff88038e540000(0000) knlGS:0000000000000000
[  349.580487] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  349.580489] CR2: 00007fe4e75c1000 CR3: 000000037f1bc000 CR4: 00000000003406e0
[  349.580491] Call Trace:
[  349.580502]  ? ttm_mem_global_alloc_zone.constprop.4+0x1c6/0x290 [ttm]
[  349.580509]  ? ttm_bo_init_reserved+0x9b6/0x1300 [ttm]
[  349.580570]  ? amdgpu_bo_do_create+0x594/0x1420 [amdgpu]
[  349.580634]  ? amdgpu_vm_bo_rmv+0xd60/0xd60 [amdgpu]
[  349.580641]  ? ttm_mem_global_alloc_page+0x9e/0xf0 [ttm]
[  349.580649]  ? ttm_pool_populate+0x46e/0xd10 [ttm]
[  349.580655]  ? kasan_kmalloc+0xad/0xe0
[  349.580664]  ? ttm_pool_unpopulate+0x290/0x290 [ttm]
[  349.580669]  ? kvmalloc_node+0x75/0x80
[  349.580676]  ? ttm_dma_tt_init+0x285/0x620 [ttm]
[  349.580679]  ? kasan_unpoison_shadow+0x35/0x50
[  349.580686]  ? ttm_tt_init+0x300/0x300 [ttm]
[  349.580691]  ? nommu_map_page+0x5c/0xa0
[  349.580754]  ? amdgpu_ttm_backend_bind+0x10f/0xc70 [amdgpu]
[  349.580817]  ? amdgpu_ttm_tt_create+0x132/0x2c0 [amdgpu]
[  349.580879]  amdgpu_bo_move_notify+0x10f/0x350 [amdgpu]
[  349.580942]  ? amdgpu_bo_get_metadata+0x200/0x200 [amdgpu]
[  349.580950]  ttm_bo_handle_move_mem+0x782/0x22f0 [ttm]
[  349.580955]  ? reservation_object_reserve_shared+0x167/0x200
[  349.580962]  ? ttm_bo_add_move_fence.isra.17+0x29/0x160 [ttm]
[  349.580969]  ? ttm_bo_mem_space+0x518/0xdf0 [ttm]
[  349.580974]  ? free_one_page+0x1560/0x1560
[  349.580983]  ttm_bo_evict+0x3ff/0xf70 [ttm]
[  349.580990]  ? ttm_bo_release_list+0x910/0x910 [ttm]
[  349.580998]  ? ttm_bo_handle_move_mem+0x22f0/0x22f0 [ttm]
[  349.581005]  ? ttm_bo_add_to_lru+0x46c/0x870 [ttm]
[  349.581009]  ? kasan_kmalloc_large+0x9c/0xd0
[  349.581017]  ? ttm_mem_global_alloc_zone.constprop.4+0x1c6/0x290 [ttm]
[  349.581020]  ? kmem_cache_alloc+0xb7/0x1c0
[  349.581081]  ? amdgpu_ttm_bo_eviction_valuable+0x199/0x2b0 [amdgpu]
[  349.581143]  ? amdgpu_vram_mgr_new+0x4e4/0x780 [amdgpu]
[  349.581152]  ttm_mem_evict_first+0x312/0x4a0 [ttm]
[  349.581215]  ? amdgpu_vram_mgr_new+0x4e4/0x780 [amdgpu]
[  349.581222]  ? ttm_bo_evict+0xf70/0xf70 [ttm]
[  349.581230]  ttm_bo_mem_space+0x84e/0xdf0 [ttm]
[  349.581233]  ? __shmem_file_setup+0x250/0x520
[  349.581241]  ttm_bo_validate+0x322/0x580 [ttm]
[  349.581244]  ? unwind_dump+0x4e0/0x4e0
[  349.581251]  ? ttm_bo_evict_mm+0xa0/0xa0 [ttm]
[  349.581255]  ? cpufreq_default_governor+0x20/0x20
[  349.581277]  ? drm_add_edid_modes+0x44a0/0x67e0 [drm]
[  349.581286]  ttm_bo_init_reserved+0x9b6/0x1300 [ttm]
[  349.581294]  ? ttm_bo_validate+0x580/0x580 [ttm]
[  349.581298]  ? dentry_path_raw+0x10/0x10
[  349.581302]  ? proc_nr_files+0x30/0x30
[  349.581305]  ? shmem_get_inode+0x668/0x8f0
[  349.581308]  ? shmem_fh_to_dentry+0x160/0x160
[  349.581313]  ? entry_SYSCALL_64_fastpath+0x1e/0xa9
[  349.581318]  ? _copy_to_user+0x90/0x90
[  349.581321]  ? alloc_file+0x16d/0x440
[  349.581324]  ? __shmem_file_setup+0x2e0/0x520
[  349.581327]  ? shmem_fill_super+0xa10/0xa10
[  349.581344]  ? drm_gem_private_object_init+0x189/0x300 [drm]
[  349.581347]  ? kasan_kmalloc+0xad/0xe0
[  349.581412]  amdgpu_bo_do_create+0x594/0x1420 [amdgpu]
[  349.581486]  ? amdgpu_fill_buffer+0xb80/0xb80 [amdgpu]
[  349.581489]  ? update_stack_state+0x402/0x780
[  349.581559]  ? amdgpu_ttm_placement_from_domain+0x8d0/0x8d0 [amdgpu]
[  349.581565]  ? show_initstate+0xb0/0xb0
[  349.581570]  ? bpf_prog_alloc+0x320/0x320
[  349.581574]  ? unwind_next_frame.part.5+0x1bb/0xc90
[  349.581582]  ? __free_insn_slot+0x6a0/0x6a0
[  349.581585]  ? unwind_dump+0x4e0/0x4e0
[  349.581590]  ? rb_erase+0x3540/0x3540
[  349.581595]  ? __mem_cgroup_threshold+0x7b0/0x7b0
[  349.581599]  ? memory_max_write+0x420/0x420
[  349.581605]  ? __kernel_text_address+0xbf/0xf0
[  349.581608]  ? unwind_get_return_address+0x66/0xb0
[  349.581612]  ? __save_stack_trace+0x92/0x100
[  349.581685]  amdgpu_bo_create+0xba/0xa00 [amdgpu]
[  349.581759]  ? amdgpu_bo_do_create+0x1420/0x1420 [amdgpu]
[  349.581763]  ? mem_cgroup_uncharge_swap+0xc0/0xc0
[  349.581767]  ? kmem_cache_alloc+0xb7/0x1c0
[  349.581771]  ? __anon_vma_prepare+0x37a/0x650
[  349.581775]  ? __handle_mm_fault+0x31ac/0x5070
[  349.581778]  ? handle_mm_fault+0x292/0x800
[  349.581782]  ? __do_page_fault+0x412/0xa00
[  349.581785]  ? do_page_fault+0x22/0x30
[  349.581788]  ? page_fault+0x28/0x30
[  349.581792]  ? memcg_oom_wake_function+0x6a0/0x6a0
[  349.581866]  amdgpu_gem_object_create+0x11f/0x240 [amdgpu]
[  349.581942]  ? amdgpu_gem_object_free+0x1d0/0x1d0 [amdgpu]
[  349.581945]  ? __alloc_pages_nodemask+0x3d8/0xe50
[  349.582021]  amdgpu_gem_create_ioctl+0x3bb/0xc10 [amdgpu]
[  349.582097]  ? amdgpu_gem_object_close+0x790/0x790 [amdgpu]
[  349.582101]  ? page_add_new_anon_rmap+0x203/0x3d0
[  349.582105]  ? __check_object_size+0x22e/0x560
[  349.582180]  ? amdgpu_gem_object_close+0x790/0x790 [amdgpu]
[  349.582201]  drm_ioctl_kernel+0x1ce/0x330 [drm]
[  349.582221]  ? drm_ioctl_permit+0x2c0/0x2c0 [drm]
[  349.582225]  ? kasan_check_write+0x14/0x20
[  349.582246]  drm_ioctl+0x79a/0xc30 [drm]
[  349.582321]  ? amdgpu_gem_object_close+0x790/0x790 [amdgpu]
[  349.582342]  ? drm_getstats+0x20/0x20 [drm]
[  349.582347]  ? do_mmap+0x641/0x10f0
[  349.582419]  amdgpu_drm_ioctl+0xd8/0x1b0 [amdgpu]
[  349.582424]  do_vfs_ioctl+0x197/0x1490
[  349.582428]  ? vm_mmap_pgoff+0x1fe/0x2c0
[  349.582431]  ? ioctl_preallocate+0x2c0/0x2c0
[  349.582435]  ? __fget_light+0x2be/0x410
[  349.582438]  ? iterate_fd+0x2e0/0x2e0
[  349.582441]  ? handle_mm_fault+0x292/0x800
[  349.582445]  ? __handle_mm_fault+0x5070/0x5070
[  349.582449]  ? __do_page_fault+0x43a/0xa00
[  349.582453]  SyS_ioctl+0x79/0x90
[  349.582457]  entry_SYSCALL_64_fastpath+0x1e/0xa9
[  349.582461] RIP: 0033:0x7ff17d1a7dc7
[  349.582463] RSP: 002b:00007ffe7a4edbf8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  349.582467] RAX: ffffffffffffffda RBX: 00007ff17d45eb00 RCX: 00007ff17d1a7dc7
[  349.582470] RDX: 00007ffe7a4edc40 RSI: 00000000c0206440 RDI: 0000000000000006
[  349.582472] RBP: 0000000040000010 R08: 00007ff17d45ebc8 R09: 0000000000000060
[  349.582475] R10: 0000000000000004 R11: 0000000000000246 R12: 0000000040001000
[  349.582477] R13: 00007ff17d45eb58 R14: 0000000000001000 R15: 00007ff17d45eb00
[  349.582480] Code: 49 8b b6 20 02 00 00 48 89 f8 48 c1 e8 03 42 80 3c 20 00 0f 85 f7 05 00 00 49 8b 47 58 48 8d b8 20 02 00 00 49 89 f9 49 c1 e9 03 <43> 80 3c 21 00 0f 85 e8 08 00 00 48 3b b0 20 02 00 00 0f 84 d9 
[  349.582596] RIP: amdgpu_vm_bo_invalidate+0x277/0xd60 [amdgpu] RSP: ffff880274266810
[  349.582600] ---[ end trace cc9c171d2cdc0539 ]---
[  360.319061] amdgpu 0000:23:00.0: Disabling VM faults because of PRT request!

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 8ab106b9-363b-4fb2-6f1a-727a5e0e7bc5-otUistvHUpPR7s880joybQ@public.gmane.org

#61 lore raw

Re: [PATCH 5/5] drm/amd/sched: signal and free remaining fences in amd_sched_entity_fini

Am 13.10.2017 um 16:34 schrieb Michel Dänzer:

Quoted 17 lines · Christian König
Quoted 7 lines

Am 12.10.2017 um 18:49 schrieb Michel Dänzer:

Quoted 5 lines · Michel Dänzer
Quoted 1 line

[0] I also got this, but I don't know yet if it's related:

No, that seems to be a separate issue; I can still reproduce it with the huge page related changes reverted. Unfortunately, it doesn't seem to happen reliably on every piglit run.

Can you enable KASAN in your kernel,

KASAN caught something else at the beginning of piglit, see the attached dmesg excerpt. Not sure it's related though.

amdgpu_job_free_cb+0x13d/0x160 decodes to:

amd_sched_get_job_priority at .../drivers/gpu/drm/amd/amdgpu/../scheduler/gpu_scheduler.h:182

static inline enum amd_sched_priority amd_sched_get_job_priority(struct amd_sched_job *job)

Quoted 7 lines
{
	return (job->s_entity->rq - job->sched->sched_rq); <===
}

(inlined by) amdgpu_job_free_cb at .../drivers/gpu/drm/amd/amdgpu/amdgpu_job.c:107

	amdgpu_ring_priority_put(job->ring, amd_sched_get_job_priority(s_job));

Sounds a lot like the code Andres added is buggy somehow. Going to take a look as well.

Quoted 7 lines
Quoted 2 lines

and please look up at which line number amdgpu_vm_bo_invalidate+0x88 is.

Looks like it's this line:

		if (evicted && bo->tbo.resv == vm->root.base.bo->tbo.resv) {

Maybe vm or vm->root.base.bo is NULL?

Ah, of course!

We need to reserve the page directory root when we release it or otherwise we can run into a race with somebody else trying to evict it.

Going to send a patch in a minute,
Christian.
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx