Skip to contentSkip to conversation

*** patch series for new gpu recover***

PATCH NAK Monk Liu 3 participants 30 messages Oct 25, 2017 – Oct 30, 2017 21 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Monk Liu

author
#0 lore raw

cover *** patch series for new gpu recover***

*** BLURB HERE ***

Monk Liu (9):
drm/amdgpu:cleanup hw fence get&put amd/scheduler:imple job skip feature drm/amdgpu:implement new GPU recover drm/amdgpu:replace deprecated gpu reset drm/amdgpu:cleanup in_sriov_reset and lock_reset drm/amdgpu:cleanup ucode_init_bo drm/amdgpu:block kms open during gpu_reset drm/amdgpu/sriov:fix memory leak in psp_load_fw drm/amdgpu:fix random missing of FLR NOTIFY

 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |   9 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    | 311 ++++++++++++--------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c     |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c       |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c       |   3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c       |  22 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c     |   3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c      |   2 -
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h      |   2 -
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c         |   6 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c         |   6 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c         |  16 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c         |   2 +-
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c |  46 ++--
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |   2 +-
 16 files changed, 230 insertions(+), 230 deletions(-)
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-1-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

author
#2 lore raw

PATCH 1/9 drm/amdgpu:cleanup hw fence get&put

In amdgpu_fence_process it shouldn't put the signaled hw fence and set RCU pointer to NULL, instead the more reasonable sequence is just leave the RCU be there untouched, and:
1) Either driver_fini() should put this RCU fence or
2) New hw fence emitting should take the RCU slot and put the old hw fence.

mapping between get & put on hw fence is:

Get Put
fence_emit:fence_init ---> free_job:put job->fence

	       		--->	run_job:put job->fence //GPU RESET case

fence_emit:rcu_assign_pointer ---> fence_driver_fini:put ring->fence_drv.fences[j]

				---> fence_emit:put(old) //after fence_wait(old)

run_job:job->fence=fence_get() --> sched_main:put(fence)

				--> job_recovery:put(fence) //for GPU RESET case

sched_main:parent=fence_get() --> sched_fence_free:put(fence->parent)
sched_job_recovery:parent=fence_get() --> sched_hw_job_reset:put(s_job->s_fence->parent) // for GPU RESET case

Change-Id I623167c1e3143233f4e1be7a400a9b698b4a1355
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +2 -2
index f43319c..d7374cf 100644
@@ -161,6 +161,8 @@ int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f)
 		dma_fence_wait(old, false);
 	}
 
+	dma_fence_put(old);
+
 	rcu_assign_pointer(*ptr, dma_fence_get(&fence->base));
 
 	*f = &fence->base;
@@ -246,7 +248,6 @@ void amdgpu_fence_process(struct amdgpu_ring *ring)
 
 		/* There is always exactly one thread signaling this fence slot */
 		fence = rcu_dereference_protected(*ptr, 1);
-		RCU_INIT_POINTER(*ptr, NULL);
 
 		if (!fence)
 			continue;
@@ -257,7 +258,6 @@ void amdgpu_fence_process(struct amdgpu_ring *ring)
 		else
 			BUG();
 
-		dma_fence_put(fence);
 	} while (last_seq != seq);
 }
 
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-2-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#4 lore raw

Re: [PATCH 1/9] drm/amdgpu:cleanup hw fence get&put

NAK, we have tried this before.

This way we can accidentally try to signal a fence twice when the fence value is somehow corrupted.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 49 lines

In amdgpu_fence_process it shouldn't put the signaled hw fence and set RCU pointer to NULL, instead the more reasonable sequence is just leave the RCU be there untouched, and:
1) Either driver_fini() should put this RCU fence or
2) New hw fence emitting should take the RCU slot and put the old hw fence.

mapping between get & put on hw fence is:

Get Put
fence_emit:fence_init ---> free_job:put job->fence

	       		--->	run_job:put job->fence //GPU RESET case

fence_emit:rcu_assign_pointer ---> fence_driver_fini:put ring->fence_drv.fences[j]

				---> fence_emit:put(old) //after fence_wait(old)

run_job:job->fence=fence_get() --> sched_main:put(fence)

				--> job_recovery:put(fence) //for GPU RESET case

sched_main:parent=fence_get() --> sched_fence_free:put(fence->parent)
sched_job_recovery:parent=fence_get() --> sched_hw_job_reset:put(s_job->s_fence->parent) // for GPU RESET case

Change-Id I623167c1e3143233f4e1be7a400a9b698b4a1355
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +2 -1
index f43319c..d7374cf 100644
@@ -161,6 +161,8 @@ int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f)
  		dma_fence_wait(old, false);
  	}
 
+	dma_fence_put(old);
+
  	rcu_assign_pointer(*ptr, dma_fence_get(&fence->base));
 
  	*f = &fence->base;
@@ -246,7 +248,6 @@ void amdgpu_fence_process(struct amdgpu_ring *ring)
 
  		/* There is always exactly one thread signaling this fence slot */
  		fence = rcu_dereference_protected(*ptr, 1);
-		RCU_INIT_POINTER(*ptr, NULL);
 
  		if (!fence)
  			continue;

drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c void amdgpu_fence_process(struct amdgpu_ring *ring) · line 263

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +0 -1
@@ -257,7 +258,6 @@ void amdgpu_fence_process(struct amdgpu_ring *ring)
  		else
  			BUG();
 
-		dma_fence_put(fence);
  	} while (last_seq != seq);
  }
 

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#5 lore raw

PATCH 2/9 amd/scheduler:imple job skip feature

In reply to a message not in the archive

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++---
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 46 +++++++++++++++++++--------
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
 3 files changed, 39 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +5 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
 	return fence;
 }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job, bool skip)
 {
 	struct dma_fence *fence = NULL;
 	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
 	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
 	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
 		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
 	} else {
 		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
 				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +33 -13

102 lines not on this page · raw message

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +1 -1
index f9e3a83..a8e5a7d 100644
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
 */
 struct amd_sched_backend_ops {
 	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool skip);
 	void (*timedout_job)(struct amd_sched_job *sched_job);
 	void (*free_job)(struct amd_sched_job *sched_job);
 };
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-3-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#7 lore raw

Re: [PATCH 2/9] amd/scheduler:imple job skip feature

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 10 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

with this feature we can introduce new gpu recover feature.

Sorry for the delay, totally swamped once more at the moment.

Quoted 24 lines
Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 46 +++++++++++++++++++--------
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
  3 files changed, 39 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -1
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
  	return fence;
  }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job, bool skip)
  {
  	struct dma_fence *fence = NULL;
  	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
  	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job) · line 201

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +4 -3
 
  	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");

Not really needed, we could just use amd_sched_job_fake_signal() here as well.

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +0 -0
  	} else {
  		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
  				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 9cbeade..29841ba 100644
@@ -330,6 +330,20 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
  	return false;
  }
 

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 342

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +10 -0
+static void amd_sched_job_fake_signal(struct amd_sched_job *job)
+{
+	struct amd_sched_fence *s_fence = job->s_fence;
+
+	dma_fence_set_error(&job->s_fence->finished, -ECANCELED);
+	/* fake signaling the scheduled fence */
+	amd_sched_fence_scheduled(s_fence);
+	/* fake signaling the finished fence */
+	dma_fence_put(&job->s_fence->finished);
+	job->sched->ops->free_job(job);

Well signaling the finished fence will free the job as well, won't it?

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+	/* call CB on the s_fence, e.g. wake up WAIT_CS */
+	amd_sched_fence_finished(s_fence);
+}
+
  static struct amd_sched_job *
  amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 367

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
@@ -345,6 +359,12 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  			return NULL;
 
  	sched_job->s_entity = NULL;
+
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		sched_job = NULL;
+	}
+

This must come after the spsc_queue_pop() below, otherwise you mess up the spsc queue.

Quoted 37 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +3 -9
  	spsc_queue_pop(&entity->job_queue);
  	return sched_job;
  }
@@ -441,13 +461,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
  	job->sched->ops->timedout_job(job);
  }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job)
-{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
  void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
  {
  	struct amd_sched_job *s_job;
@@ -466,7 +479,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  	}
  	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit) {
  		bool found = false;
 
  		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ ) {
@@ -476,7 +489,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  			list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
  				if (bad->s_fence->scheduled.context == entity->fence_context) {
  					found = true;
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
  					break;
  				}
  			}

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 528

Quoted 5 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
@@ -510,9 +524,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool found_guilty = false, skip;
+		uint64_t guilty_context;

Well if you want the variables to have an effect you should probably move them outside the loop. Otherwise the code doesn't make much sense to me.

Regards,
Christian.

Quoted 43 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +8 -4
+
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);
  		atomic_inc(&sched->hw_rq_count);
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence);
@@ -525,7 +547,6 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
  		spin_lock(&sched->job_list_lock);
@@ -650,7 +671,7 @@ static int amd_sched_main(void *param)
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
  		amd_sched_fence_scheduled(s_fence);
 
  		if (fence) {
@@ -664,7 +685,6 @@ static int amd_sched_main(void *param)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
 
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr · line 132

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +1 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
  */
  struct amd_sched_backend_ops {
  	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool skip);
  	void (*timedout_job)(struct amd_sched_job *sched_job);
  	void (*free_job)(struct amd_sched_job *sched_job);
  };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 79a7f228-3d63-e043-58c1-2986379b6296-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author
#9 lore raw

RE: [PATCH 2/9] amd/scheduler:imple job skip feature

Christian

Looks something is misunderstanding in this patch:

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job) · line 201

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +4 -2
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");

Not really needed, we could just use amd_sched_job_fake_signal() here as well.

[ml] the @skip parameter is needed, and we cannot use "fake signal", because "fake signal" may already called in entity_pop_job() stage We skip job at two place, one is in entity_pop_job() if scheduler can detect the job is guilty in that time Another one is in run_job() by this @skip in job_recovery(),

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 528

Quoted 6 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
@@ -510,9 +524,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool found_guilty = false, skip;
+		uint64_t guilty_context;

Well if you want the variables to have an effect you should probably move them outside the loop. Otherwise the code doesn't make much sense to me.

[ML] yeah, that's an error, will fix it !

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 342

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +9 -0
+static void amd_sched_job_fake_signal(struct amd_sched_job *job) {
+	struct amd_sched_fence *s_fence = job->s_fence;
+
+	dma_fence_set_error(&job->s_fence->finished, -ECANCELED);
+	/* fake signaling the scheduled fence */
+	amd_sched_fence_scheduled(s_fence);
+	/* fake signaling the finished fence */
+	dma_fence_put(&job->s_fence->finished);
+	job->sched->ops->free_job(job);

Well signaling the finished fence will free the job as well, won't it?

[ML] no, fake signal is invoked before begin_job(), so the job_finish_cb callback haven't been hooked in "finished" fence, that's why I manually call "free_job()", but keep in mind that WAIT_CS may already called so we still rely on signaling the "finished" fence to wakeup The WAIT_CS,

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 367

Quoted 5 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -0
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		sched_job = NULL;
+	}
+

This must come after the spsc_queue_pop() below, otherwise you mess up the spsc queue.

[ML] yes, I also noticed that error, will fix the sequence !

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:06
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 2/9] amd/scheduler:imple job skip feature

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 10 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

with this feature we can introduce new gpu recover feature.

Sorry for the delay, totally swamped once more at the moment.

Quoted 27 lines
Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++---
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 46 +++++++++++++++++++--------
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
  3 files changed, 39 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +2 -1
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
  	return fence;
  }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
 *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
+*sched_job, bool skip)
  {
  	struct dma_fence *fence = NULL;
  	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
  	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));

drivers/gpu/drm/amd/amdgpu/amdgpu_job.c static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job) · line 201

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +4 -3
 
  	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");

Not really needed, we could just use amd_sched_job_fake_signal() here as well.

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +0 -0
  	} else {
  		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
  				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
index 9cbeade..29841ba 100644
@@ -330,6 +330,20 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
  	return false;
  }

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 341

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +9 -0
 
+static void amd_sched_job_fake_signal(struct amd_sched_job *job) {
+	struct amd_sched_fence *s_fence = job->s_fence;
+
+	dma_fence_set_error(&job->s_fence->finished, -ECANCELED);
+	/* fake signaling the scheduled fence */
+	amd_sched_fence_scheduled(s_fence);
+	/* fake signaling the finished fence */
+	dma_fence_put(&job->s_fence->finished);
+	job->sched->ops->free_job(job);

Well signaling the finished fence will free the job as well, won't it?

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+	/* call CB on the s_fence, e.g. wake up WAIT_CS */
+	amd_sched_fence_finished(s_fence);
+}
+
  static struct amd_sched_job *
  amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 367

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
@@ -345,6 +359,12 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  			return NULL;
 
  	sched_job->s_entity = NULL;
+
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		sched_job = NULL;
+	}
+

This must come after the spsc_queue_pop() below, otherwise you mess up the spsc queue.

Quoted 37 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -7
  	spsc_queue_pop(&entity->job_queue);
  	return sched_job;
  }
@@ -441,13 +461,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
  	job->sched->ops->timedout_job(job);
  }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job) -{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
  void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
  {
  	struct amd_sched_job *s_job;
@@ -466,7 +479,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  	}
  	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit) 
+{
  		bool found = false;
 
  		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ ) 

{ @@ -476,7 +489,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
if (bad->s_fence->scheduled.context == entity->fence_context) {
found = true;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -1
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
  					break;
  				}
  			}

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 528

Quoted 5 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
@@ -510,9 +524,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool found_guilty = false, skip;
+		uint64_t guilty_context;

Well if you want the variables to have an effect you should probably move them outside the loop. Otherwise the code doesn't make much sense to me.

Regards,
Christian.

Quoted 45 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +8 -1
+
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == 
+guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);
  		atomic_inc(&sched->hw_rq_count);
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence); @@ -525,7 +547,6 @@ void 

amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
r);
dma_fence_put(fence);
} else {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -3
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
  		spin_lock(&sched->job_list_lock);
@@ -650,7 +671,7 @@ static int amd_sched_main(void *param)
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
  		amd_sched_fence_scheduled(s_fence);
 
  		if (fence) {
@@ -664,7 +685,6 @@ static int amd_sched_main(void *param)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
 
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr · line 132

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +2 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
  */
  struct amd_sched_backend_ops {
  	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool 
+skip);
  	void (*timedout_job)(struct amd_sched_job *sched_job);
  	void (*free_job)(struct amd_sched_job *sched_job);
  };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#10 lore raw

PATCH 3/9 drm/amdgpu:implement new GPU recover(v2)

In reply to a message not in the archive

1,new imple names amdgpu_gpu_recover which gives more hint on what it does compared with gpu_reset

2,gpu_recover unify bare-metal and SR-IOV, only the asic reset part is implemented differently

3,gpu_recover will increase hang job karma and mark its entity/context as guilty if exceeds limit

V2:

4,in scheduler main routine the job from guilty context will be immedialy fake signaled after it poped from queue and its fence be set with "-ECANCELED" error

5,in scheduler recovery routine all jobs from the guilty entity would be dropped

6,in run_job() routine the real IB submission would be skipped if @skip parameter equales true or there was VRAM lost occured.

Change-Id I30d30924eeb512e8f3243a8af727d061ed41f800
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   4 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 262 +++++++++++++++++++++++++++++
 2 files changed, 266 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +4 -0
index ba1ab97..003668f 100644
@@ -178,6 +178,10 @@ extern int amdgpu_cik_support;
 #define CIK_CURSOR_WIDTH 128
 #define CIK_CURSOR_HEIGHT 128
 
+/* GPU RESET flags */
+#define AMDGPU_RESET_INFO_VRAM_LOST  (1 << 0)
+#define AMDGPU_RESET_INFO_FULLRESET  (1 << 1)
+
 struct amdgpu_device;
 struct amdgpu_ib;
 struct amdgpu_cs_parser;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +262 -0

268 lines not on this page · raw message

Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-4-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#12 lore raw

Re: [PATCH 3/9] drm/amdgpu:implement new GPU recover(v2)

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 48 lines

1,new imple names amdgpu_gpu_recover which gives more hint on what it does compared with gpu_reset

2,gpu_recover unify bare-metal and SR-IOV, only the asic reset part is implemented differently

3,gpu_recover will increase hang job karma and mark its entity/context as guilty if exceeds limit

V2:

4,in scheduler main routine the job from guilty context will be immedialy fake signaled after it poped from queue and its fence be set with "-ECANCELED" error

5,in scheduler recovery routine all jobs from the guilty entity would be dropped

6,in run_job() routine the real IB submission would be skipped if @skip parameter equales true or there was VRAM lost occured.

Change-Id I30d30924eeb512e8f3243a8af727d061ed41f800
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   4 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 262 +++++++++++++++++++++++++++++
  2 files changed, 266 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +4 -0
index ba1ab97..003668f 100644
@@ -178,6 +178,10 @@ extern int amdgpu_cik_support;
  #define CIK_CURSOR_WIDTH 128
  #define CIK_CURSOR_HEIGHT 128
 
+/* GPU RESET flags */
+#define AMDGPU_RESET_INFO_VRAM_LOST  (1 << 0)
+#define AMDGPU_RESET_INFO_FULLRESET  (1 << 1)
+
  struct amdgpu_device;
  struct amdgpu_ib;
  struct amdgpu_cs_parser;
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
index a07544d..0db3b3c 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_gpu_reset(struct amdgpu_device *adev) · line 3122

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
@@ -3116,6 +3116,268 @@ int amdgpu_gpu_reset(struct amdgpu_device *adev)
  	return r;
  }
 
+static int amdgpu_reset(struct amdgpu_device *adev, uint64_t* reset_flags)
+{
+	int r;
+	bool need_full_reset, vram_lost = 0;

In general reverse tree order please. E.g. should look like this:

Quoted 45 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +40 -0
 bool need_full_reset, vram_lost = 0;
 int r;
 
 
 
+
+	need_full_reset = amdgpu_need_full_reset(adev);
+
+	if (!need_full_reset) {
+		amdgpu_pre_soft_reset(adev);
+		r = amdgpu_soft_reset(adev);
+		amdgpu_post_soft_reset(adev);
+		if (r || amdgpu_check_soft_reset(adev)) {
+			DRM_INFO("soft reset failed, will fallback to full reset!\n");
+			need_full_reset = true;
+		}
+
+	}
+
+	if (need_full_reset) {
+		r = amdgpu_suspend(adev);
+
+retry:
+		amdgpu_atombios_scratch_regs_save(adev);
+		r = amdgpu_asic_reset(adev);
+		amdgpu_atombios_scratch_regs_restore(adev);
+		/* post card */
+		amdgpu_atom_asic_init(adev->mode_info.atom_context);
+
+		if (!r) {
+			dev_info(adev->dev, "GPU reset succeeded, trying to resume\n");
+			r = amdgpu_resume_phase1(adev);
+			if (r)
+				goto out;
+
+			vram_lost = amdgpu_check_vram_lost(adev);
+			if (vram_lost) {
+				DRM_ERROR("VRAM is lost!\n");
+				atomic_inc(&adev->vram_lost_counter);
+			}
+
+			r = amdgpu_ttm_recover_gart(adev);
+			if (r)
+				goto out;
+

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_gpu_reset(struct amdgpu_device *adev) · line 3172

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +10 -0
+			r = amdgpu_resume_phase2(adev);
+			if (r)
+				goto out;
+
+			if (vram_lost)
+				amdgpu_fill_reset_magic(adev);
+		}
+	}
+
+out:

Would probably be better to move the full reset into a separate function instead of the retry/out gotos here.

Apart from that it looks like that should work.

Regards,
Christian.

Quoted 201 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +201 -0
+	if (!r) {
+		amdgpu_irq_gpu_reset_resume_helper(adev);
+		r = amdgpu_ib_ring_tests(adev);
+		if (r) {
+			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
+			r = amdgpu_suspend(adev);
+			need_full_reset = true;
+			goto retry;
+		}
+	}
+
+	if (reset_flags) {
+		if (vram_lost)
+			(*reset_flags) |= AMDGPU_RESET_INFO_VRAM_LOST;
+
+		if (need_full_reset)
+			(*reset_flags) |= AMDGPU_RESET_INFO_FULLRESET;
+	}
+
+	return r;
+}
+
+static int amdgpu_reset_sriov(struct amdgpu_device *adev, uint64_t *reset_flags, bool from_hypervisor)
+{
+	int r;
+
+	if (from_hypervisor)
+		r = amdgpu_virt_request_full_gpu(adev, true);
+	else
+		r = amdgpu_virt_reset_gpu(adev);
+	if (r)
+		return r;
+
+	/* Resume IP prior to SMC */
+	r = amdgpu_sriov_reinit_early(adev);
+	if (r)
+		goto error;
+
+	/* we need recover gart prior to run SMC/CP/SDMA resume */
+	amdgpu_ttm_recover_gart(adev);
+
+	/* now we are okay to resume SMC/CP/SDMA */
+	r = amdgpu_sriov_reinit_late(adev);
+	if (r)
+		goto error;
+
+	amdgpu_irq_gpu_reset_resume_helper(adev);
+	r = amdgpu_ib_ring_tests(adev);
+	if (r)
+		dev_err(adev->dev, "[GPU_RESET] ib ring test failed (%d).\n", r);
+
+error:
+	/* release full control of GPU after ib test */
+	amdgpu_virt_release_full_gpu(adev, true);
+
+	if (reset_flags) {
+		/* will get vram_lost from GIM in future, now all
+		 * reset request considered VRAM LOST
+		 */
+		(*reset_flags) |= ~AMDGPU_RESET_INFO_VRAM_LOST;
+		atomic_inc(&adev->vram_lost_counter);
+
+		/* VF FLR or hotlink reset is always full-reset */
+		(*reset_flags) |= AMDGPU_RESET_INFO_FULLRESET;
+	}
+
+	return r;
+}
+
+/**
+ * amdgpu_gpu_recover - reset the asic and recover scheduler
+ *
+ * @adev: amdgpu device pointer
+ * @job: which job trigger hang
+ *
+ * Attempt to reset the GPU if it has hung (all asics).
+ * Returns 0 for success or an error on failure.
+ */
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
+{
+	struct drm_atomic_state *state = NULL;
+	uint64_t reset_flags = 0;
+	int i, r, resched;
+
+	if (!amdgpu_check_soft_reset(adev)) {
+		DRM_INFO("No hardware hang detected. Did some blocks stall?\n");
+		return 0;
+	}
+
+	dev_info(adev->dev, "GPU reset begin!\n");
+
+	mutex_lock(&adev->virt.lock_reset);
+	atomic_inc(&adev->gpu_reset_counter);
+	adev->in_sriov_reset = 1;
+
+	/* block TTM */
+	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
+	/* store modesetting */
+	if (amdgpu_device_has_dc_support(adev))
+		state = drm_atomic_helper_suspend(adev->ddev);
+
+	/* block scheduler */
+	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
+		struct amdgpu_ring *ring = adev->rings[i];
+
+		if (!ring || !ring->sched.thread)
+			continue;
+
+		/* only focus on the ring hit timeout if &job not NULL */
+		if (job && job->ring->idx != i)
+			continue;
+
+		kthread_park(ring->sched.thread);
+		amd_sched_hw_job_reset(&ring->sched, &job->base);
+
+		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
+		amdgpu_fence_driver_force_completion(ring);
+	}
+
+	if (amdgpu_sriov_vf(adev))
+		r = amdgpu_reset_sriov(adev, &reset_flags, job ? false : true);
+	else
+		r = amdgpu_reset(adev, &reset_flags);
+
+	if (!r) {
+		if (((reset_flags & AMDGPU_RESET_INFO_FULLRESET) && !(adev->flags & AMD_IS_APU)) ||
+			(reset_flags & AMDGPU_RESET_INFO_VRAM_LOST)) {
+			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+			struct amdgpu_bo *bo, *tmp;
+			struct dma_fence *fence = NULL, *next = NULL;
+
+			DRM_INFO("recover vram bo from shadow\n");
+			mutex_lock(&adev->shadow_list_lock);
+			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
+				next = NULL;
+				amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
+				if (fence) {
+					r = dma_fence_wait(fence, false);
+					if (r) {
+						WARN(r, "recovery from shadow isn't completed\n");
+						break;
+					}
+				}
+
+				dma_fence_put(fence);
+				fence = next;
+			}
+			mutex_unlock(&adev->shadow_list_lock);
+			if (fence) {
+				r = dma_fence_wait(fence, false);
+				if (r)
+					WARN(r, "recovery from shadow isn't completed\n");
+			}
+			dma_fence_put(fence);
+		}
+
+		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
+			struct amdgpu_ring *ring = adev->rings[i];
+
+			if (!ring || !ring->sched.thread)
+				continue;
+
+			/* only focus on the ring hit timeout if &job not NULL */
+			if (job && job->ring->idx != i)
+				continue;
+
+			amd_sched_job_recovery(&ring->sched);
+			kthread_unpark(ring->sched.thread);
+		}
+	} else {
+		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
+			struct amdgpu_ring *ring = adev->rings[i];
+
+			if (!ring || !ring->sched.thread)
+				continue;
+
+			/* only focus on the ring hit timeout if &job not NULL */
+			if (job && job->ring->idx != i)
+				continue;
+
+			kthread_unpark(adev->rings[i]->sched.thread);
+		}
+	}
+
+	if (amdgpu_device_has_dc_support(adev)) {
+		if (drm_atomic_helper_resume(adev->ddev, state))
+			dev_info(adev->dev, "drm resume failed:%d\n", r);
+		amdgpu_dm_display_resume(adev);
+	} else {
+		drm_helper_resume_force_mode(adev->ddev);
+	}
+
+	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
+
+	if (r) {
+		/* bad news, how to tell it to userspace ? */
+		dev_info(adev->dev, "GPU reset(%d) failed\n", atomic_read(&adev->gpu_reset_counter));
+		amdgpu_vf_error_put(adev, AMDGIM_ERROR_VF_GPU_RESET_FAIL, 0, r);
+	} else {
+		dev_info(adev->dev, "GPU reset(%d) successed!\n",atomic_read(&adev->gpu_reset_counter));
+	}

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c int amdgpu_gpu_reset(struct amdgpu_device *adev) · line 3383

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +7 -0
+
+	amdgpu_vf_error_trans_all(adev);
+	adev->in_sriov_reset = 0;
+	mutex_unlock(&adev->virt.lock_reset);
+	return r;
+}
+
  void amdgpu_get_pcie_info(struct amdgpu_device *adev)
  {
  	u32 mask;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#13 lore raw

PATCH 4/9 drm/amdgpu:replace deprecated gpu reset

In reply to a message not in the archive

now use new gpu recover

Change-Id Ieccd25772c47c0e710ad81537a3dd0c1767585a1
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 298 -----------------------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  |  10 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |   1 -
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |   2 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c      |   2 +-
 8 files changed, 10 insertions(+), 312 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index 003668f..335df11 100644
@@ -1844,7 +1844,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
 #define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
 
 /* Common functions */
-int amdgpu_gpu_reset(struct amdgpu_device *adev);
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job* job);
 bool amdgpu_need_backup(struct amdgpu_device *adev);
 void amdgpu_pci_config_reset(struct amdgpu_device *adev);
 bool amdgpu_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -298

304 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +5 -5
index d7374cf..9d67bcb 100644
@@ -694,25 +694,25 @@ static int amdgpu_debugfs_fence_info(struct seq_file *m, void *data)
 }
 
 /**
- * amdgpu_debugfs_gpu_reset - manually trigger a gpu reset
+ * amdgpu_debugfs_gpu_recover - manually trigger a gpu reset & recover
  *
  * Manually trigger a gpu reset at the next fence wait.
  */
-static int amdgpu_debugfs_gpu_reset(struct seq_file *m, void *data)
+static int amdgpu_debugfs_gpu_recover(struct seq_file *m, void *data)
 {
 	struct drm_info_node *node = (struct drm_info_node *) m->private;
 	struct drm_device *dev = node->minor->dev;
 	struct amdgpu_device *adev = dev->dev_private;
 
-	seq_printf(m, "gpu reset\n");
-	amdgpu_gpu_reset(adev);
+	seq_printf(m, "gpu recover\n");
+	amdgpu_gpu_recover(adev, NULL);
 
 	return 0;
 }
 
 static const struct drm_info_list amdgpu_debugfs_fence_list[] = {
 	{"amdgpu_fence_info", &amdgpu_debugfs_fence_info, 0, NULL},
-	{"amdgpu_gpu_reset", &amdgpu_debugfs_gpu_reset, 0, NULL}
+	{"amdgpu_gpu_recover", &amdgpu_debugfs_gpu_recover, 0, NULL}
 };
 
 static const struct drm_info_list amdgpu_debugfs_fence_list_sriov[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -1
index 47c5ce9..9c2f87c 100644
@@ -88,7 +88,7 @@ static void amdgpu_irq_reset_work_func(struct work_struct *work)
 						  reset_work);
 
 	if (!amdgpu_sriov_vf(adev))
-		amdgpu_gpu_reset(adev);
+		amdgpu_gpu_recover(adev, NULL);
 }
 
 /* Disable *all* interrupts */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -4
index f08fde9..ac6a4f3 100644
@@ -37,10 +37,7 @@ static void amdgpu_job_timedout(struct amd_sched_job *s_job)
 		  atomic_read(&job->ring->fence_drv.last_seq),
 		  job->ring->fence_drv.sync_seq);
 
-	if (amdgpu_sriov_vf(job->adev))
-		amdgpu_sriov_gpu_reset(job->adev, job);
-	else
-		amdgpu_gpu_reset(job->adev);
+	amdgpu_gpu_recover(job->adev, job);
 }
 
 int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index b89d37f..3a661aa 100644
@@ -285,7 +285,6 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
 int amdgpu_virt_request_full_gpu(struct amdgpu_device *adev, bool init);
 int amdgpu_virt_release_full_gpu(struct amdgpu_device *adev, bool init);
 int amdgpu_virt_reset_gpu(struct amdgpu_device *adev);
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job);
 int amdgpu_virt_alloc_mm_table(struct amdgpu_device *adev);
 void amdgpu_virt_free_mm_table(struct amdgpu_device *adev);
 int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +1 -1
index b4906d2..f8522a0 100644
@@ -254,7 +254,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work)
 	}
 
 	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
 }
 
 static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +1 -1
index c25a831..dae6d3a 100644
@@ -514,7 +514,7 @@ static void xgpu_vi_mailbox_flr_work(struct work_struct *work)
 	}
 
 	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
 }
 
 static int xgpu_vi_set_mailbox_rcv_irq(struct amdgpu_device *adev,
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-5-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#15 lore raw

Re: [PATCH 4/9] drm/amdgpu:replace deprecated gpu reset

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 1 line

now use new gpu recover

Might be better to squash together with the previous patch.

This one doesn't introduce new functionality, but only removes the old code and switches over to the new one.

Quoted 430 lines
Change-Id Ieccd25772c47c0e710ad81537a3dd0c1767585a1
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 298 -----------------------------
  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  |  10 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   5 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |   1 -
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |   2 +-
  drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c      |   2 +-
  8 files changed, 10 insertions(+), 312 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index 003668f..335df11 100644
@@ -1844,7 +1844,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
  #define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
 
  /* Common functions */
-int amdgpu_gpu_reset(struct amdgpu_device *adev);
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job* job);
  bool amdgpu_need_backup(struct amdgpu_device *adev);
  void amdgpu_pci_config_reset(struct amdgpu_device *adev);
  bool amdgpu_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -298
index 0db3b3c..a2f9a7f 100644
@@ -2818,304 +2818,6 @@ static int amdgpu_recover_vram_from_shadow(struct amdgpu_device *adev,
  	return r;
  }
 
-/**
- * amdgpu_sriov_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- * @job: which job trigger hang
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * for SRIOV case.
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job)
-{
-	int i, j, r = 0;
-	int resched;
-	struct amdgpu_bo *bo, *tmp;
-	struct amdgpu_ring *ring;
-	struct dma_fence *fence = NULL, *next = NULL;
-
-	mutex_lock(&adev->virt.lock_reset);
-	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = true;
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-
-	/* we start from the ring trigger GPU hang */
-	j = job ? job->ring->idx : 0;
-
-	/* block scheduler */
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		kthread_park(ring->sched.thread);
-
-		if (job && j != i)
-			continue;
-
-		/* here give the last chance to check if job removed from mirror-list
-		 * since we already pay some time on kthread_park */
-		if (job && list_empty(&job->base.node)) {
-			kthread_unpark(ring->sched.thread);
-			goto give_up_reset;
-		}
-
-		if (amd_sched_invalidate_job(&job->base, amdgpu_job_hang_limit))
-			amd_sched_job_kickout(&job->base);
-
-		/* only do job_reset on the hang ring if @job not NULL */
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	/* request to take full control of GPU before re-initialization  */
-	if (job)
-		amdgpu_virt_reset_gpu(adev);
-	else
-		amdgpu_virt_request_full_gpu(adev, true);
-
-
-	/* Resume IP prior to SMC */
-	amdgpu_sriov_reinit_early(adev);
-
-	/* we need recover gart prior to run SMC/CP/SDMA resume */
-	amdgpu_ttm_recover_gart(adev);
-
-	/* now we are okay to resume SMC/CP/SDMA */
-	amdgpu_sriov_reinit_late(adev);
-
-	amdgpu_irq_gpu_reset_resume_helper(adev);
-
-	if (amdgpu_ib_ring_tests(adev))
-		dev_err(adev->dev, "[GPU_RESET] ib ring test failed (%d).\n", r);
-
-	/* release full control of GPU after ib test */
-	amdgpu_virt_release_full_gpu(adev, true);
-
-	DRM_INFO("recover vram bo from shadow\n");
-
-	ring = adev->mman.buffer_funcs_ring;
-	mutex_lock(&adev->shadow_list_lock);
-	list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-		next = NULL;
-		amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-		if (fence) {
-			r = dma_fence_wait(fence, false);
-			if (r) {
-				WARN(r, "recovery from shadow isn't completed\n");
-				break;
-			}
-		}
-
-		dma_fence_put(fence);
-		fence = next;
-	}
-	mutex_unlock(&adev->shadow_list_lock);
-
-	if (fence) {
-		r = dma_fence_wait(fence, false);
-		if (r)
-			WARN(r, "recovery from shadow isn't completed\n");
-	}
-	dma_fence_put(fence);
-
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		if (job && j != i) {
-			kthread_unpark(ring->sched.thread);
-			continue;
-		}
-
-		amd_sched_job_recovery(&ring->sched);
-		kthread_unpark(ring->sched.thread);
-	}
-
-	drm_helper_resume_force_mode(adev->ddev);
-give_up_reset:
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	} else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	adev->in_sriov_reset = false;
-	mutex_unlock(&adev->virt.lock_reset);
-	return r;
-}
-
-/**
- * amdgpu_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_gpu_reset(struct amdgpu_device *adev)
-{
-	struct drm_atomic_state *state = NULL;
-	int i, r;
-	int resched;
-	bool need_full_reset, vram_lost = false;
-
-	if (!amdgpu_check_soft_reset(adev)) {
-		DRM_INFO("No hardware hang detected. Did some blocks stall?\n");
-		return 0;
-	}
-
-	atomic_inc(&adev->gpu_reset_counter);
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-	/* store modesetting */
-	if (amdgpu_device_has_dc_support(adev))
-		state = drm_atomic_helper_suspend(adev->ddev);
-
-	/* block scheduler */
-	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-		struct amdgpu_ring *ring = adev->rings[i];
-
-		if (!ring || !ring->sched.thread)
-			continue;
-		kthread_park(ring->sched.thread);
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	need_full_reset = amdgpu_need_full_reset(adev);
-
-	if (!need_full_reset) {
-		amdgpu_pre_soft_reset(adev);
-		r = amdgpu_soft_reset(adev);
-		amdgpu_post_soft_reset(adev);
-		if (r || amdgpu_check_soft_reset(adev)) {
-			DRM_INFO("soft reset failed, will fallback to full reset!\n");
-			need_full_reset = true;
-		}
-	}
-
-	if (need_full_reset) {
-		r = amdgpu_suspend(adev);
-
-retry:
-		amdgpu_atombios_scratch_regs_save(adev);
-		r = amdgpu_asic_reset(adev);
-		amdgpu_atombios_scratch_regs_restore(adev);
-		/* post card */
-		amdgpu_atom_asic_init(adev->mode_info.atom_context);
-
-		if (!r) {
-			dev_info(adev->dev, "GPU reset succeeded, trying to resume\n");
-			r = amdgpu_resume_phase1(adev);
-			if (r)
-				goto out;
-			vram_lost = amdgpu_check_vram_lost(adev);
-			if (vram_lost) {
-				DRM_ERROR("VRAM is lost!\n");
-				atomic_inc(&adev->vram_lost_counter);
-			}
-			r = amdgpu_ttm_recover_gart(adev);
-			if (r)
-				goto out;
-			r = amdgpu_resume_phase2(adev);
-			if (r)
-				goto out;
-			if (vram_lost)
-				amdgpu_fill_reset_magic(adev);
-		}
-	}
-out:
-	if (!r) {
-		amdgpu_irq_gpu_reset_resume_helper(adev);
-		r = amdgpu_ib_ring_tests(adev);
-		if (r) {
-			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
-			r = amdgpu_suspend(adev);
-			need_full_reset = true;
-			goto retry;
-		}
-		/**
-		 * recovery vm page tables, since we cannot depend on VRAM is
-		 * consistent after gpu full reset.
-		 */
-		if (need_full_reset && amdgpu_need_backup(adev)) {
-			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
-			struct amdgpu_bo *bo, *tmp;
-			struct dma_fence *fence = NULL, *next = NULL;
-
-			DRM_INFO("recover vram bo from shadow\n");
-			mutex_lock(&adev->shadow_list_lock);
-			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-				next = NULL;
-				amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-				if (fence) {
-					r = dma_fence_wait(fence, false);
-					if (r) {
-						WARN(r, "recovery from shadow isn't completed\n");
-						break;
-					}
-				}
-
-				dma_fence_put(fence);
-				fence = next;
-			}
-			mutex_unlock(&adev->shadow_list_lock);
-			if (fence) {
-				r = dma_fence_wait(fence, false);
-				if (r)
-					WARN(r, "recovery from shadow isn't completed\n");
-			}
-			dma_fence_put(fence);
-		}
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			struct amdgpu_ring *ring = adev->rings[i];
-
-			if (!ring || !ring->sched.thread)
-				continue;
-
-			amd_sched_job_recovery(&ring->sched);
-			kthread_unpark(ring->sched.thread);
-		}
-	} else {
-		dev_err(adev->dev, "asic resume failed (%d).\n", r);
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			if (adev->rings[i] && adev->rings[i]->sched.thread) {
-				kthread_unpark(adev->rings[i]->sched.thread);
-			}
-		}
-	}
-
-	if (amdgpu_device_has_dc_support(adev)) {
-		r = drm_atomic_helper_resume(adev->ddev, state);
-		amdgpu_dm_display_resume(adev);
-	} else
-		drm_helper_resume_force_mode(adev->ddev);
-
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	}
-	else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	amdgpu_vf_error_trans_all(adev);
-	return r;
-}
-
  static int amdgpu_reset(struct amdgpu_device *adev, uint64_t* reset_flags)
  {
  	int r;
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +5 -5
index d7374cf..9d67bcb 100644
@@ -694,25 +694,25 @@ static int amdgpu_debugfs_fence_info(struct seq_file *m, void *data)
  }
 
  /**
- * amdgpu_debugfs_gpu_reset - manually trigger a gpu reset
+ * amdgpu_debugfs_gpu_recover - manually trigger a gpu reset & recover
   *
   * Manually trigger a gpu reset at the next fence wait.
   */
-static int amdgpu_debugfs_gpu_reset(struct seq_file *m, void *data)
+static int amdgpu_debugfs_gpu_recover(struct seq_file *m, void *data)
  {
  	struct drm_info_node *node = (struct drm_info_node *) m->private;
  	struct drm_device *dev = node->minor->dev;
  	struct amdgpu_device *adev = dev->dev_private;
 
-	seq_printf(m, "gpu reset\n");
-	amdgpu_gpu_reset(adev);
+	seq_printf(m, "gpu recover\n");
+	amdgpu_gpu_recover(adev, NULL);
 
  	return 0;
  }
 
  static const struct drm_info_list amdgpu_debugfs_fence_list[] = {
  	{"amdgpu_fence_info", &amdgpu_debugfs_fence_info, 0, NULL},
-	{"amdgpu_gpu_reset", &amdgpu_debugfs_gpu_reset, 0, NULL}
+	{"amdgpu_gpu_recover", &amdgpu_debugfs_gpu_recover, 0, NULL}
  };
 
  static const struct drm_info_list amdgpu_debugfs_fence_list_sriov[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -1
index 47c5ce9..9c2f87c 100644
@@ -88,7 +88,7 @@ static void amdgpu_irq_reset_work_func(struct work_struct *work)
  						  reset_work);
 
  	if (!amdgpu_sriov_vf(adev))
-		amdgpu_gpu_reset(adev);
+		amdgpu_gpu_recover(adev, NULL);
  }
 
  /* Disable *all* interrupts */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -4
index f08fde9..ac6a4f3 100644
@@ -37,10 +37,7 @@ static void amdgpu_job_timedout(struct amd_sched_job *s_job)
  		  atomic_read(&job->ring->fence_drv.last_seq),
  		  job->ring->fence_drv.sync_seq);
 
-	if (amdgpu_sriov_vf(job->adev))
-		amdgpu_sriov_gpu_reset(job->adev, job);
-	else
-		amdgpu_gpu_reset(job->adev);
+	amdgpu_gpu_recover(job->adev, job);
  }
 
  int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index b89d37f..3a661aa 100644
@@ -285,7 +285,6 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
  int amdgpu_virt_request_full_gpu(struct amdgpu_device *adev, bool init);
  int amdgpu_virt_release_full_gpu(struct amdgpu_device *adev, bool init);
  int amdgpu_virt_reset_gpu(struct amdgpu_device *adev);
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job);
  int amdgpu_virt_alloc_mm_table(struct amdgpu_device *adev);
  void amdgpu_virt_free_mm_table(struct amdgpu_device *adev);
  int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +1 -1
index b4906d2..f8522a0 100644
@@ -254,7 +254,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work)
  	}
 
  	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
  }
 
  static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +0 -0
index c25a831..dae6d3a 100644

drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c static void xgpu_vi_mailbox_flr_work(struct work_struct *work) · line 520

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +1 -1
@@ -514,7 +514,7 @@ static void xgpu_vi_mailbox_flr_work(struct work_struct *work)
  	}
 
  	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
  }
 
  static int xgpu_vi_set_mailbox_rcv_irq(struct amdgpu_device *adev,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 13ad16b1-6357-1f0e-3cdd-ddc42d4462b0-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author
#17 lore raw

RE: [PATCH 4/9] drm/amdgpu:replace deprecated gpu reset

Yeah I know I can squash them,

I spit the patch due to convenience for reviewing

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:13
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 4/9] drm/amdgpu:replace deprecated gpu reset

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 1 line

now use new gpu recover

Might be better to squash together with the previous patch.

This one doesn't introduce new functionality, but only removes the old code and switches over to the new one.

Quoted 430 lines
Change-Id Ieccd25772c47c0e710ad81537a3dd0c1767585a1
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 298 -----------------------------
  drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  |  10 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   5 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |   1 -
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |   2 +-
  drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c      |   2 +-
  8 files changed, 10 insertions(+), 312 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index 003668f..335df11 100644
@@ -1844,7 +1844,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
  #define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
 
  /* Common functions */
-int amdgpu_gpu_reset(struct amdgpu_device *adev);
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job* job);
  bool amdgpu_need_backup(struct amdgpu_device *adev);
  void amdgpu_pci_config_reset(struct amdgpu_device *adev);
  bool amdgpu_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -298
index 0db3b3c..a2f9a7f 100644
@@ -2818,304 +2818,6 @@ static int amdgpu_recover_vram_from_shadow(struct amdgpu_device *adev,
  	return r;
  }
 
-/**
- * amdgpu_sriov_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- * @job: which job trigger hang
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * for SRIOV case.
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job)
-{
-	int i, j, r = 0;
-	int resched;
-	struct amdgpu_bo *bo, *tmp;
-	struct amdgpu_ring *ring;
-	struct dma_fence *fence = NULL, *next = NULL;
-
-	mutex_lock(&adev->virt.lock_reset);
-	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = true;
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-
-	/* we start from the ring trigger GPU hang */
-	j = job ? job->ring->idx : 0;
-
-	/* block scheduler */
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		kthread_park(ring->sched.thread);
-
-		if (job && j != i)
-			continue;
-
-		/* here give the last chance to check if job removed from mirror-list
-		 * since we already pay some time on kthread_park */
-		if (job && list_empty(&job->base.node)) {
-			kthread_unpark(ring->sched.thread);
-			goto give_up_reset;
-		}
-
-		if (amd_sched_invalidate_job(&job->base, amdgpu_job_hang_limit))
-			amd_sched_job_kickout(&job->base);
-
-		/* only do job_reset on the hang ring if @job not NULL */
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	/* request to take full control of GPU before re-initialization  */
-	if (job)
-		amdgpu_virt_reset_gpu(adev);
-	else
-		amdgpu_virt_request_full_gpu(adev, true);
-
-
-	/* Resume IP prior to SMC */
-	amdgpu_sriov_reinit_early(adev);
-
-	/* we need recover gart prior to run SMC/CP/SDMA resume */
-	amdgpu_ttm_recover_gart(adev);
-
-	/* now we are okay to resume SMC/CP/SDMA */
-	amdgpu_sriov_reinit_late(adev);
-
-	amdgpu_irq_gpu_reset_resume_helper(adev);
-
-	if (amdgpu_ib_ring_tests(adev))
-		dev_err(adev->dev, "[GPU_RESET] ib ring test failed (%d).\n", r);
-
-	/* release full control of GPU after ib test */
-	amdgpu_virt_release_full_gpu(adev, true);
-
-	DRM_INFO("recover vram bo from shadow\n");
-
-	ring = adev->mman.buffer_funcs_ring;
-	mutex_lock(&adev->shadow_list_lock);
-	list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-		next = NULL;
-		amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-		if (fence) {
-			r = dma_fence_wait(fence, false);
-			if (r) {
-				WARN(r, "recovery from shadow isn't completed\n");
-				break;
-			}
-		}
-
-		dma_fence_put(fence);
-		fence = next;
-	}
-	mutex_unlock(&adev->shadow_list_lock);
-
-	if (fence) {
-		r = dma_fence_wait(fence, false);
-		if (r)
-			WARN(r, "recovery from shadow isn't completed\n");
-	}
-	dma_fence_put(fence);
-
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		if (job && j != i) {
-			kthread_unpark(ring->sched.thread);
-			continue;
-		}
-
-		amd_sched_job_recovery(&ring->sched);
-		kthread_unpark(ring->sched.thread);
-	}
-
-	drm_helper_resume_force_mode(adev->ddev);
-give_up_reset:
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	} else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	adev->in_sriov_reset = false;
-	mutex_unlock(&adev->virt.lock_reset);
-	return r;
-}
-
-/**
- * amdgpu_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_gpu_reset(struct amdgpu_device *adev)
-{
-	struct drm_atomic_state *state = NULL;
-	int i, r;
-	int resched;
-	bool need_full_reset, vram_lost = false;
-
-	if (!amdgpu_check_soft_reset(adev)) {
-		DRM_INFO("No hardware hang detected. Did some blocks stall?\n");
-		return 0;
-	}
-
-	atomic_inc(&adev->gpu_reset_counter);
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-	/* store modesetting */
-	if (amdgpu_device_has_dc_support(adev))
-		state = drm_atomic_helper_suspend(adev->ddev);
-
-	/* block scheduler */
-	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-		struct amdgpu_ring *ring = adev->rings[i];
-
-		if (!ring || !ring->sched.thread)
-			continue;
-		kthread_park(ring->sched.thread);
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	need_full_reset = amdgpu_need_full_reset(adev);
-
-	if (!need_full_reset) {
-		amdgpu_pre_soft_reset(adev);
-		r = amdgpu_soft_reset(adev);
-		amdgpu_post_soft_reset(adev);
-		if (r || amdgpu_check_soft_reset(adev)) {
-			DRM_INFO("soft reset failed, will fallback to full reset!\n");
-			need_full_reset = true;
-		}
-	}
-
-	if (need_full_reset) {
-		r = amdgpu_suspend(adev);
-
-retry:
-		amdgpu_atombios_scratch_regs_save(adev);
-		r = amdgpu_asic_reset(adev);
-		amdgpu_atombios_scratch_regs_restore(adev);
-		/* post card */
-		amdgpu_atom_asic_init(adev->mode_info.atom_context);
-
-		if (!r) {
-			dev_info(adev->dev, "GPU reset succeeded, trying to resume\n");
-			r = amdgpu_resume_phase1(adev);
-			if (r)
-				goto out;
-			vram_lost = amdgpu_check_vram_lost(adev);
-			if (vram_lost) {
-				DRM_ERROR("VRAM is lost!\n");
-				atomic_inc(&adev->vram_lost_counter);
-			}
-			r = amdgpu_ttm_recover_gart(adev);
-			if (r)
-				goto out;
-			r = amdgpu_resume_phase2(adev);
-			if (r)
-				goto out;
-			if (vram_lost)
-				amdgpu_fill_reset_magic(adev);
-		}
-	}
-out:
-	if (!r) {
-		amdgpu_irq_gpu_reset_resume_helper(adev);
-		r = amdgpu_ib_ring_tests(adev);
-		if (r) {
-			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
-			r = amdgpu_suspend(adev);
-			need_full_reset = true;
-			goto retry;
-		}
-		/**
-		 * recovery vm page tables, since we cannot depend on VRAM is
-		 * consistent after gpu full reset.
-		 */
-		if (need_full_reset && amdgpu_need_backup(adev)) {
-			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
-			struct amdgpu_bo *bo, *tmp;
-			struct dma_fence *fence = NULL, *next = NULL;
-
-			DRM_INFO("recover vram bo from shadow\n");
-			mutex_lock(&adev->shadow_list_lock);
-			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-				next = NULL;
-				amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-				if (fence) {
-					r = dma_fence_wait(fence, false);
-					if (r) {
-						WARN(r, "recovery from shadow isn't completed\n");
-						break;
-					}
-				}
-
-				dma_fence_put(fence);
-				fence = next;
-			}
-			mutex_unlock(&adev->shadow_list_lock);
-			if (fence) {
-				r = dma_fence_wait(fence, false);
-				if (r)
-					WARN(r, "recovery from shadow isn't completed\n");
-			}
-			dma_fence_put(fence);
-		}
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			struct amdgpu_ring *ring = adev->rings[i];
-
-			if (!ring || !ring->sched.thread)
-				continue;
-
-			amd_sched_job_recovery(&ring->sched);
-			kthread_unpark(ring->sched.thread);
-		}
-	} else {
-		dev_err(adev->dev, "asic resume failed (%d).\n", r);
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			if (adev->rings[i] && adev->rings[i]->sched.thread) {
-				kthread_unpark(adev->rings[i]->sched.thread);
-			}
-		}
-	}
-
-	if (amdgpu_device_has_dc_support(adev)) {
-		r = drm_atomic_helper_resume(adev->ddev, state);
-		amdgpu_dm_display_resume(adev);
-	} else
-		drm_helper_resume_force_mode(adev->ddev);
-
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	}
-	else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	amdgpu_vf_error_trans_all(adev);
-	return r;
-}
-
  static int amdgpu_reset(struct amdgpu_device *adev, uint64_t* reset_flags)
  {
  	int r;
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +5 -5
index d7374cf..9d67bcb 100644
@@ -694,25 +694,25 @@ static int amdgpu_debugfs_fence_info(struct seq_file *m, void *data)
  }
 
  /**
- * amdgpu_debugfs_gpu_reset - manually trigger a gpu reset
+ * amdgpu_debugfs_gpu_recover - manually trigger a gpu reset & recover
   *
   * Manually trigger a gpu reset at the next fence wait.
   */
-static int amdgpu_debugfs_gpu_reset(struct seq_file *m, void *data)
+static int amdgpu_debugfs_gpu_recover(struct seq_file *m, void *data)
  {
  	struct drm_info_node *node = (struct drm_info_node *) m->private;
  	struct drm_device *dev = node->minor->dev;
  	struct amdgpu_device *adev = dev->dev_private;
 
-	seq_printf(m, "gpu reset\n");
-	amdgpu_gpu_reset(adev);
+	seq_printf(m, "gpu recover\n");
+	amdgpu_gpu_recover(adev, NULL);
 
  	return 0;
  }
 
  static const struct drm_info_list amdgpu_debugfs_fence_list[] = {
  	{"amdgpu_fence_info", &amdgpu_debugfs_fence_info, 0, NULL},
-	{"amdgpu_gpu_reset", &amdgpu_debugfs_gpu_reset, 0, NULL}
+	{"amdgpu_gpu_recover", &amdgpu_debugfs_gpu_recover, 0, NULL}
  };
 
  static const struct drm_info_list amdgpu_debugfs_fence_list_sriov[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -1
index 47c5ce9..9c2f87c 100644
@@ -88,7 +88,7 @@ static void amdgpu_irq_reset_work_func(struct work_struct *work)
  						  reset_work);
 
  	if (!amdgpu_sriov_vf(adev))
-		amdgpu_gpu_reset(adev);
+		amdgpu_gpu_recover(adev, NULL);
  }
 
  /* Disable *all* interrupts */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -4
index f08fde9..ac6a4f3 100644
@@ -37,10 +37,7 @@ static void amdgpu_job_timedout(struct amd_sched_job *s_job)
  		  atomic_read(&job->ring->fence_drv.last_seq),
  		  job->ring->fence_drv.sync_seq);
 
-	if (amdgpu_sriov_vf(job->adev))
-		amdgpu_sriov_gpu_reset(job->adev, job);
-	else
-		amdgpu_gpu_reset(job->adev);
+	amdgpu_gpu_recover(job->adev, job);
  }
 
  int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index b89d37f..3a661aa 100644
@@ -285,7 +285,6 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
  int amdgpu_virt_request_full_gpu(struct amdgpu_device *adev, bool init);
  int amdgpu_virt_release_full_gpu(struct amdgpu_device *adev, bool init);
  int amdgpu_virt_reset_gpu(struct amdgpu_device *adev);
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job);
  int amdgpu_virt_alloc_mm_table(struct amdgpu_device *adev);
  void amdgpu_virt_free_mm_table(struct amdgpu_device *adev);
  int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +1 -1
index b4906d2..f8522a0 100644
@@ -254,7 +254,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work)
  	}
 
  	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
  }
 
  static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +0 -0
index c25a831..dae6d3a 100644

drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c static void xgpu_vi_mailbox_flr_work(struct work_struct *work) · line 520

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +1 -1
@@ -514,7 +514,7 @@ static void xgpu_vi_mailbox_flr_work(struct work_struct *work)
  	}
 
  	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
  }
 
  static int xgpu_vi_set_mailbox_rcv_irq(struct amdgpu_device *adev,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB04499B9B8F278BE639D144A184450-7LeqcoF/[email protected]

#19 lore raw

Re: [PATCH 4/9] drm/amdgpu:replace deprecated gpu reset

Quoted 1 line

I spit the patch due to convenience for reviewing

Thought so, but this way we can't see the delta for reviewing.

Splitting up in multiple patches makes only sense if you can limit the patch to one component or functionality change at a time.

Christian.

Am 26.10.2017 um 10:13 schrieb Liu, Monk:

Quoted 449 lines

Yeah I know I can squash them,

I spit the patch due to convenience for reviewing

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:13
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 4/9] drm/amdgpu:replace deprecated gpu reset

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 1 line

now use new gpu recover

Might be better to squash together with the previous patch.

This one doesn't introduce new functionality, but only removes the old code and switches over to the new one.

Quoted 431 lines
Change-Id Ieccd25772c47c0e710ad81537a3dd0c1767585a1
---
   drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   2 +-
   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 298 -----------------------------
   drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  |  10 +-
   drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    |   2 +-
   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   5 +-
   drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |   1 -
   drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |   2 +-
   drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c      |   2 +-
   8 files changed, 10 insertions(+), 312 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index 003668f..335df11 100644
@@ -1844,7 +1844,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
   #define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
 
   /* Common functions */
-int amdgpu_gpu_reset(struct amdgpu_device *adev);
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job* job);
   bool amdgpu_need_backup(struct amdgpu_device *adev);
   void amdgpu_pci_config_reset(struct amdgpu_device *adev);
   bool amdgpu_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -298
index 0db3b3c..a2f9a7f 100644
@@ -2818,304 +2818,6 @@ static int amdgpu_recover_vram_from_shadow(struct amdgpu_device *adev,
   	return r;
   }
 
-/**
- * amdgpu_sriov_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- * @job: which job trigger hang
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * for SRIOV case.
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job)
-{
-	int i, j, r = 0;
-	int resched;
-	struct amdgpu_bo *bo, *tmp;
-	struct amdgpu_ring *ring;
-	struct dma_fence *fence = NULL, *next = NULL;
-
-	mutex_lock(&adev->virt.lock_reset);
-	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = true;
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-
-	/* we start from the ring trigger GPU hang */
-	j = job ? job->ring->idx : 0;
-
-	/* block scheduler */
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		kthread_park(ring->sched.thread);
-
-		if (job && j != i)
-			continue;
-
-		/* here give the last chance to check if job removed from mirror-list
-		 * since we already pay some time on kthread_park */
-		if (job && list_empty(&job->base.node)) {
-			kthread_unpark(ring->sched.thread);
-			goto give_up_reset;
-		}
-
-		if (amd_sched_invalidate_job(&job->base, amdgpu_job_hang_limit))
-			amd_sched_job_kickout(&job->base);
-
-		/* only do job_reset on the hang ring if @job not NULL */
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	/* request to take full control of GPU before re-initialization  */
-	if (job)
-		amdgpu_virt_reset_gpu(adev);
-	else
-		amdgpu_virt_request_full_gpu(adev, true);
-
-
-	/* Resume IP prior to SMC */
-	amdgpu_sriov_reinit_early(adev);
-
-	/* we need recover gart prior to run SMC/CP/SDMA resume */
-	amdgpu_ttm_recover_gart(adev);
-
-	/* now we are okay to resume SMC/CP/SDMA */
-	amdgpu_sriov_reinit_late(adev);
-
-	amdgpu_irq_gpu_reset_resume_helper(adev);
-
-	if (amdgpu_ib_ring_tests(adev))
-		dev_err(adev->dev, "[GPU_RESET] ib ring test failed (%d).\n", r);
-
-	/* release full control of GPU after ib test */
-	amdgpu_virt_release_full_gpu(adev, true);
-
-	DRM_INFO("recover vram bo from shadow\n");
-
-	ring = adev->mman.buffer_funcs_ring;
-	mutex_lock(&adev->shadow_list_lock);
-	list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-		next = NULL;
-		amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-		if (fence) {
-			r = dma_fence_wait(fence, false);
-			if (r) {
-				WARN(r, "recovery from shadow isn't completed\n");
-				break;
-			}
-		}
-
-		dma_fence_put(fence);
-		fence = next;
-	}
-	mutex_unlock(&adev->shadow_list_lock);
-
-	if (fence) {
-		r = dma_fence_wait(fence, false);
-		if (r)
-			WARN(r, "recovery from shadow isn't completed\n");
-	}
-	dma_fence_put(fence);
-
-	for (i = j; i < j + AMDGPU_MAX_RINGS; ++i) {
-		ring = adev->rings[i % AMDGPU_MAX_RINGS];
-		if (!ring || !ring->sched.thread)
-			continue;
-
-		if (job && j != i) {
-			kthread_unpark(ring->sched.thread);
-			continue;
-		}
-
-		amd_sched_job_recovery(&ring->sched);
-		kthread_unpark(ring->sched.thread);
-	}
-
-	drm_helper_resume_force_mode(adev->ddev);
-give_up_reset:
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	} else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	adev->in_sriov_reset = false;
-	mutex_unlock(&adev->virt.lock_reset);
-	return r;
-}
-
-/**
- * amdgpu_gpu_reset - reset the asic
- *
- * @adev: amdgpu device pointer
- *
- * Attempt the reset the GPU if it has hung (all asics).
- * Returns 0 for success or an error on failure.
- */
-int amdgpu_gpu_reset(struct amdgpu_device *adev)
-{
-	struct drm_atomic_state *state = NULL;
-	int i, r;
-	int resched;
-	bool need_full_reset, vram_lost = false;
-
-	if (!amdgpu_check_soft_reset(adev)) {
-		DRM_INFO("No hardware hang detected. Did some blocks stall?\n");
-		return 0;
-	}
-
-	atomic_inc(&adev->gpu_reset_counter);
-
-	/* block TTM */
-	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
-	/* store modesetting */
-	if (amdgpu_device_has_dc_support(adev))
-		state = drm_atomic_helper_suspend(adev->ddev);
-
-	/* block scheduler */
-	for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-		struct amdgpu_ring *ring = adev->rings[i];
-
-		if (!ring || !ring->sched.thread)
-			continue;
-		kthread_park(ring->sched.thread);
-		amd_sched_hw_job_reset(&ring->sched, NULL);
-		/* after all hw jobs are reset, hw fence is meaningless, so force_completion */
-		amdgpu_fence_driver_force_completion(ring);
-	}
-
-	need_full_reset = amdgpu_need_full_reset(adev);
-
-	if (!need_full_reset) {
-		amdgpu_pre_soft_reset(adev);
-		r = amdgpu_soft_reset(adev);
-		amdgpu_post_soft_reset(adev);
-		if (r || amdgpu_check_soft_reset(adev)) {
-			DRM_INFO("soft reset failed, will fallback to full reset!\n");
-			need_full_reset = true;
-		}
-	}
-
-	if (need_full_reset) {
-		r = amdgpu_suspend(adev);
-
-retry:
-		amdgpu_atombios_scratch_regs_save(adev);
-		r = amdgpu_asic_reset(adev);
-		amdgpu_atombios_scratch_regs_restore(adev);
-		/* post card */
-		amdgpu_atom_asic_init(adev->mode_info.atom_context);
-
-		if (!r) {
-			dev_info(adev->dev, "GPU reset succeeded, trying to resume\n");
-			r = amdgpu_resume_phase1(adev);
-			if (r)
-				goto out;
-			vram_lost = amdgpu_check_vram_lost(adev);
-			if (vram_lost) {
-				DRM_ERROR("VRAM is lost!\n");
-				atomic_inc(&adev->vram_lost_counter);
-			}
-			r = amdgpu_ttm_recover_gart(adev);
-			if (r)
-				goto out;
-			r = amdgpu_resume_phase2(adev);
-			if (r)
-				goto out;
-			if (vram_lost)
-				amdgpu_fill_reset_magic(adev);
-		}
-	}
-out:
-	if (!r) {
-		amdgpu_irq_gpu_reset_resume_helper(adev);
-		r = amdgpu_ib_ring_tests(adev);
-		if (r) {
-			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
-			r = amdgpu_suspend(adev);
-			need_full_reset = true;
-			goto retry;
-		}
-		/**
-		 * recovery vm page tables, since we cannot depend on VRAM is
-		 * consistent after gpu full reset.
-		 */
-		if (need_full_reset && amdgpu_need_backup(adev)) {
-			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
-			struct amdgpu_bo *bo, *tmp;
-			struct dma_fence *fence = NULL, *next = NULL;
-
-			DRM_INFO("recover vram bo from shadow\n");
-			mutex_lock(&adev->shadow_list_lock);
-			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
-				next = NULL;
-				amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);
-				if (fence) {
-					r = dma_fence_wait(fence, false);
-					if (r) {
-						WARN(r, "recovery from shadow isn't completed\n");
-						break;
-					}
-				}
-
-				dma_fence_put(fence);
-				fence = next;
-			}
-			mutex_unlock(&adev->shadow_list_lock);
-			if (fence) {
-				r = dma_fence_wait(fence, false);
-				if (r)
-					WARN(r, "recovery from shadow isn't completed\n");
-			}
-			dma_fence_put(fence);
-		}
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			struct amdgpu_ring *ring = adev->rings[i];
-
-			if (!ring || !ring->sched.thread)
-				continue;
-
-			amd_sched_job_recovery(&ring->sched);
-			kthread_unpark(ring->sched.thread);
-		}
-	} else {
-		dev_err(adev->dev, "asic resume failed (%d).\n", r);
-		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
-			if (adev->rings[i] && adev->rings[i]->sched.thread) {
-				kthread_unpark(adev->rings[i]->sched.thread);
-			}
-		}
-	}
-
-	if (amdgpu_device_has_dc_support(adev)) {
-		r = drm_atomic_helper_resume(adev->ddev, state);
-		amdgpu_dm_display_resume(adev);
-	} else
-		drm_helper_resume_force_mode(adev->ddev);
-
-	ttm_bo_unlock_delayed_workqueue(&adev->mman.bdev, resched);
-	if (r) {
-		/* bad news, how to tell it to userspace ? */
-		dev_info(adev->dev, "GPU reset failed\n");
-	}
-	else {
-		dev_info(adev->dev, "GPU reset successed!\n");
-	}
-
-	amdgpu_vf_error_trans_all(adev);
-	return r;
-}
-
   static int amdgpu_reset(struct amdgpu_device *adev, uint64_t* reset_flags)
   {
   	int r;
drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +5 -5
index d7374cf..9d67bcb 100644
@@ -694,25 +694,25 @@ static int amdgpu_debugfs_fence_info(struct seq_file *m, void *data)
   }
 
   /**
- * amdgpu_debugfs_gpu_reset - manually trigger a gpu reset
+ * amdgpu_debugfs_gpu_recover - manually trigger a gpu reset & recover
    *
    * Manually trigger a gpu reset at the next fence wait.
    */
-static int amdgpu_debugfs_gpu_reset(struct seq_file *m, void *data)
+static int amdgpu_debugfs_gpu_recover(struct seq_file *m, void *data)
   {
   	struct drm_info_node *node = (struct drm_info_node *) m->private;
   	struct drm_device *dev = node->minor->dev;
   	struct amdgpu_device *adev = dev->dev_private;
 
-	seq_printf(m, "gpu reset\n");
-	amdgpu_gpu_reset(adev);
+	seq_printf(m, "gpu recover\n");
+	amdgpu_gpu_recover(adev, NULL);
 
   	return 0;
   }
 
   static const struct drm_info_list amdgpu_debugfs_fence_list[] = {
   	{"amdgpu_fence_info", &amdgpu_debugfs_fence_info, 0, NULL},
-	{"amdgpu_gpu_reset", &amdgpu_debugfs_gpu_reset, 0, NULL}
+	{"amdgpu_gpu_recover", &amdgpu_debugfs_gpu_recover, 0, NULL}
   };
 
   static const struct drm_info_list amdgpu_debugfs_fence_list_sriov[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -1
index 47c5ce9..9c2f87c 100644
@@ -88,7 +88,7 @@ static void amdgpu_irq_reset_work_func(struct work_struct *work)
   						  reset_work);
 
   	if (!amdgpu_sriov_vf(adev))
-		amdgpu_gpu_reset(adev);
+		amdgpu_gpu_recover(adev, NULL);
   }
 
   /* Disable *all* interrupts */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -4
index f08fde9..ac6a4f3 100644
@@ -37,10 +37,7 @@ static void amdgpu_job_timedout(struct amd_sched_job *s_job)
   		  atomic_read(&job->ring->fence_drv.last_seq),
   		  job->ring->fence_drv.sync_seq);
 
-	if (amdgpu_sriov_vf(job->adev))
-		amdgpu_sriov_gpu_reset(job->adev, job);
-	else
-		amdgpu_gpu_reset(job->adev);
+	amdgpu_gpu_recover(job->adev, job);
   }
 
   int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index b89d37f..3a661aa 100644
@@ -285,7 +285,6 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
   int amdgpu_virt_request_full_gpu(struct amdgpu_device *adev, bool init);
   int amdgpu_virt_release_full_gpu(struct amdgpu_device *adev, bool init);
   int amdgpu_virt_reset_gpu(struct amdgpu_device *adev);
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job);
   int amdgpu_virt_alloc_mm_table(struct amdgpu_device *adev);
   void amdgpu_virt_free_mm_table(struct amdgpu_device *adev);
   int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +1 -1
index b4906d2..f8522a0 100644
@@ -254,7 +254,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work)
   	}
 
   	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
   }
 
   static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +0 -0
index c25a831..dae6d3a 100644
@@ -514,7 +514,7 @@ static void xgpu_vi_mailbox_flr_work(struct work_struct *work)
   	}
 
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c static void xgpu_vi_mailbox_flr_work(struct work_struct *work) · line 520

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +1 -1
   	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
   }
 
   static int xgpu_vi_set_mailbox_rcv_irq(struct amdgpu_device *adev,

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#20 lore raw

PATCH 5/9 drm/amdgpu:cleanup in_sriov_reset and lock_reset

In reply to a message not in the archive

since now gpu reset is unified with gpu_recover
for both bare-metal and SR-IOV:

1)rename in_sriov_reset to in_gpu_reset
2)move lock_reset from adev->virt to adev

Change-Id I9f4dbab9a4c916fbc156f669824d15ddcd0f2322
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 +++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c    | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c  | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c   | 2 --
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   | 1 -
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 6 +++---
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 6 +++---
 8 files changed, 15 insertions(+), 16 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -1
index 335df11..6e89be5 100644
@@ -1650,7 +1650,8 @@ struct amdgpu_device {
 
 	/* record last mm index being written through WREG32*/
 	unsigned long last_mm_index;
-	bool                            in_sriov_reset;
+	bool                            in_gpu_reset;
+	struct mutex  lock_reset;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +5 -4
index a2f9a7f..4cf1146 100644
@@ -2161,6 +2161,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	mutex_init(&adev->mn_lock);
 	mutex_init(&adev->virt.vf_errors.lock);
 	hash_init(adev->mn_hash);
+	mutex_init(&adev->lock_reset);
 
 	amdgpu_check_arguments(adev);
 
@@ -2963,9 +2964,9 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
 
 	dev_info(adev->dev, "GPU reset begin!\n");
 
-	mutex_lock(&adev->virt.lock_reset);
+	mutex_lock(&adev->lock_reset);
 	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = 1;
+	adev->in_gpu_reset = 1;
 
 	/* block TTM */
 	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
@@ -3075,8 +3076,8 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
 	}
 
 	amdgpu_vf_error_trans_all(adev);
-	adev->in_sriov_reset = 0;
-	mutex_unlock(&adev->virt.lock_reset);
+	adev->in_gpu_reset = 0;
+	mutex_unlock(&adev->lock_reset);
 	return r;
 }
 
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +1 -1
index 447d446..76f531b 100644
@@ -264,7 +264,7 @@ static int psp_hw_start(struct psp_context *psp)
 	struct amdgpu_device *adev = psp->adev;
 	int ret;
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
 		ret = psp_bootloader_load_sysdrv(psp);
 		if (ret)
 			return ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +1 -1
index 6564902..edc37cc 100644
@@ -370,7 +370,7 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
 		return 0;
 	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
 		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
 					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
 					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +0 -2
index e97f80f..c249725 100644
@@ -107,8 +107,6 @@ void amdgpu_virt_init_setting(struct amdgpu_device *adev)
 	adev->enable_virtual_display = true;
 	adev->cg_flags = 0;
 	adev->pg_flags = 0;
-
-	mutex_init(&adev->virt.lock_reset);
 }
 
 uint32_t amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t reg)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index 3a661aa..a710384 100644
@@ -238,7 +238,6 @@ struct amdgpu_virt {
 	uint64_t			csa_vmid0_addr;
 	bool chained_ib_support;
 	uint32_t			reg_val_offs;
-	struct mutex                    lock_reset;
 	struct amdgpu_irq_src		ack_irq;
 	struct amdgpu_irq_src		rcv_irq;
 	struct work_struct		flr_work;
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +3 -3
index b8002ac..72ddb6e 100644
@@ -4807,7 +4807,7 @@ static int gfx_v8_0_kiq_init_queue(struct amdgpu_ring *ring)
 
 	gfx_v8_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
@@ -4844,7 +4844,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
 	struct vi_mqd *mqd = ring->mqd_ptr;
 	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
 		memset((void *)mqd, 0, sizeof(struct vi_mqd_allocation));
 		((struct vi_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
 		((struct vi_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;
@@ -4856,7 +4856,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct vi_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +3 -3
index 8738b13..d598d78 100644
@@ -2732,7 +2732,7 @@ static int gfx_v9_0_kiq_init_queue(struct amdgpu_ring *ring)
 
 	gfx_v9_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));
@@ -2770,7 +2770,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 	struct v9_mqd *mqd = ring->mqd_ptr;
 	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
 		memset((void *)mqd, 0, sizeof(struct v9_mqd_allocation));
 		((struct v9_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
 		((struct v9_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;
@@ -2782,7 +2782,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct v9_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-6-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

Reviewed-by
#22 lore raw

Re: [PATCH 5/9] drm/amdgpu:cleanup in_sriov_reset and lock_reset

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 8 lines

since now gpu reset is unified with gpu_recover
for both bare-metal and SR-IOV:

1)rename in_sriov_reset to in_gpu_reset
2)move lock_reset from adev->virt to adev

Change-Id I9f4dbab9a4c916fbc156f669824d15ddcd0f2322
Quoted 165 lines
---
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 3 ++-
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 +++++----
  drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c    | 2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c  | 2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c   | 2 --
  drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   | 1 -
  drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 6 +++---
  drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 6 +++---
  8 files changed, 15 insertions(+), 16 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -1
index 335df11..6e89be5 100644
@@ -1650,7 +1650,8 @@ struct amdgpu_device {
 
  	/* record last mm index being written through WREG32*/
  	unsigned long last_mm_index;
-	bool                            in_sriov_reset;
+	bool                            in_gpu_reset;
+	struct mutex  lock_reset;
  };
 
  static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +5 -4
index a2f9a7f..4cf1146 100644
@@ -2161,6 +2161,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
  	mutex_init(&adev->mn_lock);
  	mutex_init(&adev->virt.vf_errors.lock);
  	hash_init(adev->mn_hash);
+	mutex_init(&adev->lock_reset);
 
  	amdgpu_check_arguments(adev);
 
@@ -2963,9 +2964,9 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
 
  	dev_info(adev->dev, "GPU reset begin!\n");
 
-	mutex_lock(&adev->virt.lock_reset);
+	mutex_lock(&adev->lock_reset);
  	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = 1;
+	adev->in_gpu_reset = 1;
 
  	/* block TTM */
  	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
@@ -3075,8 +3076,8 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
  	}
 
  	amdgpu_vf_error_trans_all(adev);
-	adev->in_sriov_reset = 0;
-	mutex_unlock(&adev->virt.lock_reset);
+	adev->in_gpu_reset = 0;
+	mutex_unlock(&adev->lock_reset);
  	return r;
  }
 
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +1 -1
index 447d446..76f531b 100644
@@ -264,7 +264,7 @@ static int psp_hw_start(struct psp_context *psp)
  	struct amdgpu_device *adev = psp->adev;
  	int ret;
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
  		ret = psp_bootloader_load_sysdrv(psp);
  		if (ret)
  			return ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +1 -1
index 6564902..edc37cc 100644
@@ -370,7 +370,7 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
  		return 0;
  	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
  		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
  					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
  					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +0 -2
index e97f80f..c249725 100644
@@ -107,8 +107,6 @@ void amdgpu_virt_init_setting(struct amdgpu_device *adev)
  	adev->enable_virtual_display = true;
  	adev->cg_flags = 0;
  	adev->pg_flags = 0;
-
-	mutex_init(&adev->virt.lock_reset);
  }
 
  uint32_t amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t reg)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index 3a661aa..a710384 100644
@@ -238,7 +238,6 @@ struct amdgpu_virt {
  	uint64_t			csa_vmid0_addr;
  	bool chained_ib_support;
  	uint32_t			reg_val_offs;
-	struct mutex                    lock_reset;
  	struct amdgpu_irq_src		ack_irq;
  	struct amdgpu_irq_src		rcv_irq;
  	struct work_struct		flr_work;
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +3 -3
index b8002ac..72ddb6e 100644
@@ -4807,7 +4807,7 @@ static int gfx_v8_0_kiq_init_queue(struct amdgpu_ring *ring)
 
  	gfx_v8_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
  		/* reset MQD to a clean status */
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
@@ -4844,7 +4844,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
  	struct vi_mqd *mqd = ring->mqd_ptr;
  	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
  		memset((void *)mqd, 0, sizeof(struct vi_mqd_allocation));
  		((struct vi_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
  		((struct vi_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;
@@ -4856,7 +4856,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
 
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct vi_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
  		/* reset MQD to a clean status */
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +2 -2
index 8738b13..d598d78 100644
@@ -2732,7 +2732,7 @@ static int gfx_v9_0_kiq_init_queue(struct amdgpu_ring *ring)
 
  	gfx_v9_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
  		/* reset MQD to a clean status */
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));
@@ -2770,7 +2770,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
  	struct v9_mqd *mqd = ring->mqd_ptr;
  	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
  		memset((void *)mqd, 0, sizeof(struct v9_mqd_allocation));
  		((struct v9_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
  		((struct v9_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;

drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring) · line 2788

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +1 -1
@@ -2782,7 +2782,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct v9_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
  		/* reset MQD to a clean status */
  		if (adev->gfx.mec.mqd_backup[mqd_idx])
  			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#23 lore raw

PATCH 6/9 drm/amdgpu:cleanup ucode_init_bo

In reply to a message not in the archive

1,no sriov check since gpu recover is unified 2,need CPU_ACCESS_REQUIRED flag for VRAM if SRIOV because otherwise after following PIN the first allocated VRAM bo is wasted due to some TTM mgr reason.

Change-Id I4d029f2da8bb463942c7861d3e52f309bdba9576
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +2 -1
index edc37cc..4939e5a 100644
@@ -370,9 +370,10 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
 		return 0;
 	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
+	if (!adev->in_gpu_reset) {
 		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
 					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
+					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS|AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED :
 					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
 					NULL, NULL, 0, bo);
 		if (err) {
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-7-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#25 lore raw

Re: [PATCH 6/9] drm/amdgpu:cleanup ucode_init_bo

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 16 lines

1,no sriov check since gpu recover is unified 2,need CPU_ACCESS_REQUIRED flag for VRAM if SRIOV because otherwise after following PIN the first allocated VRAM bo is wasted due to some TTM mgr reason.

Change-Id I4d029f2da8bb463942c7861d3e52f309bdba9576
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +0 -0
index edc37cc..4939e5a 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c int amdgpu_ucode_init_bo(struct amdgpu_device *adev) · line 376

Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +2 -1
@@ -370,9 +370,10 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
  		return 0;
  	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
+	if (!adev->in_gpu_reset) {
  		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
  					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
+					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS|AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED :

Just set the AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED unconditionally here.

Apart from that feel free to add my Acked-by.

Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c int amdgpu_ucode_init_bo(struct amdgpu_device *adev) · line 379

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +0 -0
  					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
  					NULL, NULL, 0, bo);
  		if (err) {

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#26 lore raw

PATCH 7/9 drm/amdgpu:block kms open during gpu_reset

In reply to a message not in the archive
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
 	if (r < 0)
 		return r;
 
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
 	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
 	if (unlikely(!fpriv)) {
 		r = -ENOMEM;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-8-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#28 lore raw

Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 11 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
  	if (r < 0)
  		return r;
 
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
  	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
  	if (unlikely(!fpriv)) {
  		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 46bc9872-0bd7-c2b7-4879-6cab5789f252-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author NAK
#30 lore raw

RE: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 12 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
  1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 9 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
  	if (r < 0)
  		return r;
 
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
  	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
  	if (unlikely(!fpriv)) {
  		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB0449D707ECAE76FFAA13C46684450-7LeqcoF/[email protected]

#32 lore raw

Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 1 line

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 36 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
   1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
   	if (r < 0)
   		return r;
 
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
   	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
   	if (unlikely(!fpriv)) {
   		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 06a765e3-971e-698c-6d07-1dee69c1e0c8-5C7GfCeVMHo@public.gmane.org

author
#34 lore raw

RE: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
   1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
   	if (r < 0)
   		return r;
 
Quoted 7 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
   	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
   	if (unlikely(!fpriv)) {
   		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB044997E32FD02ACDBD978C7084450-7LeqcoF/[email protected]

#36 lore raw

Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Am 26.10.2017 um 13:08 schrieb Liu, Monk:

Quoted 1 line

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

I can't see any difference between the handling of existing VMs and new created ones.

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

So we need to handle this gracefully anyway,
Christian.

Quoted 57 lines

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
    1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
    	if (r < 0)
    		return r;
 
Quoted 10 lines
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
    	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
    	if (unlikely(!fpriv)) {
    		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 46a5317f-e606-5cfe-82c9-46df64818435-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author
#38 lore raw

RE: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem: use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock.
But you gave NAK on this approach, so I want to hear your idea.

Quoted 1 line

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 23:15
To: Liu, Monk <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Am 26.10.2017 um 13:08 schrieb Liu, Monk:

Quoted 1 line

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

I can't see any difference between the handling of existing VMs and new created ones.

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

So we need to handle this gracefully anyway, Christian.

Quoted 57 lines

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
    drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
    1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
    	if (r < 0)
    		return r;
 
Quoted 10 lines
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
    	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
    	if (unlikely(!fpriv)) {
    		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB0449DD842E46A942CF53484084590-7LeqcoF/[email protected]

#40 lore raw

Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 6 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem: use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock.
But you gave NAK on this approach, so I want to hear your idea.

Well, what I wanted to say is we simply don't have a problem here. The handling for new VMs is exactly the same as for existing VMs.

Regarding the lock that is still a big NAK, we just fixed the (hopefully) last deadlock between IOCTLs and GPU reset. Introducing any locks would cause problems with that once more.

What we can do is use a struct completion to prevent new changes from piling up while we are in the GPU reset code.

Quoted 1 line

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

That is not correct. All VM activity goes through the GPU scheduler as well. So we are certainly going to re-schedule everything.

Quoted 2 lines
Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

As I said we don't need to do anything, the handling is correct as it is right now.

E.g. we have a shadow copy of the page tables in system memory to recover after VRAM lost and reschedule any pending operation after GPU reset.

That should be sufficient to handle all existing VM cases and even when it's not we need to work on the general handling and not add any checks to the IOCTL code which prevents applications from loading the driver.

Regards,
Christian.

Am 30.10.2017 um 04:47 schrieb Liu, Monk:

Quoted 25 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem: use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock.
But you gave NAK on this approach, so I want to hear your idea.

Quoted 1 line

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 23:15
To: Liu, Monk <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 75 lines

Am 26.10.2017 um 13:08 schrieb Liu, Monk:

Quoted 1 line

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

I can't see any difference between the handling of existing VMs and new created ones.

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

So we need to handle this gracefully anyway, Christian.

Quoted 66 lines

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
     drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
     1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
     	if (r < 0)
     		return r;
 
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
     	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
     	if (unlikely(!fpriv)) {
     		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): f2213f71-9280-bb51-5283-5349878956f7-5C7GfCeVMHo@public.gmane.org

author
#42 lore raw

RE: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 1 line

That is not correct. All VM activity goes through the GPU scheduler as well. So we are certainly going to re-schedule everything.

I'll check on that, if all go through scheduler that will be fine, that way just don't increase the karma of the job from kernel RQ is enough To prevent VM jobs being corrupted by gpu reset, we can re-schedule those jobs if they are not signaled (but that need timedout not zero)

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月30日 15:37
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 6 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem:
use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock. But you gave NAK on this approach, so I want to hear your idea.

Well, what I wanted to say is we simply don't have a problem here. The handling for new VMs is exactly the same as for existing VMs.

Regarding the lock that is still a big NAK, we just fixed the
(hopefully) last deadlock between IOCTLs and GPU reset. Introducing any locks would cause problems with that once more.

What we can do is use a struct completion to prevent new changes from piling up while we are in the GPU reset code.

Quoted 1 line

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

That is not correct. All VM activity goes through the GPU scheduler as well. So we are certainly going to re-schedule everything.

Quoted 2 lines
Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

As I said we don't need to do anything, the handling is correct as it is right now.

E.g. we have a shadow copy of the page tables in system memory to recover after VRAM lost and reschedule any pending operation after GPU reset.

That should be sufficient to handle all existing VM cases and even when it's not we need to work on the general handling and not add any checks to the IOCTL code which prevents applications from loading the driver.

Regards,
Christian.

Am 30.10.2017 um 04:47 schrieb Liu, Monk:

Quoted 26 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem:
use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock. But you gave NAK on this approach, so I want to hear your idea.

Quoted 1 line

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 23:15
To: Liu, Monk <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 75 lines

Am 26.10.2017 um 13:08 schrieb Liu, Monk:

Quoted 1 line

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

I can't see any difference between the handling of existing VMs and new created ones.

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

So we need to handle this gracefully anyway, Christian.

Quoted 66 lines

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
     drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
     1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
     	if (r < 0)
     		return r;
 
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
     	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
     	if (unlikely(!fpriv)) {
     		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): SN1PR12MB04629FAC65DFD5F133D2E34484590-z7L1TMIYDg7VaWpRXmIMygdYzm3356FpvxpqHgZTriW3zl9H0oFU5g@public.gmane.org

#44 lore raw

Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

The only thing that is not pushed through the scheduler are the ring and IB tests and those only because we want to use them directly in the GPU reset code to check if the hardware now works properly again.

The VM updates have a separate queue for each VM, but the initial clear goes through the kernel queue which is also used for buffer moves during eviction. But both always go through the scheduler.

Originally used the VM queue for the initial clear as well, but I changed that because I thought we could save some complexity. Might be a good idea to revert that because it is not as obvious and caused trouble with the ATC support on VG10 as well.

Regards,
Christian.

Am 30.10.2017 um 08:54 schrieb Liu, Monk:

Quoted 140 lines
Quoted 1 line

That is not correct. All VM activity goes through the GPU scheduler as well. So we are certainly going to re-schedule everything.

I'll check on that, if all go through scheduler that will be fine, that way just don't increase the karma of the job from kernel RQ is enough To prevent VM jobs being corrupted by gpu reset, we can re-schedule those jobs if they are not signaled (but that need timedout not zero)

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月30日 15:37
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 6 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem:
use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock. But you gave NAK on this approach, so I want to hear your idea.

Well, what I wanted to say is we simply don't have a problem here. The handling for new VMs is exactly the same as for existing VMs.

Regarding the lock that is still a big NAK, we just fixed the
(hopefully) last deadlock between IOCTLs and GPU reset. Introducing any locks would cause problems with that once more.

What we can do is use a struct completion to prevent new changes from piling up while we are in the GPU reset code.

Quoted 1 line

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

That is not correct. All VM activity goes through the GPU scheduler as well. So we are certainly going to re-schedule everything.

Quoted 2 lines
Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

As I said we don't need to do anything, the handling is correct as it is right now.

E.g. we have a shadow copy of the page tables in system memory to recover after VRAM lost and reschedule any pending operation after GPU reset.

That should be sufficient to handle all existing VM cases and even when it's not we need to work on the general handling and not add any checks to the IOCTL code which prevents applications from loading the driver.

Regards,
Christian.

Am 30.10.2017 um 04:47 schrieb Liu, Monk:

Quoted 26 lines
Quoted 1 line

I can't see any difference between the handling of existing VMs and new created ones.

I know, for existing VMs we still have similar problems, I'm not saying this patch can save existing VM problem ...

My eldest patch series actually use a way can 100% avoid such problem:
use RW mlock on drm_ioctl and gpu_recover(), drm_ioctl() take the Read lock, and gpu_recover() take the write lock. But you gave NAK on this approach, so I want to hear your idea.

Quoted 1 line

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

The thing is some VM activity is not go through GPU scheduler (direct), if it is interrupted by gpu_recover() it's not going to be re-scheduled again ...

Quoted 1 line

So we need to handle this gracefully anyway, Christian.

Yeah I'd like to hear

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 23:15
To: Liu, Monk <[email protected]>; Koenig, Christian <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 74 lines

Am 26.10.2017 um 13:08 schrieb Liu, Monk:

Quoted 1 line

"Clear operation on the page table " is some kind of SDMA activity right? What if ASIC RESET from amd_gpu_recover() interrupted this activity in fly ???

I can't see any difference between the handling of existing VMs and new created ones.

Either we have correct handling and can redo the activity or we have corrupted VM page tables and crash again immediately.

So we need to handle this gracefully anyway, Christian.

Quoted 66 lines

BR Monk

-----Original Message-----
From: Koenig, Christian
Sent: 2017年10月26日 18:54
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

Quoted 2 lines

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

That is not a problem at all. SDMA just does some clear operation on the page tables and those are either recovered from the shadow or run after the reset.

Regards,
Christian.

Am 26.10.2017 um 10:17 schrieb Liu, Monk:

Quoted 40 lines

When amdgpu_gpu_recover() routine is in the fly, we shouldn't let UMD open our device, otherwise the VM init would be ruined by gpu_recover().

e.g. VM init need to create page table, but keep In mind that gpu_recover() calls ASIC RESET,

if we don't block device open while gpu doing recover, the vm init (SDMA working on page table creating) would be ruined by ASIC RESET

do you have any good solution ? the key point is avoid/delay/push_back hw activities from UMD side when we are running in gpu_recover() function

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 15:18
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 7/9] drm/amdgpu:block kms open during gpu_reset

NAK, why the heck should we do this? It would just block all new processes from using the device.

Christian.

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 15 lines
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
      drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
      1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +0 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
      	if (r < 0)
      		return r;
 
Quoted 6 lines

drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv) · line 821

Quoted 6 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
      	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
      	if (unlikely(!fpriv)) {
      		r = -ENOMEM;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#45 lore raw

PATCH 8/9 drm/amdgpu/sriov:fix memory leak in psp_load_fw

In reply to a message not in the archive

for SR-IOV when doing gpu reset this routine shouldn't do resource allocating otherwise memory leak

Change-Id I25da3a5b475196c75c7e639adc40751754625968
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +12 -8
index 76f531b..2157d45 100644
@@ -334,23 +334,26 @@ static int psp_load_fw(struct amdgpu_device *adev)
 	int ret;
 	struct psp_context *psp = &adev->psp;
 
+	if (amdgpu_sriov_vf(adev) && adev->in_gpu_reset != 0)
+		goto skip_memalloc;
+
 	psp->cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
 	if (!psp->cmd)
 		return -ENOMEM;
 
 	ret = amdgpu_bo_create_kernel(adev, PSP_1_MEG, PSP_1_MEG,
-				      AMDGPU_GEM_DOMAIN_GTT,
-				      &psp->fw_pri_bo,
-				      &psp->fw_pri_mc_addr,
-				      &psp->fw_pri_buf);
+					AMDGPU_GEM_DOMAIN_GTT,
+					&psp->fw_pri_bo,
+					&psp->fw_pri_mc_addr,
+					&psp->fw_pri_buf);
 	if (ret)
 		goto failed;
 
 	ret = amdgpu_bo_create_kernel(adev, PSP_FENCE_BUFFER_SIZE, PAGE_SIZE,
-				      AMDGPU_GEM_DOMAIN_VRAM,
-				      &psp->fence_buf_bo,
-				      &psp->fence_buf_mc_addr,
-				      &psp->fence_buf);
+					AMDGPU_GEM_DOMAIN_VRAM,
+					&psp->fence_buf_bo,
+					&psp->fence_buf_mc_addr,
+					&psp->fence_buf);
 	if (ret)
 		goto failed_mem2;
 
@@ -375,6 +378,7 @@ static int psp_load_fw(struct amdgpu_device *adev)
 	if (ret)
 		goto failed_mem;
 
+skip_memalloc:
 	ret = psp_hw_start(psp);
 	if (ret)
 		goto failed_mem;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-9-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#47 lore raw

Re: [PATCH 8/9] drm/amdgpu/sriov:fix memory leak in psp_load_fw

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 2 lines

for SR-IOV when doing gpu reset this routine shouldn't do resource allocating otherwise memory leak

Can't judge if this patch is correct or not, but fixing the whitespaces should be a separate patch.

Christian.

Quoted 47 lines
Change-Id I25da3a5b475196c75c7e639adc40751754625968
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 20 ++++++++++++--------
  1 file changed, 12 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +11 -8
index 76f531b..2157d45 100644
@@ -334,23 +334,26 @@ static int psp_load_fw(struct amdgpu_device *adev)
  	int ret;
  	struct psp_context *psp = &adev->psp;
 
+	if (amdgpu_sriov_vf(adev) && adev->in_gpu_reset != 0)
+		goto skip_memalloc;
+
  	psp->cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
  	if (!psp->cmd)
  		return -ENOMEM;
 
  	ret = amdgpu_bo_create_kernel(adev, PSP_1_MEG, PSP_1_MEG,
-				      AMDGPU_GEM_DOMAIN_GTT,
-				      &psp->fw_pri_bo,
-				      &psp->fw_pri_mc_addr,
-				      &psp->fw_pri_buf);
+					AMDGPU_GEM_DOMAIN_GTT,
+					&psp->fw_pri_bo,
+					&psp->fw_pri_mc_addr,
+					&psp->fw_pri_buf);
  	if (ret)
  		goto failed;
 
  	ret = amdgpu_bo_create_kernel(adev, PSP_FENCE_BUFFER_SIZE, PAGE_SIZE,
-				      AMDGPU_GEM_DOMAIN_VRAM,
-				      &psp->fence_buf_bo,
-				      &psp->fence_buf_mc_addr,
-				      &psp->fence_buf);
+					AMDGPU_GEM_DOMAIN_VRAM,
+					&psp->fence_buf_bo,
+					&psp->fence_buf_mc_addr,
+					&psp->fence_buf);
  	if (ret)
  		goto failed_mem2;
 

drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c static int psp_load_fw(struct amdgpu_device *adev) · line 384

Quoted 7 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +1 -0
@@ -375,6 +378,7 @@ static int psp_load_fw(struct amdgpu_device *adev)
  	if (ret)
  		goto failed_mem;
 
+skip_memalloc:
  	ret = psp_hw_start(psp);
  	if (ret)
  		goto failed_mem;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#48 lore raw

PATCH 9/9 drm/amdgpu:fix random missing of FLR NOTIFY

In reply to a message not in the archive
---
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +11 -3
index f8522a0..a43cffb 100644
@@ -282,9 +282,17 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev,
 		/* see what event we get */
 		r = xgpu_ai_mailbox_rcv_msg(adev, IDH_FLR_NOTIFICATION);
 
-		/* only handle FLR_NOTIFY now */
-		if (!r)
-			schedule_work(&adev->virt.flr_work);
+		/* sometimes the interrupt is delayed to inject to VM, so under such case
+		 * the IDH_FLR_NOTIFICATION is overwritten by VF FLR from GIM side, thus
+		 * above recieve message could be failed, we should schedule the flr_work
+		 * anyway
+		 */
+		if (r) {
+			DRM_ERROR("FLR_NOTIFICATION is missed\n");
+			xgpu_ai_mailbox_send_ack(adev);
+		}
+
+		schedule_work(&adev->virt.flr_work);
 	}
 
 	return 0;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1508923343-24404-10-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#50 lore raw

Re: [PATCH 9/9] drm/amdgpu:fix random missing of FLR NOTIFY

Am 25.10.2017 um 11:22 schrieb Monk Liu:

Quoted 1 line

No idea if this is correct or not, but it looks like an important fix which should go into the branch first.

Feel free to add my Acked-by: Christian König <[email protected]>.

Regards,
Christian.

Quoted 19 lines
---
  drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c | 14 +++++++++++---
  1 file changed, 11 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +4 -3
index f8522a0..a43cffb 100644
@@ -282,9 +282,17 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev,
  		/* see what event we get */
  		r = xgpu_ai_mailbox_rcv_msg(adev, IDH_FLR_NOTIFICATION);
 
-		/* only handle FLR_NOTIFY now */
-		if (!r)
-			schedule_work(&adev->virt.flr_work);
+		/* sometimes the interrupt is delayed to inject to VM, so under such case
+		 * the IDH_FLR_NOTIFICATION is overwritten by VF FLR from GIM side, thus
+		 * above recieve message could be failed, we should schedule the flr_work
+		 * anyway

drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev, · line 298

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +7 -0
+		 */
+		if (r) {
+			DRM_ERROR("FLR_NOTIFICATION is missed\n");
+			xgpu_ai_mailbox_send_ack(adev);
+		}
+
+		schedule_work(&adev->virt.flr_work);
  	}
 
  	return 0;

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx