Skip to contentSkip to conversation

*** gpu recover patches series v2 ***

PATCH In review Monk Liu 3 participants 12 messages Oct 26, 2017 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Monk Liu

author
#0 lore raw

cover *** gpu recover patches series v2 ***

v2 series:

fixed some logic error in "imple job skip feature" merge two patches into "implement new GPU recover(v3)"

Monk Liu (7):
amd/scheduler:imple job skip feature(v2) drm/amdgpu:implement new GPU recover(v3) drm/amdgpu:cleanup in_sriov_reset and lock_reset drm/amdgpu:cleanup ucode_init_bo drm/amdgpu:block kms open during gpu_reset drm/amdgpu/sriov:fix memory leak in psp_load_fw drm/amdgpu:fix random missing of FLR NOTIFY

 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |   9 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    | 311 ++++++++++++--------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c     |  10 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c       |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  14 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c       |   3 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c       |  22 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c     |   4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c      |   2 -
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h      |   2 -
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c         |   6 +-
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c         |   6 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c         |  16 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c         |   2 +-
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c |  54 +++--
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |   2 +-
 16 files changed, 236 insertions(+), 229 deletions(-)
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1509013827-8180-1-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

author
#2 lore raw

PATCH 1/7 amd/scheduler:imple job skip feature(v2)

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

v2:
some logic fix

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++--
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 54 ++++++++++++++++++++-------
 drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
 3 files changed, 47 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +5 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
 	return fence;
 }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job, bool skip)
 {
 	struct dma_fence *fence = NULL;
 	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
 	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
 	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
 		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
 	} else {
 		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
 				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +41 -13

116 lines not on this page · raw message

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +1 -1
index f9e3a83..a8e5a7d 100644
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
 */
 struct amd_sched_backend_ops {
 	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool skip);
 	void (*timedout_job)(struct amd_sched_job *sched_job);
 	void (*free_job)(struct amd_sched_job *sched_job);
 };
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1509013827-8180-2-git-send-email-Monk.Liu-5C7GfCeVMHo@public.gmane.org

#4 lore raw

Re: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Am 26.10.2017 um 12:30 schrieb Monk Liu:

Quoted 60 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

v2:
some logic fix

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++--
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 54 ++++++++++++++++++++-------
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
  3 files changed, 47 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +5 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
  	return fence;
  }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job, bool skip)
  {
  	struct dma_fence *fence = NULL;
  	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
  	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
  	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
  	} else {
  		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
  				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -0
index 9cbeade..995661e 100644
@@ -330,6 +330,28 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
  	return false;
  }
 
+static void amd_sched_job_fake_signal(struct amd_sched_job *job)
+{

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 344

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +10 -0
+	struct amd_sched_fence *s_fence = job->s_fence;
+	struct amd_gpu_scheduler *sched = job->sched;
+	struct amd_sched_entity *entity = job->s_entity;
+
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 348

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+
+	/* fake signaling the finished fence */
+	job->s_entity = NULL;
+	spsc_queue_pop(&entity->job_queue);

That code isn't up to date any more, Andrey removed job->s_entity yesterday.

Please rebase on amd-staging-drm-next before resending.

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+
+	amd_sched_process_job(NULL, &s_fence->cb);
+	dma_fence_put(&s_fence->finished);
+	sched->ops->free_job(job);
+}
+
  static struct amd_sched_job *
  amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 373

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -0
@@ -344,6 +366,11 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  		if (amd_sched_entity_add_dependency_cb(entity))
  			return NULL;
 
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		return NULL;
+	}
+

Better move this after spsc_queue_pop below and remove the extra spsc_queue_pop from amd_sched_job_fake_signal.

Quoted 50 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -9
  	sched_job->s_entity = NULL;
  	spsc_queue_pop(&entity->job_queue);
  	return sched_job;
@@ -441,13 +468,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
  	job->sched->ops->timedout_job(job);
  }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job)
-{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
  void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
  {
  	struct amd_sched_job *s_job;
@@ -466,7 +486,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  	}
  	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit) {
  		bool found = false;
 
  		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ ) {
@@ -476,7 +496,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  			list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
  				if (bad->s_fence->scheduled.context == entity->fence_context) {
  					found = true;
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
  					break;
  				}
  			}
@@ -499,6 +520,7 @@ void amd_sched_job_kickout(struct amd_sched_job *s_job)
  void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  {
  	struct amd_sched_job *s_job, *tmp;
+	bool found_guilty = false;
  	int r;
 
  	spin_lock(&sched->job_list_lock);
@@ -510,9 +532,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool skip;
+		uint64_t guilty_context;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 545

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +7 -1
+
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

Regards,
Christian.

Quoted 33 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -3
  		atomic_inc(&sched->hw_rq_count);
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence);
@@ -525,7 +555,6 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
  		spin_lock(&sched->job_list_lock);
@@ -650,7 +679,7 @@ static int amd_sched_main(void *param)
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
  		amd_sched_fence_scheduled(s_fence);
 
  		if (fence) {
@@ -664,7 +693,6 @@ static int amd_sched_main(void *param)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
 
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr · line 132

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +1 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
  */
  struct amd_sched_backend_ops {
  	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool skip);
  	void (*timedout_job)(struct amd_sched_job *sched_job);
  	void (*free_job)(struct amd_sched_job *sched_job);
  };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): a126acf5-1023-81c1-bebb-fe28be8f525f-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org

author
#6 lore raw

RE: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 344

Quoted 6 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

[ml] WHAT ??? this wake_up operate on job_scheduled, like the one at the bottom of sched_main(), It is used to wake up the thread waiting in "sched_entity_fini" on this "sched->job_scheduled", I don't understand why not necessary

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 545

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -1
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == 
+guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

[ML] no you still didn't get my point, in sched_main() we use fake_signal() because the job hadn't been linked in mirror list, so "job_finish()" Cannot be invoked for those jobs,

But for job_recovery(), it go through mirror list and only focus on jobs from that list, thus all jobs must be handled by "job_finis()" callback, So we need let those jobs go through the run_job() again, only except that it didn't need to submit to ring really , that way the job_finish() callback Can handle that job safely and seemless

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 18:52
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Am 26.10.2017 um 12:30 schrieb Monk Liu:

Quoted 63 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

v2:
some logic fix

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++--
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 54 ++++++++++++++++++++-------
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
  3 files changed, 47 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +6 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
  	return fence;
  }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
 *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
+*sched_job, bool skip)
  {
  	struct dma_fence *fence = NULL;
  	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
  	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
  	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
  	} else {
  		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
  				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -0
index 9cbeade..995661e 100644
@@ -330,6 +330,28 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
  	return false;
  }
 
+static void amd_sched_job_fake_signal(struct amd_sched_job *job) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 343

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +10 -0
+	struct amd_sched_fence *s_fence = job->s_fence;
+	struct amd_gpu_scheduler *sched = job->sched;
+	struct amd_sched_entity *entity = job->s_entity;
+
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 348

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+
+	/* fake signaling the finished fence */
+	job->s_entity = NULL;
+	spsc_queue_pop(&entity->job_queue);

That code isn't up to date any more, Andrey removed job->s_entity yesterday.

Please rebase on amd-staging-drm-next before resending.

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+
+	amd_sched_process_job(NULL, &s_fence->cb);
+	dma_fence_put(&s_fence->finished);
+	sched->ops->free_job(job);
+}
+
  static struct amd_sched_job *
  amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 373

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -0
@@ -344,6 +366,11 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  		if (amd_sched_entity_add_dependency_cb(entity))
  			return NULL;
 
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		return NULL;
+	}
+

Better move this after spsc_queue_pop below and remove the extra spsc_queue_pop from amd_sched_job_fake_signal.

Quoted 51 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -7
  	sched_job->s_entity = NULL;
  	spsc_queue_pop(&entity->job_queue);
  	return sched_job;
@@ -441,13 +468,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
  	job->sched->ops->timedout_job(job);
  }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job) -{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
  void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
  {
  	struct amd_sched_job *s_job;
@@ -466,7 +486,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  	}
  	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit) 
+{
  		bool found = false;
 
  		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ ) 

{ @@ -476,7 +496,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
if (bad->s_fence->scheduled.context == entity->fence_context) {
found = true;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -1
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
  					break;
  				}
  			}
@@ -499,6 +520,7 @@ void amd_sched_job_kickout(struct amd_sched_job *s_job)
  void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  {
  	struct amd_sched_job *s_job, *tmp;
+	bool found_guilty = false;
  	int r;
 
  	spin_lock(&sched->job_list_lock);
@@ -510,9 +532,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool skip;
+		uint64_t guilty_context;
+

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 546

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +7 -1
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == 
+guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

Regards,
Christian.

Quoted 34 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
  		atomic_inc(&sched->hw_rq_count);
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence); @@ -525,7 +555,6 @@ void 

amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
r);
dma_fence_put(fence);
} else {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -3
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
  		spin_lock(&sched->job_list_lock);
@@ -650,7 +679,7 @@ static int amd_sched_main(void *param)
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
  		amd_sched_fence_scheduled(s_fence);
 
  		if (fence) {
@@ -664,7 +693,6 @@ static int amd_sched_main(void *param)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
 
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr · line 132

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +2 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
  */
  struct amd_sched_backend_ops {
  	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool 
+skip);
  	void (*timedout_job)(struct amd_sched_job *sched_job);
  	void (*free_job)(struct amd_sched_job *sched_job);
  };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB04499876C7CC8573D9A0D2F484450-7LeqcoF/[email protected]

author
#8 lore raw

RE: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

The simple way I can think of to remove the @skip param for run_job is that we introduce a new member "skip" in sched_job, and remove fake_signal() function, So we always set "skip" in sched_job before run_job(), and in run_job() we skip the real ib_schedule if found job->skip == true,

That way the skipping logic can be unified, but that satisfies the efficiency because:

my current approach won't do begin_job(), won't link job in mirror list, etc.... but above approach actually go through all steps

what do you think

BR Monk

-----Original Message-----
From: Liu, Monk
Sent: 2017年10月26日 19:07
To: Koenig, Christian <[email protected]>; [email protected]
Subject: RE: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 344

Quoted 6 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

[ml] WHAT ??? this wake_up operate on job_scheduled, like the one at the bottom of sched_main(), It is used to wake up the thread waiting in "sched_entity_fini" on this "sched->job_scheduled", I don't understand why not necessary

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 545

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -1
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == 
+guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

[ML] no you still didn't get my point, in sched_main() we use fake_signal() because the job hadn't been linked in mirror list, so "job_finish()" Cannot be invoked for those jobs,

But for job_recovery(), it go through mirror list and only focus on jobs from that list, thus all jobs must be handled by "job_finis()" callback, So we need let those jobs go through the run_job() again, only except that it didn't need to submit to ring really , that way the job_finish() callback Can handle that job safely and seemless

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 18:52
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Am 26.10.2017 um 12:30 schrieb Monk Liu:

Quoted 63 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

v2:
some logic fix

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++--
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 54 ++++++++++++++++++++-------
  drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
  3 files changed, 47 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +6 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
  	return fence;
  }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
 *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job 
+*sched_job, bool skip)
  {
  	struct dma_fence *fence = NULL;
  	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
  	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
  	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
  		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
  	} else {
  		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
  				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -0
index 9cbeade..995661e 100644
@@ -330,6 +330,28 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
  	return false;
  }
 
+static void amd_sched_job_fake_signal(struct amd_sched_job *job) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 343

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +10 -0
+	struct amd_sched_fence *s_fence = job->s_fence;
+	struct amd_gpu_scheduler *sched = job->sched;
+	struct amd_sched_entity *entity = job->s_entity;
+
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 348

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+
+	/* fake signaling the finished fence */
+	job->s_entity = NULL;
+	spsc_queue_pop(&entity->job_queue);

That code isn't up to date any more, Andrey removed job->s_entity yesterday.

Please rebase on amd-staging-drm-next before resending.

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+
+	amd_sched_process_job(NULL, &s_fence->cb);
+	dma_fence_put(&s_fence->finished);
+	sched->ops->free_job(job);
+}
+
  static struct amd_sched_job *
  amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 373

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -0
@@ -344,6 +366,11 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
  		if (amd_sched_entity_add_dependency_cb(entity))
  			return NULL;
 
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		return NULL;
+	}
+

Better move this after spsc_queue_pop below and remove the extra spsc_queue_pop from amd_sched_job_fake_signal.

Quoted 51 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -7
  	sched_job->s_entity = NULL;
  	spsc_queue_pop(&entity->job_queue);
  	return sched_job;
@@ -441,13 +468,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
  	job->sched->ops->timedout_job(job);
  }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job) -{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
  void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
  {
  	struct amd_sched_job *s_job;
@@ -466,7 +486,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
  	}
  	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit) 
+{
  		bool found = false;
 
  		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ ) 

{ @@ -476,7 +496,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
if (bad->s_fence->scheduled.context == entity->fence_context) {
found = true;

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -1
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
  					break;
  				}
  			}
@@ -499,6 +520,7 @@ void amd_sched_job_kickout(struct amd_sched_job *s_job)
  void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  {
  	struct amd_sched_job *s_job, *tmp;
+	bool found_guilty = false;
  	int r;
 
  	spin_lock(&sched->job_list_lock);
@@ -510,9 +532,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
  	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
  		struct amd_sched_fence *s_fence = s_job->s_fence;
  		struct dma_fence *fence;
+		bool skip;
+		uint64_t guilty_context;
+

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 546

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +7 -1
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context == 
+guilty_context);
  		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

Regards,
Christian.

Quoted 34 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
  		atomic_inc(&sched->hw_rq_count);
  		if (fence) {
  			s_fence->parent = dma_fence_get(fence); @@ -525,7 +555,6 @@ void 

amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
r);
dma_fence_put(fence);
} else {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -3
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
  		spin_lock(&sched->job_list_lock);
@@ -650,7 +679,7 @@ static int amd_sched_main(void *param)
  		atomic_inc(&sched->hw_rq_count);
  		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
  		amd_sched_fence_scheduled(s_fence);
 
  		if (fence) {
@@ -664,7 +693,6 @@ static int amd_sched_main(void *param)
  					  r);
  			dma_fence_put(fence);
  		} else {
-			DRM_ERROR("Failed to run job!\n");
  			amd_sched_process_job(NULL, &s_fence->cb);
  		}
 
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr · line 132

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +2 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
  */
  struct amd_sched_backend_ops {
  	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool 
+skip);
  	void (*timedout_job)(struct amd_sched_job *sched_job);
  	void (*free_job)(struct amd_sched_job *sched_job);
  };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): BLUPR12MB0449F7595A55DC9ED73D9B0984450-7LeqcoF/[email protected]

#10 lore raw

Re: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Quoted 3 lines

Unnecessary, we know that the scheduler is already awake because it is processing this function.

[ml] WHAT ??? this wake_up operate on job_scheduled, like the one at the bottom of sched_main(), It is used to wake up the thread waiting in "sched_entity_fini" on this "sched->job_scheduled", I don't understand why not necessary

Ah! You not wake up the scheduler, but the waiter for the scheduler. In this case the order is incorrect, but let's focus on your other idea.

Quoted 2 lines

The simple way I can think of to remove the @skip param for run_job is that we introduce a new member "skip" in sched_job, and remove fake_signal() function, So we always set "skip" in sched_job before run_job(), and in run_job() we skip the real ib_schedule if found job->skip == true,

Cool idea. How about setting the fence error code? Then we test if the error code is already set and skip calling run_job() altogether when it is.

Quoted 1 line

my current approach won't do begin_job(), won't link job in mirror list, etc.... but above approach actually go through all steps

This is just for error recovery and doesn't need to be very efficient, but I see what you mean.

I think we should just keep one straight handling as much as possible and not to many corner cases. So your idea to skip the job when a flag is set sounds really good to me.

Regards,
Christian.

Am 26.10.2017 um 14:04 schrieb Liu, Monk:

Quoted 251 lines

The simple way I can think of to remove the @skip param for run_job is that we introduce a new member "skip" in sched_job, and remove fake_signal() function, So we always set "skip" in sched_job before run_job(), and in run_job() we skip the real ib_schedule if found job->skip == true,

That way the skipping logic can be unified, but that satisfies the efficiency because:

my current approach won't do begin_job(), won't link job in mirror list, etc.... but above approach actually go through all steps

what do you think

BR Monk

-----Original Message-----
From: Liu, Monk
Sent: 2017年10月26日 19:07
To: Koenig, Christian <[email protected]>; [email protected]
Subject: RE: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Review

Quoted 6 lines
(quoted diff) +6 -0
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

[ml] WHAT ??? this wake_up operate on job_scheduled, like the one at the bottom of sched_main(), It is used to wake up the thread waiting in "sched_entity_fini" on this "sched->job_scheduled", I don't understand why not necessary

Review

Quoted 8 lines
(quoted diff) +5 -1
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context ==
+guilty_context);
   		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

[ML] no you still didn't get my point, in sched_main() we use fake_signal() because the job hadn't been linked in mirror list, so "job_finish()" Cannot be invoked for those jobs,

But for job_recovery(), it go through mirror list and only focus on jobs from that list, thus all jobs must be handled by "job_finis()" callback, So we need let those jobs go through the run_job() again, only except that it didn't need to submit to ring really , that way the job_finish() callback Can handle that job safely and seemless

BR Monk

-----Original Message-----
From: Christian König [mailto:[email protected]]
Sent: 2017年10月26日 18:52
To: Liu, Monk <[email protected]>; [email protected]
Subject: Re: [PATCH 1/7] amd/scheduler:imple job skip feature(v2)

Am 26.10.2017 um 12:30 schrieb Monk Liu:

Quoted 63 lines

jobs are skipped under two cases
1)when the entity behind this job marked guilty, the job poped from this entity's queue will be dropped in sched_main loop.

2)in job_recovery(), skip the scheduling job if its karma detected above limit, and also skipped as well for other jobs sharing the same fence context. this approach is becuase job_recovery() cannot access job->entity due to entity may already dead.

v2:
some logic fix

with this feature we can introduce new gpu recover feature.

Change-Id I268b1c752c94e6ecd4ea78c87eb226ea3f52908a
---
   drivers/gpu/drm/amd/amdgpu/amdgpu_job.c       |  9 +++--
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.c | 54 ++++++++++++++++++++-------
   drivers/gpu/drm/amd/scheduler/gpu_scheduler.h |  2 +-
   3 files changed, 47 insertions(+), 18 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +6 -4
index a58e3c5..f08fde9 100644
@@ -177,7 +177,7 @@ static struct dma_fence *amdgpu_job_dependency(struct amd_sched_job *sched_job)
   	return fence;
   }
 
-static struct dma_fence *amdgpu_job_run(struct amd_sched_job
 *sched_job)
+static struct dma_fence *amdgpu_job_run(struct amd_sched_job
+*sched_job, bool skip)
   {
   	struct dma_fence *fence = NULL;
   	struct amdgpu_device *adev;
@@ -194,10 +194,11 @@ static struct dma_fence *amdgpu_job_run(struct amd_sched_job *sched_job)
   	BUG_ON(amdgpu_sync_peek_fence(&job->sync, NULL));
 
   	trace_amdgpu_sched_run_job(job);
-	/* skip ib schedule when vram is lost */
-	if (job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+
+	if (skip || job->vram_lost_counter != atomic_read(&adev->vram_lost_counter)) {
+		/* skip ib schedule if looks needed, and set error */
   		dma_fence_set_error(&job->base.s_fence->finished, -ECANCELED);
-		DRM_ERROR("Skip scheduling IBs!\n");
+		DRM_INFO("Skip scheduling IBs!\n");
   	} else {
   		r = amdgpu_ib_schedule(job->ring, job->num_ibs, job->ibs, job,
   				       &fence);
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -0
index 9cbeade..995661e 100644
@@ -330,6 +330,28 @@ static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity)
   	return false;
   }
 
+static void amd_sched_job_fake_signal(struct amd_sched_job *job) {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 343

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +10 -0
+	struct amd_sched_fence *s_fence = job->s_fence;
+	struct amd_gpu_scheduler *sched = job->sched;
+	struct amd_sched_entity *entity = job->s_entity;
+
+	dma_fence_set_error(&s_fence->finished, -ECANCELED);
+
+	/* fake signaling the scheduled fence */
+	atomic_inc(&sched->hw_rq_count);
+	amd_sched_fence_scheduled(s_fence);
+	wake_up(&sched->job_scheduled);

Unnecessary, we know that the scheduler is already awake because it is processing this function.

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c static bool amd_sched_entity_add_dependency_cb(struct amd_sched_entity *entity) · line 347

Quoted 4 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +4 -0
+
+	/* fake signaling the finished fence */
+	job->s_entity = NULL;
+	spsc_queue_pop(&entity->job_queue);

That code isn't up to date any more, Andrey removed job->s_entity yesterday.

Please rebase on amd-staging-drm-next before resending.

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -0
+
+	amd_sched_process_job(NULL, &s_fence->cb);
+	dma_fence_put(&s_fence->finished);
+	sched->ops->free_job(job);
+}
+
   static struct amd_sched_job *
   amd_sched_entity_pop_job(struct amd_sched_entity *entity)
   {

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c amd_sched_entity_pop_job(struct amd_sched_entity *entity) · line 373

Quoted 8 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +5 -0
@@ -344,6 +366,11 @@ amd_sched_entity_pop_job(struct amd_sched_entity *entity)
   		if (amd_sched_entity_add_dependency_cb(entity))
   			return NULL;
 
+	if (entity->guilty && atomic_read(entity->guilty)) {
+		amd_sched_job_fake_signal(sched_job);
+		return NULL;
+	}
+

Better move this after spsc_queue_pop below and remove the extra spsc_queue_pop from amd_sched_job_fake_signal.

Quoted 51 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +2 -7
   	sched_job->s_entity = NULL;
   	spsc_queue_pop(&entity->job_queue);
   	return sched_job;
@@ -441,13 +468,6 @@ static void amd_sched_job_timedout(struct work_struct *work)
   	job->sched->ops->timedout_job(job);
   }
 
-static void amd_sched_set_guilty(struct amd_sched_job *s_job) -{
-	if (atomic_inc_return(&s_job->karma) > s_job->sched->hang_limit)
-		if (s_job->s_entity->guilty)
-			atomic_set(s_job->s_entity->guilty, 1);
-}
-
   void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_job *bad)
   {
   	struct amd_sched_job *s_job;
@@ -466,7 +486,7 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo
   	}
   	spin_unlock(&sched->job_list_lock);
 
-	if (bad) {
+	if (bad && atomic_inc_return(&bad->karma) > bad->sched->hang_limit)
+{
   		bool found = false;
 
   		for (i = AMD_SCHED_PRIORITY_MIN; i < AMD_SCHED_PRIORITY_MAX; i++ )

{ @@ -476,7 +496,8 @@ void amd_sched_hw_job_reset(struct amd_gpu_scheduler *sched, struct amd_sched_jo

   			list_for_each_entry_safe(entity, tmp, &rq->entities, list) {
   				if (bad->s_fence->scheduled.context == entity->fence_context) {
   					found = true;
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +6 -1
-					amd_sched_set_guilty(bad);
+					if (entity->guilty)
+						atomic_set(entity->guilty, 1);
   					break;
   				}
   			}
@@ -499,6 +520,7 @@ void amd_sched_job_kickout(struct amd_sched_job *s_job)
   void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
   {
   	struct amd_sched_job *s_job, *tmp;
+	bool found_guilty = false;
   	int r;
 
   	spin_lock(&sched->job_list_lock);
@@ -510,9 +532,17 @@ void amd_sched_job_recovery(struct amd_gpu_scheduler *sched)
   	list_for_each_entry_safe(s_job, tmp, &sched->ring_mirror_list, node) {
   		struct amd_sched_fence *s_fence = s_job->s_fence;
   		struct dma_fence *fence;
+		bool skip;
+		uint64_t guilty_context;
+

drivers/gpu/drm/amd/scheduler/gpu_scheduler.c void amd_sched_job_recovery(struct amd_gpu_scheduler *sched) · line 546

Quoted 10 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +7 -1
+		if (!found_guilty && atomic_read(&s_job->karma) > sched->hang_limit) {
+			found_guilty = true;
+			guilty_context = s_job->s_fence->scheduled.context;
+		}
 
+		skip = (found_guilty && s_job->s_fence->scheduled.context ==
+guilty_context);
   		spin_unlock(&sched->job_list_lock);
-		fence = sched->ops->run_job(s_job);
+		fence = sched->ops->run_job(s_job, skip);

As far as I can see you can use amd_sched_job_fake_signal() here as well, no need for the extra skip parameter to run_job().

Regards,
Christian.

Quoted 28 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +0 -0
   		atomic_inc(&sched->hw_rq_count);
   		if (fence) {
   			s_fence->parent = dma_fence_get(fence); @@ -525,7 +555,6 @@ void

amd_sched_job_recovery(struct amd_gpu_scheduler *sched)

   					  r);
   			dma_fence_put(fence);
   		} else {
drivers/gpu/drm/amd/scheduler/gpu_scheduler.c +1 -3
-			DRM_ERROR("Failed to run job!\n");
   			amd_sched_process_job(NULL, &s_fence->cb);
   		}
   		spin_lock(&sched->job_list_lock);
@@ -650,7 +679,7 @@ static int amd_sched_main(void *param)
   		atomic_inc(&sched->hw_rq_count);
   		amd_sched_job_begin(sched_job);
 
-		fence = sched->ops->run_job(sched_job);
+		fence = sched->ops->run_job(sched_job, false);
   		amd_sched_fence_scheduled(s_fence);
 
   		if (fence) {
@@ -664,7 +693,6 @@ static int amd_sched_main(void *param)
   					  r);
   			dma_fence_put(fence);
   		} else {
-			DRM_ERROR("Failed to run job!\n");
   			amd_sched_process_job(NULL, &s_fence->cb);
   		}
 
Quoted 16 lines
Quoted 6 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +0 -0
index f9e3a83..a8e5a7d 100644

drivers/gpu/drm/amd/scheduler/gpu_scheduler.h static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr

Quoted 9 lines
drivers/gpu/drm/amd/scheduler/gpu_scheduler.h +2 -1
@@ -126,7 +126,7 @@ static inline bool amd_sched_invalidate_job(struct amd_sched_job *s_job, int thr
   */
   struct amd_sched_backend_ops {
   	struct dma_fence *(*dependency)(struct amd_sched_job *sched_job);
-	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job);
+	struct dma_fence *(*run_job)(struct amd_sched_job *sched_job, bool
+skip);
   	void (*timedout_job)(struct amd_sched_job *sched_job);
   	void (*free_job)(struct amd_sched_job *sched_job);
   };

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

author
#11 lore raw

PATCH 2/7 drm/amdgpu:implement new GPU recover(v3)

In reply to a message not in the archive

1,new imple names amdgpu_gpu_recover which gives more hint on what it does compared with gpu_reset

2,gpu_recover unify bare-metal and SR-IOV, only the asic reset part is implemented differently

3,gpu_recover will increase hang job karma and mark its entity/context as guilty if exceeds limit

V2:

4,in scheduler main routine the job from guilty context will be immedialy fake signaled after it poped from queue and its fence be set with "-ECANCELED" error

5,in scheduler recovery routine all jobs from the guilty entity would be dropped

6,in run_job() routine the real IB submission would be skipped if @skip parameter equales true or there was VRAM lost occured.

V3:

7,replace deprecated gpu reset, use new gpu recover

---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   6 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 310 +++++++++++++----------------
 drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c  |  10 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c    |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   5 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   |   1 -
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c      |   2 +-
 drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c      |   2 +-
 8 files changed, 151 insertions(+), 187 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +5 -1
index ba1ab97..335df11 100644
@@ -178,6 +178,10 @@ extern int amdgpu_cik_support;
 #define CIK_CURSOR_WIDTH 128
 #define CIK_CURSOR_HEIGHT 128
 
+/* GPU RESET flags */
+#define AMDGPU_RESET_INFO_VRAM_LOST  (1 << 0)
+#define AMDGPU_RESET_INFO_FULLRESET  (1 << 1)
+
 struct amdgpu_device;
 struct amdgpu_ib;
 struct amdgpu_cs_parser;
@@ -1840,7 +1844,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
 #define amdgpu_psp_check_fw_loading_status(adev, i) (adev)->firmware.funcs->check_fw_loading_status((adev), (i))
 
 /* Common functions */
-int amdgpu_gpu_reset(struct amdgpu_device *adev);
+int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job* job);
 bool amdgpu_need_backup(struct amdgpu_device *adev);
 void amdgpu_pci_config_reset(struct amdgpu_device *adev);
 bool amdgpu_need_post(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +137 -173

409 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c +5 -5
index f43319c..e64d93f 100644
@@ -694,25 +694,25 @@ static int amdgpu_debugfs_fence_info(struct seq_file *m, void *data)
 }
 
 /**
- * amdgpu_debugfs_gpu_reset - manually trigger a gpu reset
+ * amdgpu_debugfs_gpu_recover - manually trigger a gpu reset & recover
  *
  * Manually trigger a gpu reset at the next fence wait.
  */
-static int amdgpu_debugfs_gpu_reset(struct seq_file *m, void *data)
+static int amdgpu_debugfs_gpu_recover(struct seq_file *m, void *data)
 {
 	struct drm_info_node *node = (struct drm_info_node *) m->private;
 	struct drm_device *dev = node->minor->dev;
 	struct amdgpu_device *adev = dev->dev_private;
 
-	seq_printf(m, "gpu reset\n");
-	amdgpu_gpu_reset(adev);
+	seq_printf(m, "gpu recover\n");
+	amdgpu_gpu_recover(adev, NULL);
 
 	return 0;
 }
 
 static const struct drm_info_list amdgpu_debugfs_fence_list[] = {
 	{"amdgpu_fence_info", &amdgpu_debugfs_fence_info, 0, NULL},
-	{"amdgpu_gpu_reset", &amdgpu_debugfs_gpu_reset, 0, NULL}
+	{"amdgpu_gpu_recover", &amdgpu_debugfs_gpu_recover, 0, NULL}
 };
 
 static const struct drm_info_list amdgpu_debugfs_fence_list_sriov[] = {
drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c +1 -1
index 47c5ce9..9c2f87c 100644
@@ -88,7 +88,7 @@ static void amdgpu_irq_reset_work_func(struct work_struct *work)
 						  reset_work);
 
 	if (!amdgpu_sriov_vf(adev))
-		amdgpu_gpu_reset(adev);
+		amdgpu_gpu_recover(adev, NULL);
 }
 
 /* Disable *all* interrupts */
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -4
index f08fde9..ac6a4f3 100644
@@ -37,10 +37,7 @@ static void amdgpu_job_timedout(struct amd_sched_job *s_job)
 		  atomic_read(&job->ring->fence_drv.last_seq),
 		  job->ring->fence_drv.sync_seq);
 
-	if (amdgpu_sriov_vf(job->adev))
-		amdgpu_sriov_gpu_reset(job->adev, job);
-	else
-		amdgpu_gpu_reset(job->adev);
+	amdgpu_gpu_recover(job->adev, job);
 }
 
 int amdgpu_job_alloc(struct amdgpu_device *adev, unsigned num_ibs,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index b89d37f..3a661aa 100644
@@ -285,7 +285,6 @@ void amdgpu_virt_kiq_wreg(struct amdgpu_device *adev, uint32_t reg, uint32_t v);
 int amdgpu_virt_request_full_gpu(struct amdgpu_device *adev, bool init);
 int amdgpu_virt_release_full_gpu(struct amdgpu_device *adev, bool init);
 int amdgpu_virt_reset_gpu(struct amdgpu_device *adev);
-int amdgpu_sriov_gpu_reset(struct amdgpu_device *adev, struct amdgpu_job *job);
 int amdgpu_virt_alloc_mm_table(struct amdgpu_device *adev);
 void amdgpu_virt_free_mm_table(struct amdgpu_device *adev);
 int amdgpu_virt_fw_reserve_get_checksum(void *obj, unsigned long obj_size,
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +1 -1
index b4906d2..f8522a0 100644
@@ -254,7 +254,7 @@ static void xgpu_ai_mailbox_flr_work(struct work_struct *work)
 	}
 
 	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
 }
 
 static int xgpu_ai_set_mailbox_rcv_irq(struct amdgpu_device *adev,
drivers/gpu/drm/amd/amdgpu/mxgpu_vi.c +1 -1
index c25a831..dae6d3a 100644
@@ -514,7 +514,7 @@ static void xgpu_vi_mailbox_flr_work(struct work_struct *work)
 	}
 
 	/* Trigger recovery due to world switch failure */
-	amdgpu_sriov_gpu_reset(adev, NULL);
+	amdgpu_gpu_recover(adev, NULL);
 }
 
 static int xgpu_vi_set_mailbox_rcv_irq(struct amdgpu_device *adev,
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#12 lore raw

PATCH 3/7 drm/amdgpu:cleanup in_sriov_reset and lock_reset

In reply to a message not in the archive

since now gpu reset is unified with gpu_recover
for both bare-metal and SR-IOV:

1)rename in_sriov_reset to in_gpu_reset
2)move lock_reset from adev->virt to adev

Change-Id I9f4dbab9a4c916fbc156f669824d15ddcd0f2322
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 9 +++++----
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c    | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c  | 2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c   | 2 --
 drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h   | 1 -
 drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c      | 6 +++---
 drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c      | 6 +++---
 8 files changed, 15 insertions(+), 16 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +2 -1
index 335df11..6e89be5 100644
@@ -1650,7 +1650,8 @@ struct amdgpu_device {
 
 	/* record last mm index being written through WREG32*/
 	unsigned long last_mm_index;
-	bool                            in_sriov_reset;
+	bool                            in_gpu_reset;
+	struct mutex  lock_reset;
 };
 
 static inline struct amdgpu_device *amdgpu_ttm_adev(struct ttm_bo_device *bdev)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +5 -4
index a2f9a7f..4cf1146 100644
@@ -2161,6 +2161,7 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	mutex_init(&adev->mn_lock);
 	mutex_init(&adev->virt.vf_errors.lock);
 	hash_init(adev->mn_hash);
+	mutex_init(&adev->lock_reset);
 
 	amdgpu_check_arguments(adev);
 
@@ -2963,9 +2964,9 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
 
 	dev_info(adev->dev, "GPU reset begin!\n");
 
-	mutex_lock(&adev->virt.lock_reset);
+	mutex_lock(&adev->lock_reset);
 	atomic_inc(&adev->gpu_reset_counter);
-	adev->in_sriov_reset = 1;
+	adev->in_gpu_reset = 1;
 
 	/* block TTM */
 	resched = ttm_bo_lock_delayed_workqueue(&adev->mman.bdev);
@@ -3075,8 +3076,8 @@ int amdgpu_gpu_recover(struct amdgpu_device *adev, struct amdgpu_job *job)
 	}
 
 	amdgpu_vf_error_trans_all(adev);
-	adev->in_sriov_reset = 0;
-	mutex_unlock(&adev->virt.lock_reset);
+	adev->in_gpu_reset = 0;
+	mutex_unlock(&adev->lock_reset);
 	return r;
 }
 
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +1 -1
index 447d446..76f531b 100644
@@ -264,7 +264,7 @@ static int psp_hw_start(struct psp_context *psp)
 	struct amdgpu_device *adev = psp->adev;
 	int ret;
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
 		ret = psp_bootloader_load_sysdrv(psp);
 		if (ret)
 			return ret;
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +1 -1
index 6564902..edc37cc 100644
@@ -370,7 +370,7 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
 		return 0;
 	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_sriov_reset) {
+	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
 		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
 					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
 					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.c +0 -2
index e97f80f..c249725 100644
@@ -107,8 +107,6 @@ void amdgpu_virt_init_setting(struct amdgpu_device *adev)
 	adev->enable_virtual_display = true;
 	adev->cg_flags = 0;
 	adev->pg_flags = 0;
-
-	mutex_init(&adev->virt.lock_reset);
 }
 
 uint32_t amdgpu_virt_kiq_rreg(struct amdgpu_device *adev, uint32_t reg)
drivers/gpu/drm/amd/amdgpu/amdgpu_virt.h +0 -1
index 3a661aa..a710384 100644
@@ -238,7 +238,6 @@ struct amdgpu_virt {
 	uint64_t			csa_vmid0_addr;
 	bool chained_ib_support;
 	uint32_t			reg_val_offs;
-	struct mutex                    lock_reset;
 	struct amdgpu_irq_src		ack_irq;
 	struct amdgpu_irq_src		rcv_irq;
 	struct work_struct		flr_work;
drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c +3 -3
index b8002ac..72ddb6e 100644
@@ -4807,7 +4807,7 @@ static int gfx_v8_0_kiq_init_queue(struct amdgpu_ring *ring)
 
 	gfx_v8_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
@@ -4844,7 +4844,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
 	struct vi_mqd *mqd = ring->mqd_ptr;
 	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
 		memset((void *)mqd, 0, sizeof(struct vi_mqd_allocation));
 		((struct vi_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
 		((struct vi_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;
@@ -4856,7 +4856,7 @@ static int gfx_v8_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct vi_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct vi_mqd_allocation));
drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +3 -3
index 8738b13..d598d78 100644
@@ -2732,7 +2732,7 @@ static int gfx_v9_0_kiq_init_queue(struct amdgpu_ring *ring)
 
 	gfx_v9_0_kiq_setting(ring);
 
-	if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));
@@ -2770,7 +2770,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 	struct v9_mqd *mqd = ring->mqd_ptr;
 	int mqd_idx = ring - &adev->gfx.compute_ring[0];
 
-	if (!adev->in_sriov_reset && !adev->gfx.in_suspend) {
+	if (!adev->in_gpu_reset && !adev->gfx.in_suspend) {
 		memset((void *)mqd, 0, sizeof(struct v9_mqd_allocation));
 		((struct v9_mqd_allocation *)mqd)->dynamic_cu_mask = 0xFFFFFFFF;
 		((struct v9_mqd_allocation *)mqd)->dynamic_rb_mask = 0xFFFFFFFF;
@@ -2782,7 +2782,7 @@ static int gfx_v9_0_kcq_init_queue(struct amdgpu_ring *ring)
 
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(adev->gfx.mec.mqd_backup[mqd_idx], mqd, sizeof(struct v9_mqd_allocation));
-	} else if (adev->in_sriov_reset) { /* for GPU_RESET case */
+	} else if (adev->in_gpu_reset) { /* for GPU_RESET case */
 		/* reset MQD to a clean status */
 		if (adev->gfx.mec.mqd_backup[mqd_idx])
 			memcpy(mqd, adev->gfx.mec.mqd_backup[mqd_idx], sizeof(struct v9_mqd_allocation));
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#13 lore raw

PATCH 4/7 drm/amdgpu:cleanup ucode_init_bo

In reply to a message not in the archive

1,no sriov check since gpu recover is unified 2,need CPU_ACCESS_REQUIRED flag for VRAM if SRIOV because otherwise after following PIN the first allocated VRAM bo is wasted due to some TTM mgr reason.

Change-Id I4d029f2da8bb463942c7861d3e52f309bdba9576
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ucode.c +2 -2
index edc37cc..ab9b2d4 100644
@@ -370,10 +370,10 @@ int amdgpu_ucode_init_bo(struct amdgpu_device *adev)
 		return 0;
 	}
 
-	if (!amdgpu_sriov_vf(adev) || !adev->in_gpu_reset) {
+	if (!adev->in_gpu_reset) {
 		err = amdgpu_bo_create(adev, adev->firmware.fw_size, PAGE_SIZE, true,
 					amdgpu_sriov_vf(adev) ? AMDGPU_GEM_DOMAIN_VRAM : AMDGPU_GEM_DOMAIN_GTT,
-					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS,
+					AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS|AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED,
 					NULL, NULL, 0, bo);
 		if (err) {
 			dev_err(adev->dev, "(%d) Firmware buffer allocate failed\n", err);
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#14 lore raw

PATCH 5/7 drm/amdgpu:block kms open during gpu_reset

In reply to a message not in the archive
Change-Id Ibdb0ea9e3769d572fbbc13bbf1ef73f1af2ab7be
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +3 -0
index 4a9f749..c155ce4 100644
@@ -813,6 +813,9 @@ int amdgpu_driver_open_kms(struct drm_device *dev, struct drm_file *file_priv)
 	if (r < 0)
 		return r;
 
+	if (adev->in_gpu_reset)
+		return -ENODEV;
+
 	fpriv = kzalloc(sizeof(*fpriv), GFP_KERNEL);
 	if (unlikely(!fpriv)) {
 		r = -ENOMEM;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#15 lore raw

PATCH 6/7 drm/amdgpu/sriov:fix memory leak in psp_load_fw

In reply to a message not in the archive

for SR-IOV when doing gpu reset this routine shouldn't do resource allocating otherwise memory leak

Change-Id I25da3a5b475196c75c7e639adc40751754625968
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +12 -8
index 76f531b..2157d45 100644
@@ -334,23 +334,26 @@ static int psp_load_fw(struct amdgpu_device *adev)
 	int ret;
 	struct psp_context *psp = &adev->psp;
 
+	if (amdgpu_sriov_vf(adev) && adev->in_gpu_reset != 0)
+		goto skip_memalloc;
+
 	psp->cmd = kzalloc(sizeof(struct psp_gfx_cmd_resp), GFP_KERNEL);
 	if (!psp->cmd)
 		return -ENOMEM;
 
 	ret = amdgpu_bo_create_kernel(adev, PSP_1_MEG, PSP_1_MEG,
-				      AMDGPU_GEM_DOMAIN_GTT,
-				      &psp->fw_pri_bo,
-				      &psp->fw_pri_mc_addr,
-				      &psp->fw_pri_buf);
+					AMDGPU_GEM_DOMAIN_GTT,
+					&psp->fw_pri_bo,
+					&psp->fw_pri_mc_addr,
+					&psp->fw_pri_buf);
 	if (ret)
 		goto failed;
 
 	ret = amdgpu_bo_create_kernel(adev, PSP_FENCE_BUFFER_SIZE, PAGE_SIZE,
-				      AMDGPU_GEM_DOMAIN_VRAM,
-				      &psp->fence_buf_bo,
-				      &psp->fence_buf_mc_addr,
-				      &psp->fence_buf);
+					AMDGPU_GEM_DOMAIN_VRAM,
+					&psp->fence_buf_bo,
+					&psp->fence_buf_mc_addr,
+					&psp->fence_buf);
 	if (ret)
 		goto failed_mem2;
 
@@ -375,6 +378,7 @@ static int psp_load_fw(struct amdgpu_device *adev)
 	if (ret)
 		goto failed_mem;
 
+skip_memalloc:
 	ret = psp_hw_start(psp);
 	if (ret)
 		goto failed_mem;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
author
#16 lore raw

PATCH 7/7 drm/amdgpu:fix random missing of FLR NOTIFY

In reply to a message not in the archive
---
 drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/amdgpu/mxgpu_ai.c +11 -3
index f8522a0..a43cffb 100644
@@ -282,9 +282,17 @@ static int xgpu_ai_mailbox_rcv_irq(struct amdgpu_device *adev,
 		/* see what event we get */
 		r = xgpu_ai_mailbox_rcv_msg(adev, IDH_FLR_NOTIFICATION);
 
-		/* only handle FLR_NOTIFY now */
-		if (!r)
-			schedule_work(&adev->virt.flr_work);
+		/* sometimes the interrupt is delayed to inject to VM, so under such case
+		 * the IDH_FLR_NOTIFICATION is overwritten by VF FLR from GIM side, thus
+		 * above recieve message could be failed, we should schedule the flr_work
+		 * anyway
+		 */
+		if (r) {
+			DRM_ERROR("FLR_NOTIFICATION is missed\n");
+			xgpu_ai_mailbox_send_ack(adev);
+		}
+
+		schedule_work(&adev->virt.flr_work);
 	}
 
 	return 0;
Signature · 6 lines
2.7.4

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx