Skip to contentSkip to conversation

shadow page table support V6 ---> shadow bo support

PATCH In review Chunming Zhou 2 participants 9 messages Aug 18, 2016 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover shadow page table support V6 ---> shadow bo support

Since we cannot ensure VRAM is consistent after a GPU reset, page table shadowing is necessary. Shadowed page tables are, in a sense, a method to recover the consistent state of the page tables before the reset occurred.

We need to allocate GTT bo as the shadow of VRAM bo when creating page table, and make them the same. After gpu reset, we will need to use SDMA to copy GTT bo content to VRAM bo, then page table will be recoveried.

V2:
Shadow bo uses a shadow entity running on normal run queue, after gpu reset, we need to wait for all shadow jobs finished first, then recovery page table from shadow.

V3:
Addressed Christian comments for shadow bo part.

V4:
Switch back to update page table twice (one of two is for shadow)

V5:
make it be gerneic shadow bo support. Address Christian comments.

V6:
some improvements addressed comments from Christian.

Chunming Zhou (7):
drm/amdgpu: add need backup function V2 drm/amdgpu: add direct submision option for copy_buffer drm/amdgpu: sync bo and shadow V3
drm/amdgpu: update pd shadow while updating pd V2 drm/amdgpu: update pt shadow while updating pt V2 drm/amdgpu: link all shadow bo V2
drm/amdgpu: recover vram bo from shadow after gpu reset

 drivers/gpu/drm/amd/amdgpu/amdgpu.h           | 17 +++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c |  3 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c    | 71 +++++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c    | 78 ++++++++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h    | 12 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_test.c      |  4 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 23 +++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c        | 76 +++++++++++++++++++-------
 8 files changed, 242 insertions(+), 42 deletions(-)
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1471497513-23347-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#2 lore raw

PATCH 1/7 drm/amdgpu: add need backup function V2

V2:
add checking if need backup in amdgpu_bo_create.

Change-Id I76e2552078c9ae11b7f2a1769025230f61733659
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 8 ++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 2 +-
 3 files changed, 10 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index a996f4d..38ed9bd 100644
@@ -2445,6 +2445,7 @@ amdgpu_get_sdma_instance(struct amdgpu_ring *ring)
 
 /* Common functions */
 int amdgpu_gpu_reset(struct amdgpu_device *adev);
+bool amdgpu_need_backup(struct amdgpu_device *adev);
 void amdgpu_pci_config_reset(struct amdgpu_device *adev);
 bool amdgpu_card_posted(struct amdgpu_device *adev);
 void amdgpu_update_display_priority(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +8 -0
index 5376361..027dfb3 100644
@@ -2109,6 +2109,14 @@ static int amdgpu_post_soft_reset(struct amdgpu_device *adev)
 	return 0;
 }
 
+bool amdgpu_need_backup(struct amdgpu_device *adev)
+{
+	if (adev->flags & AMD_IS_APU)
+		return false;
+
+	return amdgpu_lockup_timeout > 0 ? true : false;
+}
+
 /**
  * amdgpu_gpu_reset - reset the asic
  *
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +1 -1
index 278017d..2b3aec3 100644
@@ -434,7 +434,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 	if (r)
 		return r;
 
-	if (flags & AMDGPU_GEM_CREATE_SHADOW) {
+	if (amdgpu_need_backup(adev) && (flags & AMDGPU_GEM_CREATE_SHADOW)) {
 		r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
 		if (r)
 			amdgpu_bo_unref(bo_ptr);
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#3 lore raw

PATCH 2/7 drm/amdgpu: add direct submision option for copy_buffer

In reply to a message not in the archive
Change-Id I3acdcf24b0ccb86d5f2b0593ffcf3dad8b6ad29f
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h           |  2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c |  3 ++-
 drivers/gpu/drm/amd/amdgpu/amdgpu_test.c      |  4 ++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c       | 23 ++++++++++++++++-------
 4 files changed, 21 insertions(+), 11 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -1
index 38ed9bd..6a16e29 100644
@@ -437,7 +437,7 @@ int amdgpu_copy_buffer(struct amdgpu_ring *ring,
 		       uint64_t dst_offset,
 		       uint32_t byte_count,
 		       struct reservation_object *resv,
-		       struct fence **fence);
+		       struct fence **fence, bool direct_submit);
 int amdgpu_fill_buffer(struct amdgpu_bo *bo,
 			uint32_t src_data,
 			struct reservation_object *resv,
drivers/gpu/drm/amd/amdgpu/amdgpu_benchmark.c +2 -1
index 33e47a4..3453052 100644
@@ -39,7 +39,8 @@ static int amdgpu_benchmark_do_move(struct amdgpu_device *adev, unsigned size,
 	start_jiffies = jiffies;
 	for (i = 0; i < n; i++) {
 		struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
-		r = amdgpu_copy_buffer(ring, saddr, daddr, size, NULL, &fence);
+		r = amdgpu_copy_buffer(ring, saddr, daddr, size, NULL, &fence,
+				       false);
 		if (r)
 			goto exit_do_move;
 		r = fence_wait(fence, false);
drivers/gpu/drm/amd/amdgpu/amdgpu_test.c +2 -2
index 05a53f4..b827c75 100644
@@ -111,7 +111,7 @@ static void amdgpu_do_test_moves(struct amdgpu_device *adev)
 		amdgpu_bo_kunmap(gtt_obj[i]);
 
 		r = amdgpu_copy_buffer(ring, gtt_addr, vram_addr,
-				       size, NULL, &fence);
+				       size, NULL, &fence, false);
 
 		if (r) {
 			DRM_ERROR("Failed GTT->VRAM copy %d\n", i);
@@ -156,7 +156,7 @@ static void amdgpu_do_test_moves(struct amdgpu_device *adev)
 		amdgpu_bo_kunmap(vram_obj);
 
 		r = amdgpu_copy_buffer(ring, vram_addr, gtt_addr,
-				       size, NULL, &fence);
+				       size, NULL, &fence, false);
 
 		if (r) {
 			DRM_ERROR("Failed VRAM->GTT copy %d\n", i);
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +16 -7
index 105bd22..58e3bd6 100644
@@ -286,7 +286,7 @@ static int amdgpu_move_blit(struct ttm_buffer_object *bo,
 
 	r = amdgpu_copy_buffer(ring, old_start, new_start,
 			       new_mem->num_pages * PAGE_SIZE, /* bytes */
-			       bo->resv, &fence);
+			       bo->resv, &fence, false);
 	if (r)
 		return r;
 
@@ -1148,7 +1148,7 @@ int amdgpu_copy_buffer(struct amdgpu_ring *ring,
 		       uint64_t dst_offset,
 		       uint32_t byte_count,
 		       struct reservation_object *resv,
-		       struct fence **fence)
+		       struct fence **fence, bool direct_submit)
 {
 	struct amdgpu_device *adev = ring->adev;
 	struct amdgpu_job *job;
@@ -1192,12 +1192,21 @@ int amdgpu_copy_buffer(struct amdgpu_ring *ring,
 
 	amdgpu_ring_pad_ib(ring, &job->ibs[0]);
 	WARN_ON(job->ibs[0].length_dw > num_dw);
-	r = amdgpu_job_submit(job, ring, &adev->mman.entity,
-			      AMDGPU_FENCE_OWNER_UNDEFINED, fence);
-	if (r)
-		goto error_free;
+	if (direct_submit) {
+		r = amdgpu_ib_schedule(ring, job->num_ibs, job->ibs,
+				       NULL, NULL, fence);
+		job->fence = fence_get(*fence);
+		if (r)
+			DRM_ERROR("Error scheduling IBs (%d)\n", r);
+		amdgpu_job_free(job);
+	} else {
+		r = amdgpu_job_submit(job, ring, &adev->mman.entity,
+				      AMDGPU_FENCE_OWNER_UNDEFINED, fence);
+		if (r)
+			goto error_free;
+	}
 
-	return 0;
+	return r;
 
 error_free:
 	amdgpu_job_free(job);
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#4 lore raw

PATCH 3/7 drm/amdgpu: sync bo and shadow V3

In reply to a message not in the archive

Use shadow flag to judge which direction to sync.
V2:
Don't need bo pin, so remove it.

V3:
1. Split to two functions, one is backup_to_shadow, another is restore_from_shadow.
2. Clean up previous shadow direction difinitions.

Change-Id I9b540970d3a24c6aebeaa94c99f66a89134c663d
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  8 ----
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 64 ++++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 12 ++++++
 3 files changed, 76 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +0 -8
index 6a16e29..15b7cf8 100644
@@ -482,12 +482,6 @@ struct amdgpu_bo_va {
 
 #define AMDGPU_GEM_DOMAIN_MAX		0x3
 
-enum amdgpu_bo_shadow {
-	AMDGPU_BO_SHADOW_TO_NONE = 0,
-	AMDGPU_BO_SHADOW_TO_PARENT,
-	AMDGPU_BO_SHADOW_TO_SHADOW,
-};
-
 struct amdgpu_bo {
 	/* Protected by gem.mutex */
 	struct list_head		list;
@@ -514,8 +508,6 @@ struct amdgpu_bo {
 	struct drm_gem_object		gem_base;
 	struct amdgpu_bo		*parent;
 	struct amdgpu_bo		*shadow;
-	/* indicate if need to sync between bo and shadow */
-	enum amdgpu_bo_shadow           backup_shadow;
 
 	struct ttm_bo_kmap_obj		dma_buf_vmap;
 	struct amdgpu_mn		*mn;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +64 -0

70 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +12 -0
index d650b42..b6a2739 100644
@@ -155,6 +155,18 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo);
 void amdgpu_bo_fence(struct amdgpu_bo *bo, struct fence *fence,
 		     bool shared);
 u64 amdgpu_bo_gpu_offset(struct amdgpu_bo *bo);
+int amdgpu_bo_backup_to_shadow(struct amdgpu_device *adev,
+			       struct amdgpu_ring *ring,
+			       struct amdgpu_bo *bo,
+			       struct reservation_object *resv,
+			       struct fence **fence, bool direct);
+int amdgpu_bo_restore_from_shadow(struct amdgpu_device *adev,
+				  struct amdgpu_ring *ring,
+				  struct amdgpu_bo *bo,
+				  struct reservation_object *resv,
+				  struct fence **fence,
+				  bool direct);
+
 
 /*
  * sub allocation
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#5 lore raw

PATCH 4/7 drm/amdgpu: update pd shadow while updating pd V2

In reply to a message not in the archive

V2:
Checking if shadow is valid.

Change-Id Ief33f9539c24e1ab696935e2bfedd3630c07fc67
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h    |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 60 +++++++++++++++++++++++-----------
 2 files changed, 42 insertions(+), 19 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 15b7cf8..8fa2dde 100644
@@ -874,6 +874,7 @@ struct amdgpu_ring {
 struct amdgpu_vm_pt {
 	struct amdgpu_bo_list_entry	entry;
 	uint64_t			addr;
+	uint64_t			shadow_addr;
 };
 
 struct amdgpu_vm {
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +41 -19

86 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#6 lore raw

PATCH 5/7 drm/amdgpu: update pt shadow while updating pt V2

In reply to a message not in the archive

V2:
move shadow parameter to amdgpu_pte_update_params.

Change-Id Ibc8fb4c5c9be38934ebd6d56f1cbd49cb82d53af
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +15 -1
index faf6cba..6c0028a 100644
@@ -66,6 +66,8 @@ struct amdgpu_pte_update_params {
 	dma_addr_t *pages_addr;
 	/* indirect buffer to fill with commands */
 	struct amdgpu_ib *ib;
+	/* indicate upodate pt or its shadow */
+	bool shadow;
 };
 
 /**
@@ -752,7 +754,11 @@ static void amdgpu_vm_update_ptes(struct amdgpu_pte_update_params *params,
 	addr = start;
 	pt_idx = addr >> amdgpu_vm_block_size;
 	pt = vm->page_tables[pt_idx].entry.robj;
-
+	if (params->shadow) {
+		if (!pt->shadow)
+			return;
+		pt = vm->page_tables[pt_idx].entry.robj->shadow;
+	}
 	if ((addr & ~mask) == (end & ~mask))
 		nptes = end - addr;
 	else
@@ -771,6 +777,11 @@ static void amdgpu_vm_update_ptes(struct amdgpu_pte_update_params *params,
 	while (addr < end) {
 		pt_idx = addr >> amdgpu_vm_block_size;
 		pt = vm->page_tables[pt_idx].entry.robj;
+		if (params->shadow) {
+			if (!pt->shadow)
+				return;
+			pt = vm->page_tables[pt_idx].entry.robj->shadow;
+		}
 
 		if ((addr & ~mask) == (end & ~mask))
 			nptes = end - addr;
@@ -970,6 +981,9 @@ static int amdgpu_vm_bo_update_mapping(struct amdgpu_device *adev,
 	if (r)
 		goto error_free;
 
+	params.shadow = true;
+	amdgpu_vm_frag_ptes(&params, vm, start, last + 1, addr, flags);
+	params.shadow = false;
 	amdgpu_vm_frag_ptes(&params, vm, start, last + 1, addr, flags);
 
 	amdgpu_ring_pad_ib(ring, params.ib);
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#7 lore raw

PATCH 6/7 drm/amdgpu: link all shadow bo V2

In reply to a message not in the archive

V2:
1. use mutex instead of spinlock for shadow list, since its process could sleep.
2. move list_del to bo destroy phase.

Change-Id I8af3b42e1b5d84ca04277f95aeca44cbc534ccb2
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |  5 +++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |  3 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 12 +++++++++++-
 3 files changed, 19 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +5 -0
index 8fa2dde..85d8b16 100644
@@ -512,6 +512,7 @@ struct amdgpu_bo {
 	struct ttm_bo_kmap_obj		dma_buf_vmap;
 	struct amdgpu_mn		*mn;
 	struct list_head		mn_list;
+	struct list_head		shadow_list;
 };
 #define gem_to_amdgpu_bo(gobj) container_of((gobj), struct amdgpu_bo, gem_base)
 
@@ -2163,6 +2164,10 @@ struct amdgpu_device {
 	struct kfd_dev          *kfd;
 
 	struct amdgpu_virtualization virtualization;
+
+	/* link all shadow bo */
+	struct list_head                shadow_list;
+	struct mutex                    shadow_list_lock;
 };
 
 bool amdgpu_device_is_px(struct drm_device *dev);
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3 -0
index 027dfb3..c9b15c0 100644
@@ -1588,6 +1588,9 @@ int amdgpu_device_init(struct amdgpu_device *adev,
 	spin_lock_init(&adev->gc_cac_idx_lock);
 	spin_lock_init(&adev->audio_endpt_idx_lock);
 
+	INIT_LIST_HEAD(&adev->shadow_list);
+	mutex_init(&adev->shadow_list_lock);
+
 	adev->rmmio_base = pci_resource_start(adev->pdev, 5);
 	adev->rmmio_size = pci_resource_len(adev->pdev, 5);
 	adev->rmmio = ioremap(adev->rmmio_base, adev->rmmio_size);
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +11 -1
index 18e82f0..0b8e4b8 100644
@@ -99,6 +99,11 @@ static void amdgpu_ttm_bo_destroy(struct ttm_buffer_object *tbo)
 
 	drm_gem_object_release(&bo->gem_base);
 	amdgpu_bo_unref(&bo->parent);
+	if (!list_empty(&bo->shadow_list)) {
+		mutex_lock(&bo->adev->shadow_list_lock);
+		list_del_init(&bo->shadow_list);
+		mutex_unlock(&bo->adev->shadow_list_lock);
+	}
 	kfree(bo->metadata);
 	kfree(bo);
 }
@@ -313,6 +318,7 @@ int amdgpu_bo_create_restricted(struct amdgpu_device *adev,
 	}
 	bo->adev = adev;
 	INIT_LIST_HEAD(&bo->list);
+	INIT_LIST_HEAD(&bo->shadow_list);
 	INIT_LIST_HEAD(&bo->va);
 	bo->prefered_domains = domain & (AMDGPU_GEM_DOMAIN_VRAM |
 					 AMDGPU_GEM_DOMAIN_GTT |
@@ -405,8 +411,12 @@ static int amdgpu_bo_create_shadow(struct amdgpu_device *adev,
 					NULL, &placement,
 					bo->tbo.resv,
 					&bo->shadow);
-	if (!r)
+	if (!r) {
 		bo->shadow->parent = amdgpu_bo_ref(bo);
+		mutex_lock(&adev->shadow_list_lock);
+		list_add_tail(&bo->shadow_list, &adev->shadow_list);
+		mutex_unlock(&adev->shadow_list_lock);
+	}
 
 	return r;
 }
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#8 lore raw

PATCH 7/7 drm/amdgpu: recover vram bo from shadow after gpu reset

In reply to a message not in the archive
Change-Id I963598ba6eb44bc8620d70e026c0175d1a1de120
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 60 +++++++++++++++++++++++++++++-
 1 file changed, 59 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +59 -1
index c9b15c0..c68d93b 100644
@@ -2120,6 +2120,40 @@ bool amdgpu_need_backup(struct amdgpu_device *adev)
 	return amdgpu_lockup_timeout > 0 ? true : false;
 }
 
+static int amdgpu_recover_vram_from_shadow(struct amdgpu_device *adev,
+					   struct amdgpu_ring *ring,
+					   struct amdgpu_bo *bo,
+					   struct fence **fence)
+{
+	struct fence *f;
+	uint32_t domain;
+	int r;
+
+       if (!bo->shadow)
+               return 0;
+
+       r = amdgpu_bo_reserve(bo, false);
+       if (r)
+               return r;
+       domain = amdgpu_mem_type_to_domain(bo->tbo.mem.mem_type);
+       /* if bo has been evicted, then no need to recover */
+       if (domain == AMDGPU_GEM_DOMAIN_VRAM) {
+               r = amdgpu_bo_restore_from_shadow(adev, ring, bo,
+						 NULL, &f, true);
+               if (r) {
+                       DRM_ERROR("recover page table failed!\n");
+                       goto err;
+               }
+               if (fence) {
+                       fence_put(*fence);
+                       *fence = fence_get(f);
+               }
+       }
+err:
+       amdgpu_bo_unreserve(bo);
+       return r;
+}
+
 /**
  * amdgpu_gpu_reset - reset the asic
  *
@@ -2202,13 +2236,37 @@ retry:
 		if (r) {
 			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
 			r = amdgpu_suspend(adev);
+			need_full_reset = true;
 			goto retry;
 		}
-
+		/**
+		 * recovery vm page tables, since we cannot depend on VRAM is
+		 * consistent after gpu full reset.
+		 */
+		if (need_full_reset && amdgpu_need_backup(adev)) {
+			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+			struct amdgpu_bo *bo, *tmp;
+			struct fence *fence = NULL;
+
+			DRM_INFO("recover vram bo from shadow\n");
+			mutex_lock(&adev->shadow_list_lock);
+			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
+				amdgpu_recover_vram_from_shadow(adev, ring, bo, &fence);
+			}
+			mutex_unlock(&adev->shadow_list_lock);
+			if (fence) {
+				r = fence_wait(fence, true);
+				if (r)
+					WARN(r, "recovery from shadow isn't comleted\n");
+			}
+			fence_put(fence);
+		}
 		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
 			struct amdgpu_ring *ring = adev->rings[i];
 			if (!ring)
 				continue;
+
+			DRM_INFO("ring:%d recover jobs\n", ring->idx);
 			amd_sched_job_recovery(&ring->sched);
 			kthread_unpark(ring->sched.thread);
 		}
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1471497513-23347-8-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#10 lore raw

Re: [PATCH 7/7] drm/amdgpu: recover vram bo from shadow after gpu reset

Patch #1-#6 are Reviewed-by: Christian König <[email protected]>

A few comments on this one:

Am 18.08.2016 um 07:18 schrieb Chunming Zhou:

Quoted 62 lines
Change-Id I963598ba6eb44bc8620d70e026c0175d1a1de120
---
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 60 +++++++++++++++++++++++++++++-
  1 file changed, 59 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +38 -1
index c9b15c0..c68d93b 100644
@@ -2120,6 +2120,40 @@ bool amdgpu_need_backup(struct amdgpu_device *adev)
  	return amdgpu_lockup_timeout > 0 ? true : false;
  }
 
+static int amdgpu_recover_vram_from_shadow(struct amdgpu_device *adev,
+					   struct amdgpu_ring *ring,
+					   struct amdgpu_bo *bo,
+					   struct fence **fence)
+{
+	struct fence *f;
+	uint32_t domain;
+	int r;
+
+       if (!bo->shadow)
+               return 0;
+
+       r = amdgpu_bo_reserve(bo, false);
+       if (r)
+               return r;
+       domain = amdgpu_mem_type_to_domain(bo->tbo.mem.mem_type);
+       /* if bo has been evicted, then no need to recover */
+       if (domain == AMDGPU_GEM_DOMAIN_VRAM) {
+               r = amdgpu_bo_restore_from_shadow(adev, ring, bo,
+						 NULL, &f, true);
+               if (r) {
+                       DRM_ERROR("recover page table failed!\n");
+                       goto err;
+               }
+               if (fence) {
+                       fence_put(*fence);
+                       *fence = fence_get(f);
+               }
+       }
+err:
+       amdgpu_bo_unreserve(bo);
+       return r;
+}
+
  /**
   * amdgpu_gpu_reset - reset the asic
   *
@@ -2202,13 +2236,37 @@ retry:
  		if (r) {
  			dev_err(adev->dev, "ib ring test failed (%d).\n", r);
  			r = amdgpu_suspend(adev);
+			need_full_reset = true;
  			goto retry;
  		}
-
+		/**
+		 * recovery vm page tables, since we cannot depend on VRAM is
+		 * consistent after gpu full reset.

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c retry: · line 2254

Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +10 -0
+		 */
+		if (need_full_reset && amdgpu_need_backup(adev)) {
+			struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+			struct amdgpu_bo *bo, *tmp;
+			struct fence *fence = NULL;
+
+			DRM_INFO("recover vram bo from shadow\n");
+			mutex_lock(&adev->shadow_list_lock);
+			list_for_each_entry_safe(bo, tmp, &adev->shadow_list, shadow_list) {
+				amdgpu_recover_vram_from_shadow(adev, ring, bo, &fence);

I think we need to make sure we don't directly submit to many jobs at the same time here.

E.g. do something like

struct amdgpu_fence *fence = NULL, *next;

amdgpu_recover_vram_from_shadow(adev, ring, bo, &next);

if (fence)

    fence_wait(fence, false);

fence_put(fence);
fence = next;

This way we always have a maximum of two IBs in flight.

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c retry: · line 2258

Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +4 -0
+			}
+			mutex_unlock(&adev->shadow_list_lock);
+			if (fence) {
+				r = fence_wait(fence, true);

The kernel thread executing this should never receive a signal, but just in case I suggest to not wait interruptible here.

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c retry: · line 2269

Quoted 11 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +7 -0
+				if (r)
+					WARN(r, "recovery from shadow isn't comleted\n");
+			}
+			fence_put(fence);
+		}
  		for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
  			struct amdgpu_ring *ring = adev->rings[i];
  			if (!ring)
  				continue;
+
+			DRM_INFO("ring:%d recover jobs\n", ring->idx);

This change looks unrelated and should probably not be part of this patch.

Regards,
Christian.

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c retry: · line 2272

Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0
  			amd_sched_job_recovery(&ring->sched);
  			kthread_unpark(ring->sched.thread);
  		}

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx