Skip to contentSkip to conversation

GART table recovery

PATCH In review Chunming Zhou 2 participants 18 messages Aug 2, 2016 – Aug 18, 2016 6 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover GART table recovery

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

 drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 +++++++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
 9 files changed, 304 insertions(+), 66 deletions(-)
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 1470124840-26170-1-git-send-email-David1.Zhou-5C7GfCeVMHo@public.gmane.org

#2 lore raw

PATCH 1/10 drm/amdgpu: make need_backup generic

It will be used other place.

Change-Id I213faf16e25a95bef4c45a65ab21f4d61db4ef41
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h    | 1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index b1a4af0..daf07ff 100644
@@ -975,6 +975,7 @@ void amdgpu_vm_manager_init(struct amdgpu_device *adev);
 void amdgpu_vm_manager_fini(struct amdgpu_device *adev);
 int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm);
 void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm);
+bool amdgpu_vm_need_backup(struct amdgpu_device *adev);
 void amdgpu_vm_get_pd_bo(struct amdgpu_vm *vm,
 			 struct list_head *validated,
 			 struct amdgpu_bo_list_entry *entry);
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +1 -1
index a34d94a..01dd888 100644
@@ -112,7 +112,7 @@ void amdgpu_vm_get_pd_bo(struct amdgpu_vm *vm,
 	list_add(&entry->tv.head, validated);
 }
 
-static bool amdgpu_vm_need_backup(struct amdgpu_device *adev)
+bool amdgpu_vm_need_backup(struct amdgpu_device *adev)
 {
 	if (adev->flags & AMD_IS_APU)
 		return false;
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#3 lore raw

PATCH 2/10 drm/amdgpu: implement gart late_init/fini

In reply to a message not in the archive

add recovery entity to gart.

Change-Id Ieb400c8a731ef25619ea3c0b5198a6e7ce56580e
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |  3 +++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 18 ++++++++++++++++++
 2 files changed, 21 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
index daf07ff..419a33b 100644
@@ -646,6 +646,7 @@ struct amdgpu_gart {
 #endif
 	bool				ready;
 	const struct amdgpu_gart_funcs *gart_funcs;
+	struct amd_sched_entity         recover_entity;
 };
 
 int amdgpu_gart_table_ram_alloc(struct amdgpu_device *adev);
@@ -656,6 +657,8 @@ int amdgpu_gart_table_vram_pin(struct amdgpu_device *adev);
 void amdgpu_gart_table_vram_unpin(struct amdgpu_device *adev);
 int amdgpu_gart_init(struct amdgpu_device *adev);
 void amdgpu_gart_fini(struct amdgpu_device *adev);
+int amdgpu_gart_late_init(struct amdgpu_device *adev);
+void amdgpu_gart_late_fini(struct amdgpu_device *adev);
 void amdgpu_gart_unbind(struct amdgpu_device *adev, unsigned offset,
 			int pages);
 int amdgpu_gart_bind(struct amdgpu_device *adev, unsigned offset,
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +18 -0
index 921bce2..c1f226b 100644
@@ -363,3 +363,21 @@ void amdgpu_gart_fini(struct amdgpu_device *adev)
 #endif
 	amdgpu_dummy_page_fini(adev);
 }
+
+int amdgpu_gart_late_init(struct amdgpu_device *adev)
+{
+	struct amd_sched_rq *rq;
+	struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+
+	rq = &ring->sched.sched_rq[AMD_SCHED_PRIORITY_RECOVER];
+	return amd_sched_entity_init(&ring->sched, &adev->gart.recover_entity,
+				     rq, amdgpu_sched_jobs);
+
+}
+
+void amdgpu_gart_late_fini(struct amdgpu_device *adev)
+{
+	struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+
+	amd_sched_entity_fini(&ring->sched, &adev->gart.recover_entity);
+}
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#4 lore raw

PATCH 3/10 drm/amdgpu: add gart_late_init/fini to gmc V7/8

In reply to a message not in the archive
Change-Id I47b132d1ac5ed57f5805f759d5698948c35721ba
---
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c | 24 ++++++++++++++++++++----
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c | 24 ++++++++++++++++++++----
 2 files changed, 40 insertions(+), 8 deletions(-)
drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +20 -4
index 0b0f086..0771c04 100644
@@ -887,11 +887,26 @@ static int gmc_v7_0_early_init(void *handle)
 static int gmc_v7_0_late_init(void *handle)
 {
 	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	int r;
 
-	if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS)
-		return amdgpu_irq_get(adev, &adev->mc.vm_fault, 0);
-	else
-		return 0;
+	r = amdgpu_gart_late_init(adev);
+	if (r)
+		return r;
+
+	if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS) {
+		r = amdgpu_irq_get(adev, &adev->mc.vm_fault, 0);
+		if (r)
+			amdgpu_gart_late_fini(adev);
+	}
+
+	return r;
+}
+
+static void gmc_v7_0_late_fini(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+	amdgpu_gart_late_fini(adev);
 }
 
 static int gmc_v7_0_sw_init(void *handle)
@@ -1242,6 +1257,7 @@ const struct amd_ip_funcs gmc_v7_0_ip_funcs = {
 	.name = "gmc_v7_0",
 	.early_init = gmc_v7_0_early_init,
 	.late_init = gmc_v7_0_late_init,
+	.late_fini = gmc_v7_0_late_fini,
 	.sw_init = gmc_v7_0_sw_init,
 	.sw_fini = gmc_v7_0_sw_fini,
 	.hw_init = gmc_v7_0_hw_init,
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +20 -4
index 0a23b83..c26bee9 100644
@@ -876,11 +876,26 @@ static int gmc_v8_0_early_init(void *handle)
 static int gmc_v8_0_late_init(void *handle)
 {
 	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+	int r;
 
-	if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS)
-		return amdgpu_irq_get(adev, &adev->mc.vm_fault, 0);
-	else
-		return 0;
+	r = amdgpu_gart_late_init(adev);
+	if (r)
+		return r;
+
+	if (amdgpu_vm_fault_stop != AMDGPU_VM_FAULT_STOP_ALWAYS) {
+		r = amdgpu_irq_get(adev, &adev->mc.vm_fault, 0);
+		if (r)
+			amdgpu_gart_late_fini(adev);
+	}
+
+	return r;
+}
+
+static void gmc_v8_0_late_fini(void *handle)
+{
+	struct amdgpu_device *adev = (struct amdgpu_device *)handle;
+
+	amdgpu_gart_late_fini(adev);
 }
 
 #define mmMC_SEQ_MISC0_FIJI 0xA71
@@ -1434,6 +1449,7 @@ const struct amd_ip_funcs gmc_v8_0_ip_funcs = {
 	.name = "gmc_v8_0",
 	.early_init = gmc_v8_0_early_init,
 	.late_init = gmc_v8_0_late_init,
+	.late_fini = gmc_v8_0_late_fini,
 	.sw_init = gmc_v8_0_sw_init,
 	.sw_fini = gmc_v8_0_sw_fini,
 	.hw_init = gmc_v8_0_hw_init,
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#5 lore raw

PATCH 4/10 drm/amdgpu: abstract amdgpu_bo_create_shadow

In reply to a message not in the archive
Change-Id Id0e89f350a05f8668ea00e3fff8c0bd6f3049cec
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 40 ++++++++++++++++++++----------
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  3 +++
 2 files changed, 30 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +27 -13
index e6ecf16..c1111c9 100644
@@ -380,6 +380,32 @@ fail_free:
 	return r;
 }
 
+int amdgpu_bo_create_shadow(struct amdgpu_device *adev,
+			    unsigned long size, int byte_align,
+			    struct amdgpu_bo *bo)
+{
+	struct ttm_placement placement = {0};
+	struct ttm_place placements[AMDGPU_GEM_DOMAIN_MAX + 1];
+
+	if (bo->shadow)
+		return 0;
+
+	bo->flags |= AMDGPU_GEM_CREATE_SHADOW;
+	memset(&placements, 0,
+	       (AMDGPU_GEM_DOMAIN_MAX + 1) * sizeof(struct ttm_place));
+
+	amdgpu_ttm_placement_init(adev, &placement,
+				  placements, AMDGPU_GEM_DOMAIN_GTT,
+				  AMDGPU_GEM_CREATE_CPU_GTT_USWC);
+
+	return amdgpu_bo_create_restricted(adev, size, byte_align, true,
+					   AMDGPU_GEM_DOMAIN_GTT,
+					   AMDGPU_GEM_CREATE_CPU_GTT_USWC,
+					   NULL, &placement,
+					   bo->tbo.resv,
+					   &bo->shadow);
+}
+
 int amdgpu_bo_create(struct amdgpu_device *adev,
 		     unsigned long size, int byte_align,
 		     bool kernel, u32 domain, u64 flags,
@@ -404,19 +430,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 		return r;
 
 	if (flags & AMDGPU_GEM_CREATE_SHADOW) {
-		memset(&placements, 0,
-		       (AMDGPU_GEM_DOMAIN_MAX + 1) * sizeof(struct ttm_place));
-
-		amdgpu_ttm_placement_init(adev, &placement,
-					  placements, AMDGPU_GEM_DOMAIN_GTT,
-					  AMDGPU_GEM_CREATE_CPU_GTT_USWC);
-
-		r = amdgpu_bo_create_restricted(adev, size, byte_align, kernel,
-						AMDGPU_GEM_DOMAIN_GTT,
-						AMDGPU_GEM_CREATE_CPU_GTT_USWC,
-						NULL, &placement,
-						(*bo_ptr)->tbo.resv,
-						&(*bo_ptr)->shadow);
+		r = amdgpu_bo_create_shadow(adev, size, byte_align, (*bo_ptr));
 		if (r)
 			amdgpu_bo_unref(bo_ptr);
 	} else
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +3 -0
index d650b42..b994fd4 100644
@@ -117,6 +117,9 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 			    struct sg_table *sg,
 			    struct reservation_object *resv,
 			    struct amdgpu_bo **bo_ptr);
+int amdgpu_bo_create_shadow(struct amdgpu_device *adev,
+			    unsigned long size, int byte_align,
+			    struct amdgpu_bo *bo);
 int amdgpu_bo_create_restricted(struct amdgpu_device *adev,
 				unsigned long size, int byte_align,
 				bool kernel, u32 domain, u64 flags,
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#6 lore raw

PATCH 5/10 drm/amdgpu: shadow gart table support

In reply to a message not in the archive

allocate gart shadow bo, and using shadow bo to backup gart table.

Change-Id Ib2beae9cea1ad1314c57f0fcdcc254816f39b9b2
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |  3 ++
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 47 +++++++++++++++++++++++++++++++-
 drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c    | 15 ++++++++++
 drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c    | 16 +++++++++++
 4 files changed, 80 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +3 -0
index 419a33b..2985578d 100644
@@ -638,6 +638,7 @@ struct amdgpu_gart {
 	dma_addr_t			table_addr;
 	struct amdgpu_bo		*robj;
 	void				*ptr;
+	void				*shadow_ptr;
 	unsigned			num_gpu_pages;
 	unsigned			num_cpu_pages;
 	unsigned			table_size;
@@ -655,6 +656,8 @@ int amdgpu_gart_table_vram_alloc(struct amdgpu_device *adev);
 void amdgpu_gart_table_vram_free(struct amdgpu_device *adev);
 int amdgpu_gart_table_vram_pin(struct amdgpu_device *adev);
 void amdgpu_gart_table_vram_unpin(struct amdgpu_device *adev);
+int amdgpu_gart_table_vram_shadow_pin(struct amdgpu_device *adev);
+void amdgpu_gart_table_vram_shadow_unpin(struct amdgpu_device *adev);
 int amdgpu_gart_init(struct amdgpu_device *adev);
 void amdgpu_gart_fini(struct amdgpu_device *adev);
 int amdgpu_gart_late_init(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +46 -1

71 lines not on this page · raw message

drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c +15 -0
index 0771c04..5470a28 100644
@@ -589,7 +589,21 @@ static int gmc_v7_0_gart_enable(struct amdgpu_device *adev)
 		 (unsigned)(adev->mc.gtt_size >> 20),
 		 (unsigned long long)adev->gart.table_addr);
 	adev->gart.ready = true;
+	if (amdgpu_vm_need_backup(adev) && adev->gart.robj) {
+		r = amdgpu_bo_create_shadow(adev, adev->gart.table_size,
+					    PAGE_SIZE, adev->gart.robj);
+		if (r)
+			goto err;
+		r = amdgpu_gart_table_vram_shadow_pin(adev);
+		if (r)
+			goto err;
+	}
+
 	return 0;
+err:
+	amdgpu_gart_table_vram_unpin(adev);
+
+	return r;
 }
 
 static int gmc_v7_0_gart_init(struct amdgpu_device *adev)
@@ -634,6 +648,7 @@ static void gmc_v7_0_gart_disable(struct amdgpu_device *adev)
 	WREG32(mmVM_L2_CNTL, tmp);
 	WREG32(mmVM_L2_CNTL2, 0);
 	amdgpu_gart_table_vram_unpin(adev);
+	amdgpu_gart_table_vram_shadow_unpin(adev);
 }
 
 /**
drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +16 -0
index c26bee9..6c2b5de 100644
@@ -704,7 +704,22 @@ static int gmc_v8_0_gart_enable(struct amdgpu_device *adev)
 		 (unsigned)(adev->mc.gtt_size >> 20),
 		 (unsigned long long)adev->gart.table_addr);
 	adev->gart.ready = true;
+
+	if (amdgpu_vm_need_backup(adev) && adev->gart.robj) {
+		r = amdgpu_bo_create_shadow(adev, adev->gart.table_size,
+					    PAGE_SIZE, adev->gart.robj);
+		if (r)
+			goto err;
+		r = amdgpu_gart_table_vram_shadow_pin(adev);
+		if (r)
+			goto err;
+	}
+
 	return 0;
+err:
+	amdgpu_gart_table_vram_unpin(adev);
+
+	return r;
 }
 
 static int gmc_v8_0_gart_init(struct amdgpu_device *adev)
@@ -749,6 +764,7 @@ static void gmc_v8_0_gart_disable(struct amdgpu_device *adev)
 	WREG32(mmVM_L2_CNTL, tmp);
 	WREG32(mmVM_L2_CNTL2, 0);
 	amdgpu_gart_table_vram_unpin(adev);
+	amdgpu_gart_table_vram_shadow_unpin(adev);
 }
 
 /**
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#7 lore raw

PATCH 6/10 drm/amdgpu: make recover_bo_from_shadow be generic

In reply to a message not in the archive
Change-Id I74758b9ca84058f3f2db5509822d8aad840d283e
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.c | 40 +++++++++++++++++++++++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |  6 ++++
 drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     | 48 ++++--------------------------
 3 files changed, 51 insertions(+), 43 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +40 -0
index c1111c9..1d0bdfb 100644
@@ -439,6 +439,46 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
 	return r;
 }
 
+int amdgpu_bo_recover_bo_from_shadow(struct amdgpu_device *adev,
+				     struct amdgpu_ring *ring,
+				     struct amd_sched_entity *entity,
+				     struct amdgpu_bo *bo,
+				     struct reservation_object *resv,
+				     struct fence **fence)
+
+{
+	int r;
+	uint64_t vram_addr, gtt_addr;
+
+	r = amdgpu_bo_pin(bo, AMDGPU_GEM_DOMAIN_VRAM, &vram_addr);
+	if (r) {
+		DRM_ERROR("Failed to pin bo object\n");
+		goto err1;
+	}
+	r = amdgpu_bo_pin(bo->shadow, AMDGPU_GEM_DOMAIN_GTT, &gtt_addr);
+	if (r) {
+		DRM_ERROR("Failed to pin bo shadow object\n");
+		goto err2;
+	}
+
+	r = reservation_object_reserve_shared(bo->tbo.resv);
+	if (r)
+		goto err3;
+
+	r = amdgpu_copy_buffer(ring, entity, gtt_addr,
+			       vram_addr, amdgpu_bo_size(bo), resv, fence);
+	if (!r)
+		amdgpu_bo_fence(bo, *fence, true);
+
+err3:
+	amdgpu_bo_unpin(bo->shadow);
+err2:
+	amdgpu_bo_unpin(bo);
+err1:
+
+	return r;
+}
+
 int amdgpu_bo_kmap(struct amdgpu_bo *bo, void **ptr)
 {
 	bool is_iomem;
drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +6 -0
index b994fd4..f35fd68 100644
@@ -158,6 +158,12 @@ int amdgpu_bo_fault_reserve_notify(struct ttm_buffer_object *bo);
 void amdgpu_bo_fence(struct amdgpu_bo *bo, struct fence *fence,
 		     bool shared);
 u64 amdgpu_bo_gpu_offset(struct amdgpu_bo *bo);
+int amdgpu_bo_recover_bo_from_shadow(struct amdgpu_device *adev,
+				     struct amdgpu_ring *ring,
+				     struct amd_sched_entity *entity,
+				     struct amdgpu_bo *bo,
+				     struct reservation_object *resv,
+				     struct fence **fence);
 
 /*
  * sub allocation
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +5 -43

66 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#8 lore raw

PATCH 7/10 drm/amdgpu: implement gart recovery

In reply to a message not in the archive

recover gart bo from its shadow bo.

Change-Id Idbb91d62b1c3cf73f7d90b5f2c650f2690e5a42b
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 31 +++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 2985578d..3ee01fe 100644
@@ -667,6 +667,7 @@ void amdgpu_gart_unbind(struct amdgpu_device *adev, unsigned offset,
 int amdgpu_gart_bind(struct amdgpu_device *adev, unsigned offset,
 		     int pages, struct page **pagelist,
 		     dma_addr_t *dma_addr, uint32_t flags);
+int amdgpu_gart_table_recover_from_shadow(struct amdgpu_device *adev);
 
 /*
  * GPU MC structures, functions & helpers
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +31 -0
index b306684..baeaee2 100644
@@ -135,6 +135,37 @@ int amdgpu_gart_table_vram_alloc(struct amdgpu_device *adev)
 	return 0;
 }
 
+int amdgpu_gart_table_recover_from_shadow(struct amdgpu_device *adev)
+{
+	struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring;
+	struct fence *fence;
+	int r;
+
+	if (!amdgpu_vm_need_backup(adev))
+		return 0;
+	/* bo and shadow use same resv, so reserve one time */
+	r = amdgpu_bo_reserve(adev->gart.robj, false);
+	if (unlikely(r != 0))
+		return r;
+
+	r = amdgpu_bo_recover_bo_from_shadow(adev, ring,
+					     &adev->gart.recover_entity,
+					     adev->gart.robj,
+					     NULL, &fence);
+	amdgpu_bo_unreserve(adev->gart.robj);
+	if (r) {
+		DRM_ERROR("recover page table failed!\n");
+		goto err;
+	}
+
+	if (fence)
+		r = fence_wait(fence, false);
+	fence_put(fence);
+
+err:
+	return r;
+}
+
 /**
  * amdgpu_gart_table_vram_pin - pin gart page table in vram
  *
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#9 lore raw

PATCH 8/10 drm/amdgpu: recover gart table first when full reset

In reply to a message not in the archive
Change-Id Iad7a90646dbb5df930a8ba177ce6bdc48415ff7d
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +2 -0
index b7b4cf8..16ba37d 100644
@@ -2191,6 +2191,8 @@ retry:
 				kthread_unpark(ring->sched.thread);
 				unpark_bits |= 1 << ring->idx;
 			}
+			DRM_INFO("recover gart table first\n");
+			amdgpu_gart_table_recover_from_shadow(adev);
 
 			spin_lock(&adev->vm_list_lock);
 			list_for_each_entry_safe(vm, tmp, &adev->vm_list, list) {
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#10 lore raw

PATCH 9/10 drm/amdgpu: sync gart table before initialization completed

In reply to a message not in the archive

Since the shadow is in GTT, shadow itslef pte isn't in shadow, We need to do sync before initialization is completed

Change-Id I29b433da6c71fc790a32ef202dd85a72ab6b5787
---
 drivers/gpu/drm/amd/amdgpu/amdgpu.h      |  1 +
 drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c | 47 +++++++++++++++++++++++++++++++-
 2 files changed, 47 insertions(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
index 3ee01fe..4cad4b2 100644
@@ -648,6 +648,7 @@ struct amdgpu_gart {
 	bool				ready;
 	const struct amdgpu_gart_funcs *gart_funcs;
 	struct amd_sched_entity         recover_entity;
+	u64                             shadow_gpu_addr;
 };
 
 int amdgpu_gart_table_ram_alloc(struct amdgpu_device *adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c +46 -1

66 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#11 lore raw

PATCH 10/10 drm/amdgpu: fix memory leak of sched fence

In reply to a message not in the archive

amdgpu_job_free_resources is already called by submit_job.
If it is called in run_job, the sched fence could be got twice in sa bo free, then memory leak happens.

Change-Id I833612e31cf22b62174f3f76546fd11c9ea38780
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +1 -1
index 8d87a9a..d56247d 100644
@@ -183,7 +183,7 @@ static struct fence *amdgpu_job_run(struct amd_sched_job *sched_job)
 	/* if gpu reset, hw fence will be replaced here */
 	fence_put(job->fence);
 	job->fence = fence_get(fence);
-	amdgpu_job_free_resources(job);
+
 	return fence;
 }
 
Signature · 6 lines
1.9.1

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
#12 lore raw
In reply to a message not in the archive

Well you have been hardworking during my vacation :)

Looks pretty good to me, but hope that I can get a closer look tomorrow.

Is there any particular order the three sets must be applied?

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 11 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 +++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): f1b6c786-7e9c-ff61-1de9-299bc4daed15-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author
#14 lore raw
Quoted 5 lines · Christian König

Well you have been hardworking during my vacation :)

Looks pretty good to me, but hope that I can get a closer look tomorrow.

Is there any particular order the three sets must be applied?

they are depending on my development order:
1. [PATCH 00/13] shadow page table support
2. [PATCH 00/11] add recovery entity and run queue
3. [PATCH 00/10] GART table recovery

Thanks,
David Zhou

Quoted 34 lines

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 28 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 
+++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

#15 lore raw
In reply to a message not in the archive

Well patch #10 is incorrect. The SA BO will be set to NULL by amdgpu_sa_bo_free(), so it can't be freed twice and so you can't reference the fence twice.

Additional to that the whole approach here of restoring the GART from the backup using the SDMA won't work either. For the SDMA to work you need the GART to access the ring buffer.

So you run into a chicken and egg problem here, for the ring buffer to work you need the GART and for the GART backup to work you need the ring buffer.

We should just restore the GART content from the housekeeping structure instead. Going to evaluate if and how that might be possible.

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 11 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 +++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 54bb3255-2dda-f6ad-3682-8e4396ec932a-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author
#17 lore raw
Quoted 1 line · Christian König

Well patch #10 is incorrect. The SA BO will be set to NULL by amdgpu_sa_bo_free(), so it can't be freed twice and so you can't reference the fence twice.

I see.
But amdgpu_job_free_resources still shouldn't be called twice, right? That's an obvious duplication although it seems no effect now. Is there any other reason?

Quoted 4 lines

Additional to that the whole approach here of restoring the GART from the backup using the SDMA won't work either. For the SDMA to work you need the GART to access the ring buffer.

So you run into a chicken and egg problem here, for the ring buffer to work you need the GART and for the GART backup to work you need the ring buffer.

Good catch, ring buffer is a GTT buffer as well.

Then Can we use memcpy to copy GTT to VRAM? Fortunately, the GART bo is only one bo.

Regards,
David Zhou

Quoted 37 lines

We should just restore the GART content from the housekeeping structure instead. Going to evaluate if and how that might be possible.

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 12 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 
+++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)
Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#19 lore raw

Am 04.08.2016 um 05:35 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

Well patch #10 is incorrect. The SA BO will be set to NULL by amdgpu_sa_bo_free(), so it can't be freed twice and so you can't reference the fence twice.

I see.
But amdgpu_job_free_resources still shouldn't be called twice, right? That's an obvious duplication although it seems no effect now. Is there any other reason?

It's actually called from a couple of different locations:
1. From the CS path in amdgpu_cs.c as soon as we have a scheduler fence.
2. From the amdgpu_job_submit() path as soon as we have a scheduler fence.
3. From amdgpu_job_run() after submitting the job to the hardware ring.
4. From amdgpu_job_free(), this is for direct submissions or for freeing the job when something went wrong.

Thinking about it you could be right and we could probably drop the one in amdgpu_job_run(), because amdgpu_job_submit() should have already taken care of that. But I'm not 100% sure of that.

Quoted 8 lines
Quoted 5 lines

Additional to that the whole approach here of restoring the GART from the backup using the SDMA won't work either. For the SDMA to work you need the GART to access the ring buffer.

So you run into a chicken and egg problem here, for the ring buffer to work you need the GART and for the GART backup to work you need the ring buffer.

Good catch, ring buffer is a GTT buffer as well.

Then Can we use memcpy to copy GTT to VRAM? Fortunately, the GART bo is only one bo.

Yeah that is what we did with radeon as well. Unfortunately the double housekeeping costs quite a bunch of memory.

And actually we have the exactly same information in the TTM MM as well, we would just need to bind all BOs again.

Give me a day or two to double check that. Might be that the solution is rather simple.

Regards,
Christian.

Quoted 46 lines

Regards,
David Zhou

Quoted 41 lines

We should just restore the GART content from the housekeeping structure instead. Going to evaluate if and how that might be possible.

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 12 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 
+++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): 077bb11d-957d-c6f2-2f87-248fbc19304a-ANTagKRnAhcb1SvskN2V4Q@public.gmane.org

author
#21 lore raw
Quoted 23 lines · Christian König

Am 04.08.2016 um 05:35 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

Well patch #10 is incorrect. The SA BO will be set to NULL by amdgpu_sa_bo_free(), so it can't be freed twice and so you can't reference the fence twice.

I see.
But amdgpu_job_free_resources still shouldn't be called twice, right? That's an obvious duplication although it seems no effect now. Is there any other reason?

It's actually called from a couple of different locations:
1. From the CS path in amdgpu_cs.c as soon as we have a scheduler fence.
2. From the amdgpu_job_submit() path as soon as we have a scheduler fence.
3. From amdgpu_job_run() after submitting the job to the hardware ring.
4. From amdgpu_job_free(), this is for direct submissions or for freeing the job when something went wrong.

Thinking about it you could be right and we could probably drop the one in amdgpu_job_run(), because amdgpu_job_submit() should have already taken care of that. But I'm not 100% sure of that.

Quoted 7 lines
Quoted 5 lines

Additional to that the whole approach here of restoring the GART from the backup using the SDMA won't work either. For the SDMA to work you need the GART to access the ring buffer.

So you run into a chicken and egg problem here, for the ring buffer to work you need the GART and for the GART backup to work you need the ring buffer.

Good catch, ring buffer is a GTT buffer as well.

Quoted 7 lines
Quoted 1 line

Then Can we use memcpy to copy GTT to VRAM? Fortunately, the GART bo is only one bo.

Yeah that is what we did with radeon as well. Unfortunately the double housekeeping costs quite a bunch of memory.

And actually we have the exactly same information in the TTM MM as well, we would just need to bind all BOs again.

Give me a day or two to double check that. Might be that the solution is rather simple.

How about this? Do you have any better idea for it? or just change ring buffer to VRAM bo?

Regards,
David Zhou

Quoted 51 lines

Regards,
Christian.

Quoted 46 lines

Regards,
David Zhou

Quoted 41 lines

We should just restore the GART content from the housekeeping structure instead. Going to evaluate if and how that might be possible.

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 12 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 
+++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

A message that is not in the archive (sent off-list or not yet received): [email protected]

#23 lore raw

Am 18.08.2016 um 10:50 schrieb zhoucm1:

Quoted 26 lines · Christian König
Quoted 23 lines

Am 04.08.2016 um 05:35 schrieb zhoucm1:

Quoted 6 lines · Christian König
Quoted 1 line

Well patch #10 is incorrect. The SA BO will be set to NULL by amdgpu_sa_bo_free(), so it can't be freed twice and so you can't reference the fence twice.

I see.
But amdgpu_job_free_resources still shouldn't be called twice, right? That's an obvious duplication although it seems no effect now. Is there any other reason?

It's actually called from a couple of different locations:
1. From the CS path in amdgpu_cs.c as soon as we have a scheduler fence.
2. From the amdgpu_job_submit() path as soon as we have a scheduler fence.
3. From amdgpu_job_run() after submitting the job to the hardware ring.
4. From amdgpu_job_free(), this is for direct submissions or for freeing the job when something went wrong.

Thinking about it you could be right and we could probably drop the one in amdgpu_job_run(), because amdgpu_job_submit() should have already taken care of that. But I'm not 100% sure of that.

Quoted 7 lines
Quoted 5 lines

Additional to that the whole approach here of restoring the GART from the backup using the SDMA won't work either. For the SDMA to work you need the GART to access the ring buffer.

So you run into a chicken and egg problem here, for the ring buffer to work you need the GART and for the GART backup to work you need the ring buffer.

Good catch, ring buffer is a GTT buffer as well.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

Then Can we use memcpy to copy GTT to VRAM? Fortunately, the GART bo is only one bo.

Yeah that is what we did with radeon as well. Unfortunately the double housekeeping costs quite a bunch of memory.

And actually we have the exactly same information in the TTM MM as well, we would just need to bind all BOs again.

Give me a day or two to double check that. Might be that the solution is rather simple.

How about this? Do you have any better idea for it?

Sorry for not answering earlier, but you know humans have only one head and two hands to type :)

The informations needed to restore the GART is already stored in the amdgpu_ttm_tt structures.

We just need to link them together in amdgpu_ttm_backend_bind() and unlink them in amdgpu_ttm_backend_unbind() to be able to restore the GART table after a reset.

Quoted 1 line

or just change ring buffer to VRAM bo?

Interesting idea, but considering how much overhead it has to write to VRAM from the CPU it clearly wouldn't be such a good idea to do in general.

Regards,
Christian.

Quoted 55 lines

Regards,
David Zhou

Quoted 51 lines

Regards,
Christian.

Quoted 46 lines

Regards,
David Zhou

Quoted 41 lines

We should just restore the GART content from the housekeeping structure instead. Going to evaluate if and how that might be possible.

Regards,
Christian.

Am 02.08.2016 um 10:00 schrieb Chunming Zhou:

Quoted 16 lines

gart table is stored in one bo which must be ready before gart init, but the shadow bo must be created after gart is ready, so they cannot be created at a same time. shado bo itself aslo is included in gart table, So shadow bo needs a synchronization after device init. After sync, the contents of bo and shadwo bo will be same, and be updated at a same time. Then we will be able to recover gart table from shadow bo when gpu full reset.

patch10 is a fix for memory leak.

Chunming Zhou (10):
drm/amdgpu: make need_backup generic
drm/amdgpu: implement gart late_init/fini drm/amdgpu: add gart_late_init/fini to gmc V7/8 drm/amdgpu: abstract amdgpu_bo_create_shadow drm/amdgpu: shadow gart table support
drm/amdgpu: make recover_bo_from_shadow be generic drm/amdgpu: implement gart recovery
drm/amdgpu: recover gart table first when full reset drm/amdgpu: sync gart table before initialization completed drm/amdgpu: fix memory leak of sched fence

Quoted 12 lines
  drivers/gpu/drm/amd/amdgpu/amdgpu.h        |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c |   2 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_gart.c   | 139 
+++++++++++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_job.c    |   2 +-
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |  80 ++++++++++++++---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.h |   9 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c     |  50 ++---------
  drivers/gpu/drm/amd/amdgpu/gmc_v7_0.c      |  39 +++++++-
  drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c      |  40 ++++++++-
  9 files changed, 304 insertions(+), 66 deletions(-)

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 4 lines

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx