Skip to contentSkip to conversation

drm/i915: vma/ppgtt lifetime rules

PATCH In review Michel Thierry 4 participants 7 messages Jul 29, 2014 – Jul 30, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 drm/i915: vma/ppgtt lifetime rules

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +8 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);
+
 	list_del(&vma->vma_link);
 
 	kfree(vma);
drivers/gpu/drm/i915/i915_gem_context.c +3 -20
index 5b5af6c..59272f9 100644
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this shouldn't be
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
drivers/gpu/drm/i915/i915_gem_gtt.c +5 -0
index 1411613..90c3d0f 100644
@@ -2159,10 +2159,15 @@ i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }
Signature · 1 line
1.9.0
#1 lore raw
Quoted 43 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +2 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);

drivers/gpu/drm/i915/i915_gem.c struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj, · line 4514

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +5 -0
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);

Hm, this has the risk that we leave unwanted vmas around for a bit longer. They will get cleaned up though when the final object references goes away, so the leak is fairly restricted. And currently we don't have a shrinker to just whack out vma objects even ...

It's definitely a much neater solution than what I had in mind with moving vmas to full-blown active tracking like we do objects. So I'm tempted to go with, but have a bit a lurking feeling that I'm missing something.

Chris?

Cheers, Daniel

Quoted 69 lines
drivers/gpu/drm/i915/i915_gem.c +1 -0
+
 	list_del(&vma->vma_link);
 
 	kfree(vma);
drivers/gpu/drm/i915/i915_gem_context.c +3 -20
index 5b5af6c..59272f9 100644
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this shouldn't be
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
drivers/gpu/drm/i915/i915_gem_gtt.c +5 -0
index 1411613..90c3d0f 100644
@@ -2159,10 +2159,15 @@ i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }
Signature · 6 lines
1.9.0

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#2 lore raw
Quoted 60 lines · Daniel Vetter
Quoted 43 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +2 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);

drivers/gpu/drm/i915/i915_gem.c struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj, · line 4514

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem.c +5 -0
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);

Hm, this has the risk that we leave unwanted vmas around for a bit longer. They will get cleaned up though when the final object references goes away, so the leak is fairly restricted. And currently we don't have a shrinker to just whack out vma objects even ...

Quoted 5 lines

It's definitely a much neater solution than what I had in mind with moving vmas to full-blown active tracking like we do objects. So I'm tempted to go with, but have a bit a lurking feeling that I'm missing something.

Chris?

I don't think that only taking the reference for whilst the vma is active would add much extra code or complexity and being consistent with the existing active tracking has the advantages you mention.

If we could clean up the vma handling in move_to_inactive that would remove a major wart all by itself.
-Chris

Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
#3 lore raw
In reply to Michel Thierry
Quoted 105 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +8 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);
+
 	list_del(&vma->vma_link);
 
 	kfree(vma);
drivers/gpu/drm/i915/i915_gem_context.c +3 -20
index 5b5af6c..59272f9 100644
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this shouldn't be
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index 1411613..90c3d0f 100644
@@ -2159,10 +2159,15 @@ i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 

drivers/gpu/drm/i915/i915_gem_gtt.c i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj, · line 2173

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +4 -0
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }

Seems like a reasonable fix to me, and it's simpler than Chris' request to ref when active. Logically his makes more sense, but it's not quite as simple to review. Actually active isn't the right thing to track IMO, bound is what you want to track. I think if anything, do that one as a patch on top, it does address the problem mentioned below. I also wonder given the recent reset lolz how this behaves there; and I'd make sure to thoroughly test the various reset/hang conditions.

Also, this puts a hard limit on the maximum number of VMAs in an address space based on sizeof(atomic_t). (unless you do what I said above) I doubt it's a problem in the near future, but it should be documented somewhere and I hope kref will explode loudly if you overflow it.

For future safety, I'd recommend dropping vm->ctx at this point. It shouldn't be hard at all to get rid of (quick grep shows only debugfs).

Signature · 1 line
Ben Widawsky, Intel Open Source Technology Center
#4 lore raw
Quoted 110 lines · Ben Widawsky
Quoted 109 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +8 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);
+
 	list_del(&vma->vma_link);
 
 	kfree(vma);
drivers/gpu/drm/i915/i915_gem_context.c +3 -20
index 5b5af6c..59272f9 100644
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this shouldn't be
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index 1411613..90c3d0f 100644
@@ -2159,10 +2159,15 @@ i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
Quoted 11 lines

drivers/gpu/drm/i915/i915_gem_gtt.c i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj, · line 2173

Quoted 6 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +4 -0
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }

Seems like a reasonable fix to me, and it's simpler than Chris' request to ref when active. Logically his makes more sense, but it's not quite as simple to review. Actually active isn't the right thing to track IMO, bound is what you want to track.

I didn't mean to imply this is anything but a semantic difference with the driver as it exists today.

Quoted 9 lines

I think if anything, do that one as a
patch on top, it does address the problem mentioned below. I also wonder given the recent reset lolz how this behaves there; and I'd make sure to thoroughly test the various reset/hang conditions.

Also, this puts a hard limit on the maximum number of VMAs in an address space based on sizeof(atomic_t). (unless you do what I said above) I doubt it's a problem in the near future, but it should be documented somewhere and I hope kref will explode loudly if you overflow it.

Also, on second thought, this is a don't care.

Quoted 3 lines

For future safety, I'd recommend dropping vm->ctx at this point. It shouldn't be hard at all to get rid of (quick grep shows only debugfs).

Signature · 1 line
Ben Widawsky, Intel Open Source Technology Center
#5 lore raw
Quoted 123 lines · Ben Widawsky
Quoted 110 lines · Ben Widawsky
Quoted 109 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context was holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing referencing it releases the ref, and it can just call the base cleanup and free the ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23 +++--------------------
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index 2acc03f..a879a93 100644
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);
 
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct i915_hw_ppgtt, base)
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt, base)
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/i915/i915_gem.c +8 -0
index dcd8d7b..25a32b9 100644
@@ -4499,12 +4499,20 @@ struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,
 
 void i915_gem_vma_destroy(struct i915_vma *vma)
 {
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);
 
 	/* Keep the vma as a placeholder in the execbuffer reservation lists */
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);
+
 	list_del(&vma->vma_link);
 
 	kfree(vma);
drivers/gpu/drm/i915/i915_gem_context.c +3 -20
index 5b5af6c..59272f9 100644
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this shouldn't be
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-				    list_is_singular(&vma->vma_link)))
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
index 1411613..90c3d0f 100644
@@ -2159,10 +2159,15 @@ i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
Quoted 11 lines

drivers/gpu/drm/i915/i915_gem_gtt.c i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj, · line 2173

Quoted 6 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +4 -0
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }

Seems like a reasonable fix to me, and it's simpler than Chris' request to ref when active. Logically his makes more sense, but it's not quite as simple to review. Actually active isn't the right thing to track IMO, bound is what you want to track.

Quoted 2 lines

I didn't mean to imply this is anything but a semantic difference with the driver as it exists today.

Well using active would be what we want, with bound we'd have a reference
loop: ppgtt keeps the vmas bound and the vmas keeps the ppgtt around.

But active isn't the right thing either: Currently we keep the active state per-bo, but have per-vma lrus, so a mix of obj/vma tracking. We've had lots of discussions about this and imo neither of the possible active approaches is really worth the fuss of converting to it, now that the code is in.

Assume I don't spot any hidden tricky bits after a good night's worth of sleep (or anyone else) I'll pull this in tomorrow. So please keep poking holes ...

Quoted 11 lines
Quoted 9 lines

I think if anything, do that one as a
patch on top, it does address the problem mentioned below. I also wonder given the recent reset lolz how this behaves there; and I'd make sure to thoroughly test the various reset/hang conditions.

Also, this puts a hard limit on the maximum number of VMAs in an address space based on sizeof(atomic_t). (unless you do what I said above) I doubt it's a problem in the near future, but it should be documented somewhere and I hope kref will explode loudly if you overflow it.

Also, on second thought, this is a don't care.

Yeah, the kernel generally assumes that an atomic_t or long is sufficiently big for a refcount for all of memory. Maybe giant 32bit pae boxes violated that, but meh.

Quoted 2 lines
Quoted 2 lines

For future safety, I'd recommend dropping vm->ctx at this point. It shouldn't be hard at all to get rid of (quick grep shows only debugfs).

Hm, that's a good point since that pointers is a loop in the refcounting. So if we really don't need it then I agree we should ditch it. Michel, can you please throw that follow-up patch on top if that's all there is to it indeed?

Thanks, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
#6 lore raw
Quoted 9 lines

-----Original Message-----
From: Daniel Vetter [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Tuesday, July 29, 2014 8:20 PM
To: Ben Widawsky
Cc: Thierry, Michel; [email protected]
Subject: Re: [Intel-gfx] [PATCH] drm/i915: vma/ppgtt lifetime rules

Quoted 6 lines
Quoted 5 lines · Ben Widawsky
Quoted 4 lines · Ben Widawsky
Quoted 3 lines · Michel Thierry

VMAs should take a reference of the address space they use.

Now, when the fd is closed, it will release the ref that the context

was

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

holding, but it will still be referenced by any vmas that are still active.

ppgtt_release() should then only be called when the last thing

referencing

Quoted 1 line
Quoted 1 line
Quoted 1 line

it releases the ref, and it can just call the base cleanup and free

the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

ppgtt.

---
 drivers/gpu/drm/i915/i915_drv.h         |  2 ++
 drivers/gpu/drm/i915/i915_gem.c         |  8 ++++++++
 drivers/gpu/drm/i915/i915_gem_context.c | 23

+++--------------------

Quoted 30 lines
Quoted 4 lines
Quoted 4 lines

Review

Quoted 4 lines
 drivers/gpu/drm/i915/i915_gem_gtt.c     |  5 +++++
 4 files changed, 18 insertions(+), 20 deletions(-)
(quoted diff) +0 -0

b/drivers/gpu/drm/i915/i915_drv.h

Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/i915/i915_drv.h void i915_gem_object_ggtt_unpin(struct

Quoted 4 lines

index 2acc03f..a879a93 100644

drivers/gpu/drm/i915/i915_drv.h +0 -0
@@ -2495,7 +2495,9 @@ void i915_gem_object_ggtt_unpin(struct

drm_i915_gem_object *obj);

Quoted 3 lines
Quoted 3 lines

drivers/gpu/drm/i915/i915_drv.h void i915_gem_object_ggtt_unpin(struct · line 2496

Quoted 3 lines
drivers/gpu/drm/i915/i915_drv.h +0 -0
 /* i915_gem_context.c */
 #define ctx_to_ppgtt(ctx) container_of((ctx)->vm, struct

i915_hw_ppgtt, base)

Quoted 1 line
Quoted 1 line

drivers/gpu/drm/i915/i915_drv.h void i915_gem_object_ggtt_unpin(struct · line 2497

Quoted 1 line
drivers/gpu/drm/i915/i915_drv.h +1 -0
+#define vm_to_ppgtt(vm) container_of(vm, struct i915_hw_ppgtt,

base)

Quoted 5 lines
Quoted 5 lines

drivers/gpu/drm/i915/i915_drv.h void i915_gem_object_ggtt_unpin(struct drm_i915_gem_object *obj);

Quoted 5 lines
drivers/gpu/drm/i915/i915_drv.h +1 -0
 int __must_check i915_gem_context_init(struct drm_device *dev);
+void ppgtt_release(struct kref *kref);
 void i915_gem_context_fini(struct drm_device *dev);
 void i915_gem_context_reset(struct drm_device *dev);
 int i915_gem_context_open(struct drm_device *dev, struct drm_file

*file);

Quoted 1 line
Quoted 1 line

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/gpu/drm/i915/i915_gem.c

Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/i915/i915_gem.c struct i915_vma

Quoted 4 lines

index dcd8d7b..25a32b9 100644

drivers/gpu/drm/i915/i915_gem.c +0 -0
@@ -4499,12 +4499,20 @@ struct i915_vma

*i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/gpu/drm/i915/i915_gem.c struct i915_vma *i915_gem_obj_to_vma(struct drm_i915_gem_object *obj,

Quoted 7 lines

void i915_gem_vma_destroy(struct i915_vma *vma)
{

drivers/gpu/drm/i915/i915_gem.c +2 -0
+	struct i915_address_space *vm = NULL;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 	WARN_ON(vma->node.allocated);
 
 	/* Keep the vma as a placeholder in the execbuffer

reservation lists

Quoted 20 lines

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/gpu/drm/i915/i915_gem.c +6 -0
 	if (!list_empty(&vma->exec_list))
 		return;
 
+	vm = vma->vm;
+	ppgtt = vm_to_ppgtt(vm);
+
+	if (ppgtt)
+		kref_put(&ppgtt->ref, ppgtt_release);
+
 	list_del(&vma->vma_link);
Quoted 2 lines
Quoted 2 lines

Review

Quoted 2 lines
drivers/gpu/drm/i915/i915_gem.c +0 -0
 	kfree(vma);
(quoted diff) +0 -0

b/drivers/gpu/drm/i915/i915_gem_context.c

Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/i915/i915_gem_context.c static void do_ppgtt_cleanup(struct

Quoted 4 lines

index 5b5af6c..59272f9 100644

drivers/gpu/drm/i915/i915_gem_context.c +0 -0
@@ -108,30 +108,13 @@ static void do_ppgtt_cleanup(struct

i915_hw_ppgtt *ppgtt)

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/gpu/drm/i915/i915_gem_context.c static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt)

Quoted 7 lines
drivers/gpu/drm/i915/i915_gem_context.c +0 -4
 		return;
 	}
 
-	/*
-	 * Make sure vmas are unbound before we take down the drm_mm
-	 *
-	 * FIXME: Proper refcounting should take care of this, this

shouldn't

Quoted 9 lines

be

Quoted 8 lines
Quoted 8 lines

drivers/gpu/drm/i915/i915_gem_context.c static void do_ppgtt_cleanup(struct i915_hw_ppgtt *ppgtt) · line 124

Quoted 8 lines
drivers/gpu/drm/i915/i915_gem_context.c +0 -8
-	 * needed at all.
-	 */
-	if (!list_empty(&vm->active_list)) {
-		struct i915_vma *vma;
-
-		list_for_each_entry(vma, &vm->active_list, mm_list)
-			if (WARN_ON(list_empty(&vma->vma_link) ||
-

list_is_singular(&vma->vma_link)))

Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/i915/i915_gem_context.c +2 -7
-				break;
-
-		i915_gem_evict_vm(&ppgtt->base, true);
-	} else {
-		i915_gem_retire_requests(dev);
-		i915_gem_evict_vm(&ppgtt->base, false);
-	}
+	/* vmas should already be unbound */
+	WARN_ON(!list_empty(&vm->active_list));
 
 	ppgtt->base.cleanup(&ppgtt->base);
 }
 
Quoted 6 lines
Quoted 6 lines

Review

Quoted 6 lines
drivers/gpu/drm/i915/i915_gem_context.c +1 -1
-static void ppgtt_release(struct kref *kref)
+void ppgtt_release(struct kref *kref)
 {
 	struct i915_hw_ppgtt *ppgtt =
 		container_of(kref, struct i915_hw_ppgtt, ref);
(quoted diff) +0 -0

b/drivers/gpu/drm/i915/i915_gem_gtt.c

Quoted 22 lines
Quoted 4 lines
Quoted 4 lines

drivers/gpu/drm/i915/i915_gem_gtt.c

Quoted 4 lines

index 1411613..90c3d0f 100644

drivers/gpu/drm/i915/i915_gem_gtt.c +0 -0
@@ -2159,10 +2159,15 @@

i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj,

Quoted 17 lines
Quoted 17 lines
Quoted 5 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +1 -0
 				  struct i915_address_space *vm)
 {
 	struct i915_vma *vma;
+	struct i915_hw_ppgtt *ppgtt = NULL;
 

drivers/gpu/drm/i915/i915_gem_gtt.c i915_gem_obj_lookup_or_create_vma(struct drm_i915_gem_object *obj, · line 2173

Quoted 10 lines
drivers/gpu/drm/i915/i915_gem_gtt.c +4 -0
 	vma = i915_gem_obj_to_vma(obj, vm);
 	if (!vma)
 		vma = __i915_gem_vma_create(obj, vm);
 
+	ppgtt = vm_to_ppgtt(vm);
+	if (ppgtt)
+		kref_get(&ppgtt->ref);
+
 	return vma;
 }

Seems like a reasonable fix to me, and it's simpler than Chris'

request

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

to ref when active. Logically his makes more sense, but it's not quite as simple to review. Actually active isn't the right thing to track

IMO,

Quoted 14 lines
Quoted 4 lines
Quoted 1 line

bound is what you want to track.

I didn't mean to imply this is anything but a semantic difference with the driver as it exists today.

Well using active would be what we want, with bound we'd have a reference
loop: ppgtt keeps the vmas bound and the vmas keeps the ppgtt around.

But active isn't the right thing either: Currently we keep the active state per-bo, but have per-vma lrus, so a mix of obj/vma tracking. We've had lots of discussions about this and imo neither of the possible active approaches is really worth the fuss of converting to it, now that the code is in.

Quoted 8 lines

Assume I don't spot any hidden tricky bits after a good night's worth of sleep (or anyone else) I'll pull this in tomorrow. So please keep poking holes ...

Quoted 2 lines
Quoted 2 lines

I think if anything, do that one as a
patch on top, it does address the problem mentioned below. I also

wonder

Quoted 1 line
Quoted 1 line

given the recent reset lolz how this behaves there; and I'd make sure

to

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

thoroughly test the various reset/hang conditions.

Also, this puts a hard limit on the maximum number of VMAs in an

address

Quoted 4 lines
Quoted 3 lines

space based on sizeof(atomic_t). (unless you do what I said above) I doubt it's a problem in the near future, but it should be documented somewhere and I hope kref will explode loudly if you overflow it.

Quoted 8 lines
Quoted 1 line

Also, on second thought, this is a don't care.

Yeah, the kernel generally assumes that an atomic_t or long is sufficiently big for a refcount for all of memory. Maybe giant 32bit pae boxes violated that, but meh.

Quoted 2 lines
Quoted 2 lines

For future safety, I'd recommend dropping vm->ctx at this point. It shouldn't be hard at all to get rid of (quick grep shows only

debugfs).

Quoted 5 lines

Hm, that's a good point since that pointers is a loop in the refcounting. So if we really don't need it then I agree we should ditch it. Michel, can you please throw that follow-up patch on top if that's all there is to it indeed?

Sure, I'll prepare that follow-up patch.

-Michel

Quoted 6 lines

Thanks, Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx