Skip to contentSkip to conversation

drm/ttm Improved io_mem_reserve / io_mem_free_calling

RFC PATCH In review Thomas Hellstrom 3 participants 6 messages Nov 11, 2010 – Nov 12, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover drm/ttm Improved io_mem_reserve / io_mem_free_calling

The following patch is really intended for the next merge window.

RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?

The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.

The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.

As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.

Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.

#1 lore raw

PATCH 1/1 drm/ttm: Fix up io_mem_reserve / io_mem_free calling

This patch attempts to fix up shortcomings with the current calling sequences.

1) There's a fastpath where no locking occurs and only io_mem_reserved is called to obtain needed info for mapping. The fastpath is set per memory type manager.
2) If the fastpath is disabled, io_mem_reserve and io_mem_free will be exactly balanced and not called recursively for the same struct ttm_mem_reg.
3) Optionally the driver can choose to enable a per memory type manager LRU eviction mechanism that, when io_mem_reserve returns -EAGAIN will attempt to kill user-space mappings of memory in that manager to free up needed resources

---
 drivers/gpu/drm/ttm/ttm_bo.c      |   44 ++++++++++--
 drivers/gpu/drm/ttm/ttm_bo_util.c |  129 +++++++++++++++++++++++++++++++++----
 drivers/gpu/drm/ttm/ttm_bo_vm.c   |   23 ++++--
 include/drm/ttm/ttm_bo_api.h      |    6 ++-
 include/drm/ttm/ttm_bo_driver.h   |  104 ++++++++++++++++--------------
 5 files changed, 226 insertions(+), 80 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +36 -8
index 336a39d..489bde8 100644
@@ -361,8 +361,13 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,
 	int ret = 0;
 
 	if (old_is_pci || new_is_pci ||
-	    ((mem->placement & bo->mem.placement & TTM_PL_MASK_CACHING) == 0))
-		ttm_bo_unmap_virtual(bo);
+	    ((mem->placement & bo->mem.placement & TTM_PL_MASK_CACHING) == 0)) {
+		ret = ttm_mem_io_lock(old_man, true);
+		if (unlikely(ret != 0))
+			goto out_err;
+		ttm_bo_unmap_virtual_locked(bo);
+		ttm_mem_io_unlock(old_man);
+	}
 
 	/*
 	 * Create and bind a ttm if required.
@@ -451,7 +456,6 @@ static void ttm_bo_cleanup_memtype_use(struct ttm_buffer_object *bo)
 		ttm_tt_destroy(bo->ttm);
 		bo->ttm = NULL;
 	}
-
 	ttm_bo_mem_put(bo, &bo->mem);
 
 	atomic_set(&bo->reserved, 0);
@@ -651,6 +655,7 @@ static void ttm_bo_release(struct kref *kref)
 	struct ttm_buffer_object *bo =
 	    container_of(kref, struct ttm_buffer_object, kref);
 	struct ttm_bo_device *bdev = bo->bdev;
+	struct ttm_mem_type_manager *man = &bdev->man[bo->mem.mem_type];
 
 	if (likely(bo->vm_node != NULL)) {
 		rb_erase(&bo->vm_rb, &bdev->addr_space_rb);
@@ -658,6 +663,9 @@ static void ttm_bo_release(struct kref *kref)
 		bo->vm_node = NULL;
 	}
 	write_unlock(&bdev->vm_lock);
+	ttm_mem_io_lock(man, false);
+	ttm_mem_io_free_vm(bo);
+	ttm_mem_io_unlock(man);
 	ttm_bo_cleanup_refs_or_queue(bo);
 	kref_put(&bo->list_kref, ttm_bo_release_list);
 	write_lock(&bdev->vm_lock);
@@ -714,7 +722,8 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo, bool interruptible,
 
 	evict_mem = bo->mem;
 	evict_mem.mm_node = NULL;
-	evict_mem.bus.io_reserved = false;
+	evict_mem.bus.io_reserved_vm = false;
+	evict_mem.bus.io_reserved_count = 0;
 
 	placement.fpfn = 0;
 	placement.lpfn = 0;
@@ -1051,7 +1060,8 @@ int ttm_bo_move_buffer(struct ttm_buffer_object *bo,
 	mem.num_pages = bo->num_pages;
 	mem.size = mem.num_pages << PAGE_SHIFT;
 	mem.page_alignment = bo->mem.page_alignment;
-	mem.bus.io_reserved = false;
+	mem.bus.io_reserved_vm = false;
+	mem.bus.io_reserved_count = 0;
 	/*
 	 * Determine where to move the buffer.
 	 */
@@ -1171,6 +1181,7 @@ int ttm_bo_init(struct ttm_bo_device *bdev,
 	INIT_LIST_HEAD(&bo->lru);
 	INIT_LIST_HEAD(&bo->ddestroy);
 	INIT_LIST_HEAD(&bo->swap);
+	INIT_LIST_HEAD(&bo->io_reserve_lru);
 	bo->bdev = bdev;
 	bo->glob = bdev->glob;
 	bo->type = type;
@@ -1180,7 +1191,8 @@ int ttm_bo_init(struct ttm_bo_device *bdev,
 	bo->mem.num_pages = bo->num_pages;
 	bo->mem.mm_node = NULL;
 	bo->mem.page_alignment = page_alignment;
-	bo->mem.bus.io_reserved = false;
+	bo->mem.bus.io_reserved_vm = false;
+	bo->mem.bus.io_reserved_count = 0;
 	bo->buffer_start = buffer_start & PAGE_MASK;
 	bo->priv_flags = 0;
 	bo->mem.placement = (TTM_PL_FLAG_SYSTEM | TTM_PL_FLAG_CACHED);
@@ -1354,6 +1366,10 @@ int ttm_bo_init_mm(struct ttm_bo_device *bdev, unsigned type,
 	BUG_ON(type >= TTM_NUM_MEM_TYPES);
 	man = &bdev->man[type];
 	BUG_ON(man->has_type);
+	man->io_reserve_fastpath = true;
+	man->use_io_reserve_lru = false;
+	mutex_init(&man->io_reserve_mutex);
+	INIT_LIST_HEAD(&man->io_reserve_lru);
 
 	ret = bdev->driver->init_mem_type(bdev, type, man);
 	if (ret)
@@ -1560,7 +1576,7 @@ bool ttm_mem_reg_is_pci(struct ttm_bo_device *bdev, struct ttm_mem_reg *mem)
 	return true;
 }
 
-void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
+void ttm_bo_unmap_virtual_locked(struct ttm_buffer_object *bo)
 {
 	struct ttm_bo_device *bdev = bo->bdev;
 	loff_t offset = (loff_t) bo->addr_space_offset;
@@ -1569,8 +1585,20 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
 	if (!bdev->dev_mapping)
 		return;
 	unmap_mapping_range(bdev->dev_mapping, offset, holelen, 1);
-	ttm_mem_io_free(bdev, &bo->mem);
+	ttm_mem_io_free_vm(bo);
 }
+
+void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)
+{
+	struct ttm_bo_device *bdev = bo->bdev;
+	struct ttm_mem_type_manager *man = &bdev->man[bo->mem.mem_type];
+
+	ttm_mem_io_lock(man, false);
+	ttm_bo_unmap_virtual_locked(bo);
+	ttm_mem_io_unlock(man);
+}
+
+
 EXPORT_SYMBOL(ttm_bo_unmap_virtual);
 
 static void ttm_bo_vm_insert_rb(struct ttm_buffer_object *bo)
drivers/gpu/drm/ttm/ttm_bo_util.c +115 -14

224 lines not on this page · raw message

drivers/gpu/drm/ttm/ttm_bo_vm.c +15 -8
index fe6cb77..73202cb 100644
@@ -83,6 +83,8 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	int i;
 	unsigned long address = (unsigned long)vmf->virtual_address;
 	int retval = VM_FAULT_NOPAGE;
+	struct ttm_mem_type_manager *man =
+		&bdev->man[bo->mem.mem_type];
 
 	/*
 	 * Work around locking order reversal in fault / nopfn
@@ -130,12 +132,16 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	} else
 		spin_unlock(&bo->lock);
 
-
-	ret = ttm_mem_io_reserve(bdev, &bo->mem);
-	if (ret) {
-		retval = VM_FAULT_SIGBUS;
+	ret = ttm_mem_io_lock(man, true);
+	if (unlikely(ret != 0)) {
+		retval = VM_FAULT_NOPAGE;
 		goto out_unlock;
 	}
+	ret = ttm_mem_io_reserve_vm(bo);
+	if (unlikely(ret != 0)) {
+		retval = VM_FAULT_SIGBUS;
+		goto out_io_unlock;
+	}
 
 	page_offset = ((address - vma->vm_start) >> PAGE_SHIFT) +
 	    bo->vm_node->start - vma->vm_pgoff;
@@ -144,7 +150,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 
 	if (unlikely(page_offset >= bo->num_pages)) {
 		retval = VM_FAULT_SIGBUS;
-		goto out_unlock;
+		goto out_io_unlock;
 	}
 
 	/*
@@ -182,7 +188,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 			page = ttm_tt_get_page(ttm, page_offset);
 			if (unlikely(!page && i == 0)) {
 				retval = VM_FAULT_OOM;
-				goto out_unlock;
+				goto out_io_unlock;
 			} else if (unlikely(!page)) {
 				break;
 			}
@@ -200,14 +206,15 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 		else if (unlikely(ret != 0)) {
 			retval =
 			    (ret == -ENOMEM) ? VM_FAULT_OOM : VM_FAULT_SIGBUS;
-			goto out_unlock;
+			goto out_io_unlock;
 		}
 
 		address += PAGE_SIZE;
 		if (unlikely(++page_offset >= page_last))
 			break;
 	}
-
+out_io_unlock:
+	ttm_mem_io_unlock(man);
 out_unlock:
 	ttm_bo_unreserve(bo);
 	return retval;
include/drm/ttm/ttm_bo_api.h +5 -1
index beafc15..5510a8c 100644
@@ -74,6 +74,8 @@ struct ttm_placement {
  * @is_iomem:		is this io memory ?
  * @size:		size in byte
  * @offset:		offset from the base address
+ * @io_reserved_vm:     The VM system has a refcount in @io_reserved_count
+ * @io_reserved_count:  Refcounting the numbers of callers to ttm_mem_io_reserve
  *
  * Structure indicating the bus placement of an object.
  */
@@ -83,7 +85,8 @@ struct ttm_bus_placement {
 	unsigned long	size;
 	unsigned long	offset;
 	bool		is_iomem;
-	bool		io_reserved;
+	bool		io_reserved_vm;
+	uint64_t        io_reserved_count;
 };
 
 
@@ -237,6 +240,7 @@ struct ttm_buffer_object {
 	struct list_head lru;
 	struct list_head ddestroy;
 	struct list_head swap;
+	struct list_head io_reserve_lru;
 	uint32_t val_seq;
 	bool seq_valid;
 
include/drm/ttm/ttm_bo_driver.h +55 -49

134 lines not on this page · raw message

Signature · 1 line
1.6.2.5
#2 lore raw
In reply to Thomas Hellstrom
Quoted 26 lines · Thomas Hellstrom

The following patch is really intended for the next merge window.

RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?

The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.

The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.

As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.

Quoted 6 lines

Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.

Radeon can use fast path, i think nouveau can too. I am not sure we can consider io_mem_reserve as atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.

Quick review of the patch looks good, i will try to take a closer look latter.

Cheers,
Jerome Glisse

#3 lore raw
Quoted 35 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom

The following patch is really intended for the next merge window.

RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?

The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.

The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.

As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.

Quoted 7 lines

Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.

Quoted 6 lines

Radeon can use fast path, i think nouveau can too. I am not sure we can consider io_mem_reserve as atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.

In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of umap_mapping_range() that may in some cases be long. Perhaps too long to use a spinlock.

Quoted 5 lines

Quick review of the patch looks good, i will try to take a closer look latter.

Cheers,
Jerome Glisse

Great. Thanks,
Thomas

#4 lore raw
Quoted 43 lines · Thomas Hellstrom
Quoted 35 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom

The following patch is really intended for the next merge window.

RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?

The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.

The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.

As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.

Quoted 7 lines

Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.

Quoted 6 lines

Radeon can use fast path, i think nouveau can too. I am not sure we can consider io_mem_reserve as atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.

Quoted 5 lines

In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of umap_mapping_range() that may in some cases be long. Perhaps too long to use a spinlock.

I'd rather keep the mutex personally, the code I have in development uses mutexes itself beyond the io_mem_reserve/io_mem_free calls. An earlier revision used spinlocks, but it wasn't very nice.

Ben.

Quoted 10 lines
Quoted 6 lines

Quick review of the patch looks good, i will try to take a closer look latter.

Cheers,
Jerome Glisse

Great. Thanks,
Thomas

#5 lore raw
Quoted 54 lines · Ben Skeggs
Quoted 46 lines · Thomas Hellstrom
Quoted 37 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom

The following patch is really intended for the next merge window.

RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?

The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.

The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.

As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.

Quoted 8 lines

Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.

Quoted 7 lines

Radeon can use fast path, i think nouveau can too. I am not sure we can consider io_mem_reserve as atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.

Quoted 6 lines

In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of umap_mapping_range() that may in some cases be long. Perhaps too long to use a spinlock.

Quoted 6 lines

I'd rather keep the mutex personally, the code I have in development uses mutexes itself beyond the io_mem_reserve/io_mem_free calls. An earlier revision used spinlocks, but it wasn't very nice.

Ben.

OK.
Note that any per-mem-type shared objects accessed by io_mem_reserve / io_mem_free don't need any further protection beyond the lock we're discussing. For the same mem_type, io_mem_reserve / io_mem_free will be completely serialized with this patch.

Anyway, let's keep the mutex.

/Thomas