drm/ttm Improved io_mem_reserve / io_mem_free_calling
Subsystems (3)
cover drm/ttm Improved io_mem_reserve / io_mem_free_calling
The following patch is really intended for the next merge window.
RFC:
1) Are there any implementations of driver::io_mem_reserve today that can't use the fastpath?
2) Can we put an atomic requirement on driver::io_mem_reserve / driver::io_mem_free?
The patch improves on the io_mem_reserve / io_mem_free calling sequences by introducing a fastpath and an optional eviction mechanism.
The fastpath is enabled by default and is switched off by the driver setting struct ttm_mem_type_manager::io_reserve_fastpath to false on mem type init. With the fastpath no locking occurs, and io_mem_free is never called. I'm not sure if there are any implementations today that could not use the fastpath.
As mentioned in the patch, if the fastpath is disabled, calls to io_mem_reserve and io_mem_free are exactly balanced, and refcounted within struct ttm_mem_reg so that io_mem_reserve should never be called recursively for the same struct ttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm using man::io_reserve_mutex for this. Can we use a spinlock? That would require io_mem_reserve and io_mem_free to be atomic.
Optionally, there is an eviction mechanism that is activated by setting struct ttm_mem_type_manager::use_io_reserve_lru to true when initialized. If the eviction mechanism is activated, and io_mem_reserve returns -EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.
PATCH 1/1 drm/ttm: Fix up io_mem_reserve / io_mem_free calling
This patch attempts to fix up shortcomings with the current calling sequences.
1) There's a fastpath where no locking occurs and only io_mem_reserved is called to obtain needed info for mapping. The fastpath is set per memory type manager.
2) If the fastpath is disabled, io_mem_reserve and io_mem_free will be exactly balanced and not called recursively for the same struct ttm_mem_reg.
3) Optionally the driver can choose to enable a per memory type manager LRU eviction mechanism that, when io_mem_reserve returns -EAGAIN will attempt to kill user-space mappings of memory in that manager to free up needed resources
--- drivers/gpu/drm/ttm/ttm_bo.c | 44 ++++++++++-- drivers/gpu/drm/ttm/ttm_bo_util.c | 129 +++++++++++++++++++++++++++++++++---- drivers/gpu/drm/ttm/ttm_bo_vm.c | 23 ++++-- include/drm/ttm/ttm_bo_api.h | 6 ++- include/drm/ttm/ttm_bo_driver.h | 104 ++++++++++++++++-------------- 5 files changed, 226 insertions(+), 80 deletions(-)
drivers/gpu/drm/ttm/ttm_bo.c +36 -8
@@ -361,8 +361,13 @@ static int ttm_bo_handle_move_mem(struct ttm_buffer_object *bo,int ret =0 ;if (old_is_pci || new_is_pci || - ((mem->placement & bo->mem.placement &TTM_PL_MASK_CACHING ) ==0 )) -ttm_bo_unmap_virtual (bo); + ((mem->placement & bo->mem.placement &TTM_PL_MASK_CACHING ) ==0 )) { + ret =ttm_mem_io_lock (old_man, true); +if (unlikely (ret !=0 )) + goto out_err; +ttm_bo_unmap_virtual_locked (bo); +ttm_mem_io_unlock (old_man); + }/* * Create and bind a ttm if required.
@@ -451,7 +456,6 @@ static void ttm_bo_cleanup_memtype_use(struct ttm_buffer_object *bo)ttm_tt_destroy (bo->ttm); bo->ttm =NULL ; } -ttm_bo_mem_put (bo, &bo->mem);atomic_set (&bo->reserved,0 );
@@ -651,6 +655,7 @@ static void ttm_bo_release(struct kref *kref)struct ttm_buffer_object *bo = container_of(kref, struct ttm_buffer_object, kref);struct ttm_bo_device *bdev = bo->bdev; +struct ttm_mem_type_manager *man = &bdev->man[bo->mem.mem_type];if (likely (bo->vm_node !=NULL )) {rb_erase (&bo->vm_rb, &bdev->addr_space_rb);
@@ -658,6 +663,9 @@ static void ttm_bo_release(struct kref *kref) bo->vm_node =NULL ; }write_unlock (&bdev->vm_lock); +ttm_mem_io_lock (man, false); +ttm_mem_io_free_vm (bo); +ttm_mem_io_unlock (man);ttm_bo_cleanup_refs_or_queue (bo);kref_put (&bo->list_kref, ttm_bo_release_list);write_lock (&bdev->vm_lock);
@@ -714,7 +722,8 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo, bool interruptible, evict_mem = bo->mem; evict_mem.mm_node =NULL ; - evict_mem.bus.io_reserved = false; + evict_mem.bus.io_reserved_vm = false; + evict_mem.bus.io_reserved_count =0 ; placement.fpfn =0 ; placement.lpfn =0 ;
@@ -1051,7 +1060,8 @@ int ttm_bo_move_buffer(struct ttm_buffer_object *bo, mem.num_pages = bo->num_pages; mem.size = mem.num_pages <<PAGE_SHIFT ; mem.page_alignment = bo->mem.page_alignment; - mem.bus.io_reserved = false; + mem.bus.io_reserved_vm = false; + mem.bus.io_reserved_count =0 ;/* * Determine where to move the buffer. */
@@ -1171,6 +1181,7 @@ int ttm_bo_init(struct ttm_bo_device *bdev,INIT_LIST_HEAD (&bo->lru);INIT_LIST_HEAD (&bo->ddestroy);INIT_LIST_HEAD (&bo->swap); +INIT_LIST_HEAD (&bo->io_reserve_lru); bo->bdev = bdev; bo->glob = bdev->glob; bo->type = type;
@@ -1180,7 +1191,8 @@ int ttm_bo_init(struct ttm_bo_device *bdev, bo->mem.num_pages = bo->num_pages; bo->mem.mm_node =NULL ; bo->mem.page_alignment = page_alignment; - bo->mem.bus.io_reserved = false; + bo->mem.bus.io_reserved_vm = false; + bo->mem.bus.io_reserved_count =0 ; bo->buffer_start = buffer_start &PAGE_MASK ; bo->priv_flags =0 ; bo->mem.placement = (TTM_PL_FLAG_SYSTEM |TTM_PL_FLAG_CACHED );
@@ -1354,6 +1366,10 @@ int ttm_bo_init_mm(struct ttm_bo_device *bdev, unsigned type,BUG_ON (type >=TTM_NUM_MEM_TYPES ); man = &bdev->man[type];BUG_ON (man->has_type); + man->io_reserve_fastpath = true; + man->use_io_reserve_lru = false; +mutex_init (&man->io_reserve_mutex); +INIT_LIST_HEAD (&man->io_reserve_lru); ret = bdev->driver->init_mem_type (bdev, type, man);if (ret)
@@ -1560,7 +1576,7 @@ bool ttm_mem_reg_is_pci(struct ttm_bo_device *bdev, struct ttm_mem_reg *mem)return true; } -void ttm_bo_unmap_virtual (struct ttm_buffer_object *bo) +void ttm_bo_unmap_virtual_locked (struct ttm_buffer_object *bo) {struct ttm_bo_device *bdev = bo->bdev;loff_t offset = (loff_t ) bo->addr_space_offset;
@@ -1569,8 +1585,20 @@ void ttm_bo_unmap_virtual(struct ttm_buffer_object *bo)if (!bdev->dev_mapping)return ;unmap_mapping_range (bdev->dev_mapping, offset, holelen,1 ); -ttm_mem_io_free (bdev, &bo->mem); +ttm_mem_io_free_vm (bo); } + +void ttm_bo_unmap_virtual (struct ttm_buffer_object *bo) +{ +struct ttm_bo_device *bdev = bo->bdev; +struct ttm_mem_type_manager *man = &bdev->man[bo->mem.mem_type]; + +ttm_mem_io_lock (man, false); +ttm_bo_unmap_virtual_locked (bo); +ttm_mem_io_unlock (man); +} + +EXPORT_SYMBOL (ttm_bo_unmap_virtual);static void ttm_bo_vm_insert_rb (struct ttm_buffer_object *bo)
drivers/gpu/drm/ttm/ttm_bo_util.c +115 -14
224 lines not on this page · raw message
drivers/gpu/drm/ttm/ttm_bo_vm.c +15 -8
@@ -83,6 +83,8 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)int i;unsigned long address = (unsigned long )vmf->virtual_address;int retval =VM_FAULT_NOPAGE ; +struct ttm_mem_type_manager *man = + &bdev->man[bo->mem.mem_type];/* * Work around locking order reversal in fault / nopfn
@@ -130,12 +132,16 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf) }else spin_unlock (&bo->lock); - - ret =ttm_mem_io_reserve (bdev, &bo->mem); -if (ret) { - retval =VM_FAULT_SIGBUS ; + ret =ttm_mem_io_lock (man, true); +if (unlikely (ret !=0 )) { + retval =VM_FAULT_NOPAGE ; goto out_unlock; } + ret =ttm_mem_io_reserve_vm (bo); +if (unlikely (ret !=0 )) { + retval =VM_FAULT_SIGBUS ; + goto out_io_unlock; + } page_offset = ((address - vma->vm_start) >>PAGE_SHIFT ) + bo->vm_node->start - vma->vm_pgoff;
@@ -144,7 +150,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)if (unlikely (page_offset >= bo->num_pages)) { retval =VM_FAULT_SIGBUS ; - goto out_unlock; + goto out_io_unlock; }/*
@@ -182,7 +188,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf) page =ttm_tt_get_page (ttm, page_offset);if (unlikely (!page && i ==0 )) { retval =VM_FAULT_OOM ; - goto out_unlock; + goto out_io_unlock; }else if (unlikely (!page)) {break ; }
@@ -200,14 +206,15 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)else if (unlikely (ret !=0 )) { retval = (ret == -ENOMEM ) ?VM_FAULT_OOM :VM_FAULT_SIGBUS ; - goto out_unlock; + goto out_io_unlock; } address +=PAGE_SIZE ;if (unlikely (++page_offset >= page_last))break ; } - +out_io_unlock: +ttm_mem_io_unlock (man); out_unlock:ttm_bo_unreserve (bo);return retval;
include/drm/ttm/ttm_bo_api.h +5 -1
@@ -74,6 +74,8 @@ struct ttm_placement {* @is_iomem: is this io memory ? * @size: size in byte * @offset: offset from the base address +* @io_reserved_vm: The VM system has a refcount in @io_reserved_count +* @io_reserved_count: Refcounting the numbers of callers to ttm_mem_io_reserve * * Structure indicating the bus placement of an object. */
@@ -83,7 +85,8 @@ struct ttm_bus_placement {unsigned long size;unsigned long offset;bool is_iomem; -bool io_reserved; +bool io_reserved_vm; +uint64_t io_reserved_count; };
@@ -237,6 +240,7 @@ struct ttm_buffer_object {struct list_head lru;struct list_head ddestroy;struct list_head swap; +struct list_head io_reserve_lru;uint32_t val_seq;bool seq_valid;
include/drm/ttm/ttm_bo_driver.h +55 -49
134 lines not on this page · raw message
Signature · 1 line
1.6.2.5
Quoted 26 lines · Thomas Hellstrom
The following patch is really intended for the next merge window.
RFC:
1) Are there any implementations ofdriver::io_mem_reservetoday that can't use the fastpath?
2) Can we put an atomic requirement ondriver::io_mem_reserve/driver::io_mem_free?The patch improves on the
io_mem_reserve/io_mem_freecalling sequences by introducing a fastpath and an optional eviction mechanism.The fastpath is enabled by default and is switched off by the driver setting struct
ttm_mem_type_manager::io_reserve_fastpathto false on mem type init. With the fastpath no locking occurs, andio_mem_freeis never called. I'm not sure if there are any implementations today that could not use the fastpath.As mentioned in the patch, if the fastpath is disabled, calls to
io_mem_reserveandio_mem_freeare exactly balanced, and refcounted within structttm_mem_regso thatio_mem_reserveshould never be called recursively for the same structttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm usingman::io_reserve_mutexfor this. Can we use a spinlock? That would requireio_mem_reserveandio_mem_freeto be atomic.
Quoted 6 lines
Optionally, there is an eviction mechanism that is activated by setting struct
ttm_mem_type_manager::use_io_reserve_lruto true when initialized. If the eviction mechanism is activated, andio_mem_reservereturns-EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.
Radeon can use fast path, i think nouveau can too. I am not sure we can consider io_mem_reserve as atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.
Quick review of the patch looks good, i will try to take a closer look latter.
Cheers,
Jerome Glisse
Quoted 35 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom
The following patch is really intended for the next merge window.
RFC:
1) Are there any implementations ofdriver::io_mem_reservetoday that can't use the fastpath?
2) Can we put an atomic requirement ondriver::io_mem_reserve/driver::io_mem_free?The patch improves on the
io_mem_reserve/io_mem_freecalling sequences by introducing a fastpath and an optional eviction mechanism.The fastpath is enabled by default and is switched off by the driver setting struct
ttm_mem_type_manager::io_reserve_fastpathto false on mem type init. With the fastpath no locking occurs, andio_mem_freeis never called. I'm not sure if there are any implementations today that could not use the fastpath.As mentioned in the patch, if the fastpath is disabled, calls to
io_mem_reserveandio_mem_freeare exactly balanced, and refcounted within structttm_mem_regso thatio_mem_reserveshould never be called recursively for the same structttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm usingman::io_reserve_mutexfor this. Can we use a spinlock? That would requireio_mem_reserveandio_mem_freeto be atomic.Quoted 7 lines
Optionally, there is an eviction mechanism that is activated by setting struct
ttm_mem_type_manager::use_io_reserve_lruto true when initialized. If the eviction mechanism is activated, andio_mem_reservereturns-EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.
Quoted 6 lines
Radeon can use fast path, i think nouveau can too. I am not sure we can consider
io_mem_reserveas atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.
In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of umap_mapping_range() that may in some cases be long. Perhaps too long to use a spinlock.
Quoted 5 lines
Quick review of the patch looks good, i will try to take a closer look latter.
Cheers,
Jerome Glisse
Great. Thanks,
Thomas
Quoted 43 lines · Thomas Hellstrom
Quoted 35 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom
The following patch is really intended for the next merge window.
RFC:
1) Are there any implementations ofdriver::io_mem_reservetoday that can't use the fastpath?
2) Can we put an atomic requirement ondriver::io_mem_reserve/driver::io_mem_free?The patch improves on the
io_mem_reserve/io_mem_freecalling sequences by introducing a fastpath and an optional eviction mechanism.The fastpath is enabled by default and is switched off by the driver setting struct
ttm_mem_type_manager::io_reserve_fastpathto false on mem type init. With the fastpath no locking occurs, andio_mem_freeis never called. I'm not sure if there are any implementations today that could not use the fastpath.As mentioned in the patch, if the fastpath is disabled, calls to
io_mem_reserveandio_mem_freeare exactly balanced, and refcounted within structttm_mem_regso thatio_mem_reserveshould never be called recursively for the same structttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm usingman::io_reserve_mutexfor this. Can we use a spinlock? That would requireio_mem_reserveandio_mem_freeto be atomic.Quoted 7 lines
Optionally, there is an eviction mechanism that is activated by setting struct
ttm_mem_type_manager::use_io_reserve_lruto true when initialized. If the eviction mechanism is activated, andio_mem_reservereturns-EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.Quoted 6 lines
Radeon can use fast path, i think nouveau can too. I am not sure we can consider
io_mem_reserveas atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.
Quoted 5 lines
In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of
umap_mapping_range()that may in some cases be long. Perhaps too long to use a spinlock.
I'd rather keep the mutex personally, the code I have in development uses mutexes itself beyond the io_mem_reserve/io_mem_free calls. An earlier revision used spinlocks, but it wasn't very nice.
Ben.
Quoted 10 lines
Quoted 6 lines
Quick review of the patch looks good, i will try to take a closer look latter.
Cheers,
Jerome GlisseGreat. Thanks,
Thomas
Quoted 54 lines · Ben Skeggs
Quoted 46 lines · Thomas Hellstrom
Quoted 37 lines · Jerome Glisse
Quoted 26 lines · Thomas Hellstrom
The following patch is really intended for the next merge window.
RFC:
1) Are there any implementations ofdriver::io_mem_reservetoday that can't use the fastpath?
2) Can we put an atomic requirement ondriver::io_mem_reserve/driver::io_mem_free?The patch improves on the
io_mem_reserve/io_mem_freecalling sequences by introducing a fastpath and an optional eviction mechanism.The fastpath is enabled by default and is switched off by the driver setting struct
ttm_mem_type_manager::io_reserve_fastpathto false on mem type init. With the fastpath no locking occurs, andio_mem_freeis never called. I'm not sure if there are any implementations today that could not use the fastpath.As mentioned in the patch, if the fastpath is disabled, calls to
io_mem_reserveandio_mem_freeare exactly balanced, and refcounted within structttm_mem_regso thatio_mem_reserveshould never be called recursively for the same structttm_mem_reg.
Locking is required to make sure that ptes are never present on when the underlying memory region is not reserved. Currently I'm usingman::io_reserve_mutexfor this. Can we use a spinlock? That would requireio_mem_reserveandio_mem_freeto be atomic.Quoted 8 lines
Optionally, there is an eviction mechanism that is activated by setting struct
ttm_mem_type_manager::use_io_reserve_lruto true when initialized. If the eviction mechanism is activated, andio_mem_reservereturns-EAGAIN, it will attempt to kill user-space mappings to free up reserved regions. Kernel mappings (ttm_bo_kmap) are not affected.Quoted 7 lines
Radeon can use fast path, i think nouveau can too. I am not sure we can consider
io_mem_reserveas atomic. Use case i fear is GPU with remappable apperture i don't know what kind of code we would need for that and it might sleep. Thought my first guess is that it likely can be done atomicly.Quoted 6 lines
In that case, I think I will change it to a spinlock, with a code comment that it can be changed to a mutex later if it turns out to be very hard / impossible to implement atomic operations. Another possible concern is the execution of
umap_mapping_range()that may in some cases be long. Perhaps too long to use a spinlock.
Quoted 6 lines
I'd rather keep the mutex personally, the code I have in development uses mutexes itself beyond the io_mem_reserve/io_mem_free calls. An earlier revision used spinlocks, but it wasn't very nice.
Ben.
OK.
Note that any per-mem-type shared objects accessed by io_mem_reserve / io_mem_free don't need any further protection beyond the lock we're discussing. For the same mem_type, io_mem_reserve / io_mem_free will be completely serialized with this patch.
Anyway, let's keep the mutex.
/Thomas