drm/ttm: Pool allocator
Subsystems (7)
cover drm/ttm: Pool allocator
When allocating wc/uc pages cache state transition requires cache flush which is expensive operation. To avoid cache flushes allocation of wc/uc pages should be done in large groups when only single cache flush is required for whole group of pages.
In some cases drivers need t oallocate and deallocate many pages in a short time frame. In this case we can avoid cache flushes if we keep pages in the pool before actually freeing them later.
arch/x86 was missing set_pages_array_wc and set_memory_array_wc. Patch 6 and 7 add missing functions and hooks set_pages_array_wc to the pool allocator.
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
PATCH 1/7 drm/ttm: add pool wc/uc page allocator
On AGP system we might allocate/free routinely uncached or wc memory, changing page from cached (wb) to uc or wc is very expensive and involves a lot of flushing. To improve performance this allocator use a pool of uc,wc pages.
Pools are protected with spinlocks to allow multiple threads to allocate pages simultanously. Expensive operations are done outside of spinlock to maximize concurrency.
Pools are linked lists of pages that were recently freed. Shrink callback is used for removing pages from pool. Half of pages not used between two subsequent shrink calls will be freed.
Pool fill is using separate spinlock to protect that parallel allocations won't result exceess allocations to the pool. To prevent dead lock chances fill_lock is locked with spin_trylock if it is not locked.
Based on Jerome Glisse's and Dave Airlie's pool allocator.
--- drivers/gpu/drm/ttm/Makefile | 2 +- drivers/gpu/drm/ttm/ttm_memory.c | 7 +- drivers/gpu/drm/ttm/ttm_page_alloc.c | 775 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/ttm/ttm_tt.c | 44 +- include/drm/ttm/ttm_page_alloc.h | 64 +++ 5 files changed, 867 insertions(+), 25 deletions(-) create mode 100644 drivers/gpu/drm/ttm/ttm_page_alloc.c create mode 100644 include/drm/ttm/ttm_page_alloc.h
drivers/gpu/drm/ttm/Makefile +1 -1
@@ -4,6 +4,6 @@ ccflags-y := -Iinclude/drm ttm-y := ttm_agp_backend.o ttm_memory.o ttm_tt.o ttm_bo.o \ ttm_bo_util.o ttm_bo_vm.o ttm_module.o ttm_global.o \ - ttm_object.o ttm_lock.o ttm_execbuf_util.o + ttm_object.o ttm_lock.o ttm_execbuf_util.o ttm_page_alloc.o obj-$(CONFIG_DRM_TTM) += ttm.o
drivers/gpu/drm/ttm/ttm_memory.c +6 -1
@@ -27,6 +27,7 @@#include "ttm/ttm_memory.h" #include "ttm/ttm_module.h" +#include "ttm/ttm_page_alloc.h" #include <linux/spinlock.h> #include <linux/sched.h> #include <linux/wait.h>
@@ -394,6 +395,7 @@ int ttm_mem_global_init(struct ttm_mem_global *glob)"Zone %7s: Available graphics memory: %llu kiB.\n" , zone->name, (unsigned long long) zone->max_mem >> 10); } +ttm_page_alloc_init (glob);return 0 ; out_no_zone:ttm_mem_global_release (glob);
@@ -406,6 +408,9 @@ void ttm_mem_global_release(struct ttm_mem_global *glob)unsigned int i;struct ttm_mem_zone *zone; +/* let the page allocator first stop the shrink work. */ +ttm_page_alloc_fini (); +flush_workqueue (glob->swap_queue);destroy_workqueue (glob->swap_queue); glob->swap_queue =NULL ;
@@ -413,7 +418,7 @@ void ttm_mem_global_release(struct ttm_mem_global *glob) zone = glob->zones[i];kobject_del (&zone->kobj);kobject_put (&zone->kobj); - } + }kobject_del (&glob->kobj);kobject_put (&glob->kobj); }
drivers/gpu/drm/ttm/ttm_page_alloc.c new file +775 -0
775 lines not on this page · raw message
drivers/gpu/drm/ttm/ttm_tt.c +21 -23
95 lines not on this page · raw message
include/drm/ttm/ttm_page_alloc.h new file +64 -0
64 lines not on this page · raw message
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 2/7 drm/ttm: Add debugfs output entry to pool allocator.
ttm_page_alloc_debugfs can be registered to output the state of pools.
Debugfs file will output number of pages freed from the pool, number of pages in pool now and the lowes number of pages in pool since previous shrink.
--- drivers/gpu/drm/ttm/ttm_page_alloc.c | 46 +++++++++++++++++++++++++++++---- include/drm/ttm/ttm_page_alloc.h | 4 +++ 2 files changed, 44 insertions(+), 6 deletions(-)
drivers/gpu/drm/ttm/ttm_page_alloc.c +40 -6
@@ -34,6 +34,7 @@#include <linux/spinlock.h> #include <linux/highmem.h> #include <linux/mm_types.h> +#include <linux/module.h> #include <linux/jiffies.h> #include <linux/timer.h> #include <linux/workqueue.h>
@@ -73,6 +74,9 @@ struct ttm_page_pool {unsigned npages;unsigned nlowpages;unsigned alloc_size; +char *name; +unsigned long nfrees; +unsigned long nrefills; };#define NUM_POOLS 4
@@ -240,6 +244,7 @@ static bool ttm_page_pool_free_pages_locked(struct ttm_page_pool *pool, {unsigned tmp; pool->npages -= freed_pages; + pool->nfrees += freed_pages;/* Calculate number of pages taken from nlowpages * npages_to_free = 1/2*nlowpages => * nlowpages_delta = 2*freed_pages
@@ -320,6 +325,7 @@ restart: } pool->npages -= freed_pages; + pool->nfrees += freed_pages;/* set nlowpages to zero to prevent extra freeing in thsi patch. * nlowpages is reseted later after all work has been finnished. **/
@@ -537,6 +543,7 @@ static void ttm_page_pool_fill_locked(struct ttm_page_pool *pool,if (!r) {list_splice (&new_pages, &pool->list); + ++pool->nrefills; pool->npages += pool->alloc_size;/* Have to remmber to update the low number of pages * too */
@@ -724,14 +731,16 @@ void ttm_put_pages(struct list_head *pages, int flags, round_jiffies(_manager.free_interval)); } -static void ttm_page_pool_init_locked (struct ttm_page_pool *pool,int flags) +static void ttm_page_pool_init_locked (struct ttm_page_pool *pool,int flags, +char *name) {spin_lock_init (&pool->lock); pool->fill_lock = false;INIT_LIST_HEAD (&pool->list); - pool->npages = pool->nlowpages =0 ; + pool->npages = pool->nlowpages = pool->nfrees =0 ; pool->alloc_size =NUM_PAGES_TO_ALLOC ; pool->gfp_flags = flags; + pool->name = name; }int ttm_page_alloc_init (struct ttm_mem_global *glob)
@@ -741,13 +750,15 @@ int ttm_page_alloc_init(struct ttm_mem_global *glob)printk (KERN_INFO "[ttm] Initializing pool allocator.\n" ); -ttm_page_pool_init_locked (&_manager.wc_pool,GFP_HIGHUSER ); +ttm_page_pool_init_locked (&_manager.wc_pool,GFP_HIGHUSER ,"wc" ); -ttm_page_pool_init_locked (&_manager.uc_pool,GFP_HIGHUSER ); +ttm_page_pool_init_locked (&_manager.uc_pool,GFP_HIGHUSER ,"uc" ); -ttm_page_pool_init_locked (&_manager.wc_pool_dma32,GFP_USER |GFP_DMA32 ); +ttm_page_pool_init_locked (&_manager.wc_pool_dma32,GFP_USER |GFP_DMA32 , +"wc dma" ); -ttm_page_pool_init_locked (&_manager.uc_pool_dma32,GFP_USER |GFP_DMA32 ); +ttm_page_pool_init_locked (&_manager.uc_pool_dma32,GFP_USER |GFP_DMA32 , +"uc dma" ); _manager.free_interval =msecs_to_jiffies (PAGE_FREE_INTERVAL ); _manager.small_allocation =SMALL_ALLOCATION ;
@@ -773,3 +784,26 @@ void ttm_page_alloc_fini()for (i =0 ; i <NUM_POOLS ; ++i)ttm_page_pool_free (&_manager.pools[i],FREE_ALL_PAGES ); } + +int ttm_page_alloc_debugfs (struct seq_file *m,void *data) +{ +struct ttm_page_pool *p; +unsigned i; +char *h[] = {"pool" ,"refills" ,"pages freed" ,"size" ,"min size" }; +if (atomic_read (&_manager.page_alloc_inited) ==0 ) { +seq_printf (m,"No pool allocator running.\n" ); +return 0 ; + } +seq_printf (m,"%6s %12s %13s %8s %8s\n" , + h[0 ], h[1 ], h[2 ], h[3 ], h[4 ]); +for (i =0 ; i <NUM_POOLS ; ++i) { + p = &_manager.pools[i]; + +seq_printf (m,"%6s %12ld %13ld %8d %8d\n" , + p->name, p->nrefills, + p->nfrees, p->npages, + p->nlowpages); + } +return 0 ; +} +EXPORT_SYMBOL (ttm_page_alloc_debugfs);
include/drm/ttm/ttm_page_alloc.h +4 -0
@@ -61,4 +61,8 @@ int ttm_page_alloc_init(struct ttm_mem_global *glob);*/ void ttm_page_alloc_fini (void ); +/** +* Output the state of pools to debugfs file +*/ +extern int ttm_page_alloc_debugfs (struct seq_file *m,void *data);#endif
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 3/7 drm/radeon/kms: Add ttm page pool debugfs file.
ttm_page_pool file is hooked ttm_page_alloc_debugfs for pool allocator state.
--- drivers/gpu/drm/radeon/radeon_ttm.c | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-)
drivers/gpu/drm/radeon/radeon_ttm.c +10 -3
@@ -33,6 +33,7 @@#include <ttm/ttm_bo_driver.h> #include <ttm/ttm_placement.h> #include <ttm/ttm_module.h> +#include <ttm/ttm_page_alloc.h> #include <drm/drmP.h> #include <drm/radeon_drm.h> #include <linux/seq_file.h>
@@ -744,8 +745,8 @@ static int radeon_mm_dump_table(struct seq_file *m, void *data)static int radeon_ttm_debugfs_init (struct radeon_device *rdev) {#if defined(CONFIG_DEBUG_FS ) -static struct drm_info_list radeon_mem_types_list[RADEON_DEBUGFS_MEM_TYPES ]; -static char radeon_mem_types_names[RADEON_DEBUGFS_MEM_TYPES ][32 ]; +static struct drm_info_list radeon_mem_types_list[RADEON_DEBUGFS_MEM_TYPES +1 ]; +static char radeon_mem_types_names[RADEON_DEBUGFS_MEM_TYPES +1 ][32 ];unsigned i;for (i =0 ; i <RADEON_DEBUGFS_MEM_TYPES ; i++) {
@@ -762,7 +763,13 @@ static int radeon_ttm_debugfs_init(struct radeon_device *rdev) radeon_mem_types_list[i].data = &rdev->mman.bdev.man[TTM_PL_TT].manager; } -return radeon_debugfs_add_files (rdev, radeon_mem_types_list,RADEON_DEBUGFS_MEM_TYPES ); +/* Add ttm page pool to debugfs */ +sprintf (radeon_mem_types_names[i],"ttm_page_pool" ); + radeon_mem_types_list[i].name = radeon_mem_types_names[i]; + radeon_mem_types_list[i].show = &ttm_page_alloc_debugfs; + radeon_mem_types_list[i].driver_features =0 ; + radeon_mem_types_list[i].data =NULL ; +return radeon_debugfs_add_files (rdev, radeon_mem_types_list,RADEON_DEBUGFS_MEM_TYPES +1 );#endif return 0 ;
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 4/7 drm/nouveau: Add ttm page pool debugfs file.
ttm_page_pool file is hooked ttm_page_alloc_debugfs for pool allocator state.
--- drivers/gpu/drm/nouveau/nouveau_debugfs.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
drivers/gpu/drm/nouveau/nouveau_debugfs.c +3 -0
@@ -33,6 +33,8 @@#include "drmP.h" #include "nouveau_drv.h" +#include <ttm/ttm_page_alloc.h> +static int nouveau_debugfs_channel_info (struct seq_file *m,void *data) {
@@ -160,6 +162,7 @@ static struct drm_info_list nouveau_debugfs_list[] = { { "chipset", nouveau_debugfs_chipset_info, 0, NULL }, { "memory", nouveau_debugfs_memory_info, 0, NULL }, { "vbios.rom", nouveau_debugfs_vbios_image, 0, NULL }, + { "ttm_page_pool", ttm_page_alloc_debugfs, 0, NULL }, };#define NOUVEAU_DEBUGFS_ENTRIES ARRAY_SIZE(nouveau_debugfs_list)
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 5/7 drm/ttm: Dynamically scale the allocation sizes of pools.
If we need large number of pages allocations in short time frame allocating large number of pages in single operation performs better than many small operations. If there is less allocation operations allocating large number of pages in a fill operation would waste memory and performance.
The allocation sizes of pools is doubled everytime refill is done more than once between shrink calls. While the size is halfed happens if there is no refills for manager->shrink_alloc_interval seconds.
--- drivers/gpu/drm/ttm/ttm_page_alloc.c | 105 ++++++++++++++++++++++++++-------- 1 files changed, 81 insertions(+), 24 deletions(-)
drivers/gpu/drm/ttm/ttm_page_alloc.c +81 -24
@@ -47,8 +47,9 @@#define NUM_PAGES_TO_ALLOC 256 -#define SMALL_ALLOCATION 64 +#define SMALL_ALLOCATION 16#define FREE_ALL_PAGES 1 +#define SHRINK_ALLOC_INTERVAL 8/* times are in msecs */ #define PAGE_FREE_INTERVAL 1000
@@ -63,7 +64,10 @@* @gfp_flags: Flags to pass for alloc_page. * @npages: Number of pages in pool * @nlowpages: Minimum nubmer of pages in pool since previous shrink +* @freed_pages: Number of pages freed in this shrinker run. * @alloc_size: Allocation sizes of this pool. +* @last_refill: Number of shrink calls (with more than alloc_size pages free) +* since last refill call. * operation. */ struct ttm_page_pool {
@@ -73,7 +77,9 @@ struct ttm_page_pool {int gfp_flags;unsigned npages;unsigned nlowpages; +unsigned freed_pages;unsigned alloc_size; +unsigned last_refill;char *name;unsigned long nfrees;unsigned long nrefills;
@@ -101,6 +107,7 @@ struct ttm_pool_manager {atomic_t page_alloc_inited;struct delayed_work work;unsigned small_allocation; +unsigned shrink_alloc_interval;union {struct ttm_page_pool pools[NUM_POOLS ];
@@ -199,17 +206,33 @@ static void ttm_pages_put(struct page *pages[], unsigned npages)**/ static bool ttm_reset_pools (struct ttm_pool_manager *manager) { +struct ttm_page_pool *pool;unsigned long irq_flags; -bool pages_in_pool = false; +bool more_delayed_work = false;unsigned i;for (i =0 ; i <NUM_POOLS ; ++i) { -spin_lock_irqsave (&manager->pools[i].lock, irq_flags); - manager->pools[i].nlowpages = manager->pools[i].npages; - pages_in_pool = pages_in_pool - || manager->pools[i].npages > manager->pools[i].alloc_size; -spin_unlock_irqrestore (&manager->pools[i].lock, irq_flags); + pool = &manager->pools[i]; +spin_lock_irqsave (&pool->lock, irq_flags); +/** +* Check if pool has shrink work in a second. +**/ +if (pool->alloc_size > manager->small_allocation) { +/* Reduce pool sizes if there is no refills for +* manager->shrink_alloc_interval seconds. */ +if (++pool->last_refill % manager->shrink_alloc_interval ==0 ) + pool->alloc_size /=2 ; + + more_delayed_work = true; + }else + more_delayed_work = more_delayed_work + || pool->npages > pool->alloc_size; + + pool->nlowpages = pool->npages; + pool->freed_pages =0 ; + +spin_unlock_irqrestore (&pool->lock, irq_flags); } -return pages_in_pool; +return more_delayed_work; }/**
@@ -220,16 +243,19 @@ static bool ttm_reset_pools(struct ttm_pool_manager *manager)static unsigned ttm_page_pool_get_npages_to_free_locked (struct ttm_page_pool *pool) {unsigned r; +unsigned low = pool->nlowpages - pool->freed_pages; +if (pool->nlowpages < pool->freed_pages) + low =0 ;/* If less than alloc sizes was the lowest number of pages we don't * free any */ -if (pool->nlowpages < pool->alloc_size) +if (low < pool->alloc_size)return 0 ;/* leave half of unused pages to pool */ - r = (pool->nlowpages - pool->alloc_size)/2 ; + r = (low - pool->alloc_size)/2 ;if (r)return r;/* make sure we remove all pages even when there is rounding down */ -if (pool->nlowpages) +if (low)return 1 ;return 0 ; }
@@ -251,12 +277,12 @@ static bool ttm_page_pool_free_pages_locked(struct ttm_page_pool *pool,*/ tmp =2 *freed_pages;/* protect against rounding errors */ -if (tmp < pool->nlowpages) { - pool->nlowpages -= tmp; +if (tmp < pool->nlowpages - pool->freed_pages) { + pool->freed_pages += tmp;return true; } - pool->nlowpages =0 ; + pool->freed_pages = pool->nlowpages;return false; }
@@ -329,7 +355,7 @@ restart:/* set nlowpages to zero to prevent extra freeing in thsi patch. * nlowpages is reseted later after all work has been finnished. **/ - pool->nlowpages =0 ; + pool->freed_pages = pool->nlowpages;/* remove range of pages from the pool */ if (freed_pages)
@@ -516,6 +542,8 @@ static void ttm_page_pool_fill_locked(struct ttm_page_pool *pool,struct page *p, *tmp;int r;unsigned cpages =0 ; +unsigned alloc_size; +bool queue_shrink = false;/** * Only allow one pool fill operation at a time. * If pool doesn't have enough pages for the allocation new pages are
@@ -528,6 +556,24 @@ static void ttm_page_pool_fill_locked(struct ttm_page_pool *pool,if (count < _manager.small_allocation && count > pool->npages) { +/* Increase allocation sizes if we are refilling pool multiple +* times between shrink calls. Clamp alloc size of the pool +* to NUM_PAGES_TO_ALLOC * 2. */ +if (pool->last_refill ==0 + && pool->alloc_size <NUM_PAGES_TO_ALLOC *2 ) { + pool->alloc_size *=2 ; + queue_shrink = true; + } + + pool->last_refill =0 ; + alloc_size = pool->alloc_size; + +/* clamp size to NUM_PAGES_TO_ALLOC because it is maximum +* number of pages in a single caching change. +*/ +if (alloc_size >NUM_PAGES_TO_ALLOC ) + alloc_size =NUM_PAGES_TO_ALLOC ; +/* If allocation request is small and there is not enough * pages in pool we fill the pool first */ INIT_LIST_HEAD (&new_pages);
@@ -538,16 +584,20 @@ static void ttm_page_pool_fill_locked(struct ttm_page_pool *pool,*/ spin_unlock_irqrestore (&pool->lock, irq_flags); r =ttm_alloc_new_pages (&new_pages, pool->gfp_flags, ttm_flags, - cstate, pool->alloc_size); + cstate, alloc_size); +if (queue_shrink) + (void )queue_delayed_work (_manager.glob->swap_queue, + &_manager.work, +round_jiffies (_manager.free_interval));spin_lock_irqsave (&pool->lock, irq_flags);if (!r) {list_splice (&new_pages, &pool->list); ++pool->nrefills; - pool->npages += pool->alloc_size; + pool->npages += alloc_size;/* Have to remmber to update the low number of pages * too */ - pool->nlowpages += pool->alloc_size; + pool->nlowpages += alloc_size; }else {printk (KERN_ERR "[ttm] Failed to fill pool (%p)." , pool);/* If we have any pages left put them to the pool. */
@@ -738,7 +788,9 @@ static void ttm_page_pool_init_locked(struct ttm_page_pool *pool, int flags, pool->fill_lock = false;INIT_LIST_HEAD (&pool->list); pool->npages = pool->nlowpages = pool->nfrees =0 ; - pool->alloc_size =NUM_PAGES_TO_ALLOC ; + pool->last_refill =0 ; + pool->freed_pages =0 ; + pool->alloc_size =SMALL_ALLOCATION ; pool->gfp_flags = flags; pool->name = name; }
@@ -762,10 +814,14 @@ int ttm_page_alloc_init(struct ttm_mem_global *glob) _manager.free_interval =msecs_to_jiffies (PAGE_FREE_INTERVAL ); _manager.small_allocation =SMALL_ALLOCATION ; + _manager.shrink_alloc_interval =SHRINK_ALLOC_INTERVAL ; _manager.glob = glob;INIT_DELAYED_WORK (&_manager.work, ttm_pool_shrink); + (void )queue_delayed_work (_manager.glob->swap_queue, + &_manager.work, +round_jiffies (_manager.free_interval));return 0 ; }
@@ -789,20 +845,21 @@ int ttm_page_alloc_debugfs(struct seq_file *m, void *data) {struct ttm_page_pool *p;unsigned i; -char *h[] = {"pool" ,"refills" ,"pages freed" ,"size" ,"min size" }; +char *h[] = {"pool" ,"refills" ,"pages freed" ,"size" ,"min size" , +"alloc size" };if (atomic_read (&_manager.page_alloc_inited) ==0 ) {seq_printf (m,"No pool allocator running.\n" );return 0 ; } -seq_printf (m,"%6s %12s %13s %8s %8s\n" , - h[0 ], h[1 ], h[2 ], h[3 ], h[4 ]); +seq_printf (m,"%6s %12s %13s %8s %8s %8s\n" , + h[0 ], h[1 ], h[2 ], h[3 ], h[4 ], h[5 ]);for (i =0 ; i <NUM_POOLS ; ++i) { p = &_manager.pools[i]; -seq_printf (m,"%6s %12ld %13ld %8d %8d\n" , +seq_printf (m,"%6s %12ld %13ld %8d %8d %8d\n" , p->name, p->nrefills, p->nfrees, p->npages, - p->nlowpages); + p->nlowpages, p->alloc_size); }return 0 ; }
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 6/7 arch/x86: Add array variants for setting memory to wc caching.
Setting single memory pages at a time to wc takes a lot time in cache flush. To reduce number of cache flush set_pages_array_wc and set_memory_array_wc can be used to set multiple pages to WC with single cache flush.
This improves allocation performance for wc cached pages in drm/ttm.
--- arch/x86/include/asm/cacheflush.h | 2 + arch/x86/mm/pageattr.c | 53 +++++++++++++++++++++++++++++++----- 2 files changed, 47 insertions(+), 8 deletions(-)
arch/x86/include/asm/cacheflush.h +2 -0
@@ -139,9 +139,11 @@ int set_memory_np(unsigned long addr, int numpages);int set_memory_4k (unsigned long addr,int numpages);int set_memory_array_uc (unsigned long *addr,int addrinarray); +int set_memory_array_wc (unsigned long *addr,int addrinarray);int set_memory_array_wb (unsigned long *addr,int addrinarray);int set_pages_array_uc (struct page **pages,int addrinarray); +int set_pages_array_wc (struct page **pages,int addrinarray);int set_pages_array_wb (struct page **pages,int addrinarray);/*
arch/x86/mm/pageattr.c +45 -8
107 lines not on this page · raw message
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
PATCH 7/7 drm/ttm: Use set_pages_array_wc instead of set_memory_wc.
Using single call to set multiple pages to wc reduces number of expensive cache flushes.
--- drivers/gpu/drm/ttm/ttm_page_alloc.c | 12 +----------- 1 files changed, 1 insertions(+), 11 deletions(-)
drivers/gpu/drm/ttm/ttm_page_alloc.c +1 -11
@@ -124,17 +124,7 @@ static struct ttm_pool_manager _manager = { .page_alloc_inited = ATOMIC_INIT(0) }; -#ifdef CONFIG_X86 -/* TODO: add this to x86 like _uc, this version here is inefficient */ -static int set_pages_array_wc (struct page **pages,int addrinarray) -{ -int i; - -for (i =0 ; i < addrinarray; i++) -set_memory_wc ((unsigned long )page_address (pages[i]),1 ); -return 0 ; -} -#else +#ifndef CONFIG_X86 static int set_pages_array_wb (struct page **pages,int addrinarray) {#ifdef TTM_HAS_AGP
Signature · 10 lines
1.6.3.3 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --
Dave Airlie
maintainerRe: [PATCH 6/7] arch/x86: Add array variants for setting memory to wc caching.
In reply to Pauli NieminenQuoted 3 lines · Pauli Nieminen
Setting single memory pages at a time to wc takes a lot time in cache flush. To reduce number of cache flush
set_pages_array_wcandset_memory_array_wccan be used to set multiple pages to WC with single cache flush.
I don't think this is correct, I've cc'ed Suresh and Venki who looked at this before,
but I think there is an array in the x86 code that stores the state and it only has a
single bit in it, it needs to be expanded in order to at WC support.
Dave.
Quoted 142 lines
This improves allocation performance for wc cached pages in drm/ttm.
Signed-off-by Pauli Nieminen <[email protected]>---
arch/x86/include/asm/cacheflush.h | 2 +arch/x86/mm/pageattr.c| 53 +++++++++++++++++++++++++++++++-----2 files changed, 47 insertions(+), 8 deletions(-)arch/x86/include/asm/cacheflush.h +2
-0@@ -139,9 +139,11 @@ int set_memory_np(unsigned long addr, int numpages); int set_memory_4k(unsigned long addr, int numpages); int set_memory_array_uc(unsigned long *addr, int addrinarray); +int set_memory_array_wc (unsigned long *addr,int addrinarray); int set_memory_array_wb(unsigned long *addr, int addrinarray); int set_pages_array_uc(struct page **pages, int addrinarray); +int set_pages_array_wc (struct page **pages,int addrinarray); int set_pages_array_wb(struct page **pages, int addrinarray); /*arch/x86/mm/pageattr.c +34
-8@@ -997,7 +997,8 @@ out_err: } EXPORT_SYMBOL(set_memory_uc); -int set_memory_array_uc (unsigned long *addr,int addrinarray) +int _set_memory_array (unsigned long *addr,int addrinarray, + unsigned long new_type) { int i, j; int ret;@@ -1007,13 +1008,19 @@ int set_memory_array_uc(unsigned long *addr, int addrinarray)*/ for (i = 0; i < addrinarray; i++) { ret = reserve_memtype(__pa(addr[i]), __pa(addr[i]) + PAGE_SIZE, - _PAGE_CACHE_UC_MINUS, NULL); + new_type, NULL); if (ret) goto out_free; } ret = change_page_attr_set(addr, addrinarray, __pgprot(_PAGE_CACHE_UC_MINUS), 1); + + if (!ret && new_type == _PAGE_CACHE_WC) + ret = change_page_attr_set_clr(addr, addrinarray, + __pgprot(_PAGE_CACHE_WC), + __pgprot(_PAGE_CACHE_MASK), + 0, CPA_ARRAY, NULL); if (ret) goto out_free;@@ -1025,8 +1032,19 @@ out_free: return ret; } + +int set_memory_array_uc (unsigned long *addr,int addrinarray) +{ + return _set_memory_array(addr, addrinarray, _PAGE_CACHE_UC_MINUS); +} EXPORT_SYMBOL(set_memory_array_uc); +int set_memory_array_wc (unsigned long *addr,int addrinarray) +{ + return _set_memory_array(addr, addrinarray, _PAGE_CACHE_WC); +} +EXPORT_SYMBOL (set_memory_array_wc); + int _set_memory_wc(unsigned long addr, int numpages) { int ret;@@ -1153,26 +1171,34 @@ int set_pages_uc(struct page *page, int numpages) } EXPORT_SYMBOL(set_pages_uc); -int set_pages_array_uc (struct page **pages,int addrinarray) +static int _set_pages_array (struct page **pages,int addrinarray, + unsigned long new_type) { unsigned long start; unsigned long end; int i; int free_idx; + int ret; for (i = 0; i < addrinarray; i++) { if (PageHighMem(pages[i])) continue; start = page_to_pfn(pages[i]) << PAGE_SHIFT; end = start + PAGE_SIZE; - if (reserve_memtype(start, end, _PAGE_CACHE_UC_MINUS, NULL)) + if (reserve_memtype(start, end, new_type, NULL)) goto err_out; } - if (cpa_set_pages_array(pages, addrinarray, - __pgprot(_PAGE_CACHE_UC_MINUS)) == 0) { - return 0; /* Success */ - } + ret = cpa_set_pages_array(pages, addrinarray, + __pgprot(_PAGE_CACHE_UC_MINUS)); + if (!ret && new_type == _PAGE_CACHE_WC) + ret = change_page_attr_set_clr(NULL, addrinarray, + __pgprot(_PAGE_CACHE_WC), + __pgprot(_PAGE_CACHE_MASK), + 0, CPA_PAGES_ARRAY, pages); + if (ret) + goto err_out; + return 0; /* Success */err_out:
free_idx = i;
for (i = 0; i <free_idx; i++) {arch/x86/mm/pageattr.c +11
-1@@ -1184,8 +1210,19 @@ err_out: } return -EINVAL; } + +int set_pages_array_uc (struct page **pages,int addrinarray) +{ + return _set_pages_array(pages, addrinarray, _PAGE_CACHE_UC_MINUS); +} EXPORT_SYMBOL(set_pages_array_uc); +int set_pages_array_wc (struct page **pages,int addrinarray) +{ + return _set_pages_array(pages, addrinarray, _PAGE_CACHE_WC); +} +EXPORT_SYMBOL (set_pages_array_wc); + int set_pages_wb(struct page *page, int numpages) { unsigned long addr = (unsigned long)page_address(page); --
Quoted 3 lines
1.6.3.3
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
Re: [PATCH 6/7] arch/x86: Add array variants for setting memory to wc caching.
Quoted 5 lines · Dave Airlie
Quoted 3 lines · Pauli Nieminen
Setting single memory pages at a time to wc takes a lot time in cache flush. To reduce number of cache flush
set_pages_array_wcandset_memory_array_wccan be used to set multiple pages to WC with single cache flush.
Quoted 8 lines
I don't think this is correct, I've cc'ed Suresh and Venki who looked at this before,
but I think there is an array in the x86 code that stores the state and it only has a
single bit in it, it needs to be expanded in order to at WC support.Dave.
This worked for me but there might be some hw specific cases when it doesn't work. (all page act like wc cached ones when running memcpy test on gtt bo)
This is only minor improvemnet for corner cases. Pool refills general don't happen in normal use except for 3D. For others cases number of pages was fairly static so pool code can avoid refills. Of course there might be some corner case that I didn't know to test. Even for them pool refills happened about once a second in average. But allocation were happening in small bursts so this should avoid frame rate hit in case of a lot of allocation in very short time.
Quoted 132 lines
Quoted 132 lines
This improves allocation performance for wc cached pages in drm/ttm.
Signed-off-by Pauli Nieminen <[email protected]>---
arch/x86/include/asm/cacheflush.h | 2 +arch/x86/mm/pageattr.c| 53 +++++++++++++++++++++++++++++++-----2 files changed, 47 insertions(+), 8 deletions(-)arch/x86/include/asm/cacheflush.h +2
-0@@ -139,9 +139,11 @@ int set_memory_np(unsigned long addr, int numpages); int set_memory_4k(unsigned long addr, int numpages); int set_memory_array_uc(unsigned long *addr, int addrinarray); +int set_memory_array_wc (unsigned long *addr,int addrinarray); int set_memory_array_wb(unsigned long *addr, int addrinarray); int set_pages_array_uc(struct page **pages, int addrinarray); +int set_pages_array_wc (struct page **pages,int addrinarray); int set_pages_array_wb(struct page **pages, int addrinarray); /*arch/x86/mm/pageattr.c +34
-8@@ -997,7 +997,8 @@ out_err: } EXPORT_SYMBOL(set_memory_uc); -int set_memory_array_uc (unsigned long *addr,int addrinarray) +int _set_memory_array (unsigned long *addr,int addrinarray, + unsigned long new_type) { int i, j; int ret;@@ -1007,13 +1008,19 @@ int set_memory_array_uc(unsigned long *addr, int addrinarray)*/ for (i = 0; i < addrinarray; i++) { ret = reserve_memtype(__pa(addr[i]), __pa(addr[i]) + PAGE_SIZE, - _PAGE_CACHE_UC_MINUS, NULL); + new_type, NULL); if (ret) goto out_free; } ret = change_page_attr_set(addr, addrinarray, __pgprot(_PAGE_CACHE_UC_MINUS), 1); + + if (!ret && new_type == _PAGE_CACHE_WC) + ret = change_page_attr_set_clr(addr, addrinarray, + __pgprot(_PAGE_CACHE_WC), + __pgprot(_PAGE_CACHE_MASK), + 0, CPA_ARRAY, NULL); if (ret) goto out_free;@@ -1025,8 +1032,19 @@ out_free: return ret; } + +int set_memory_array_uc (unsigned long *addr,int addrinarray) +{ + return _set_memory_array(addr, addrinarray, _PAGE_CACHE_UC_MINUS); +} EXPORT_SYMBOL(set_memory_array_uc); +int set_memory_array_wc (unsigned long *addr,int addrinarray) +{ + return _set_memory_array(addr, addrinarray, _PAGE_CACHE_WC); +} +EXPORT_SYMBOL (set_memory_array_wc); + int _set_memory_wc(unsigned long addr, int numpages) { int ret;@@ -1153,26 +1171,34 @@ int set_pages_uc(struct page *page, int numpages) } EXPORT_SYMBOL(set_pages_uc); -int set_pages_array_uc (struct page **pages,int addrinarray) +static int _set_pages_array (struct page **pages,int addrinarray, + unsigned long new_type) { unsigned long start; unsigned long end; int i; int free_idx; + int ret; for (i = 0; i < addrinarray; i++) { if (PageHighMem(pages[i])) continue; start = page_to_pfn(pages[i]) << PAGE_SHIFT; end = start + PAGE_SIZE; - if (reserve_memtype(start, end, _PAGE_CACHE_UC_MINUS, NULL)) + if (reserve_memtype(start, end, new_type, NULL)) goto err_out; } - if (cpa_set_pages_array(pages, addrinarray, - __pgprot(_PAGE_CACHE_UC_MINUS)) == 0) { - return 0; /* Success */ - } + ret = cpa_set_pages_array(pages, addrinarray, + __pgprot(_PAGE_CACHE_UC_MINUS)); + if (!ret && new_type == _PAGE_CACHE_WC) + ret = change_page_attr_set_clr(NULL, addrinarray, + __pgprot(_PAGE_CACHE_WC), + __pgprot(_PAGE_CACHE_MASK), + 0, CPA_PAGES_ARRAY, pages); + if (ret) + goto err_out; + return 0; /* Success */err_out:
free_idx = i;
for (i = 0; i <free_idx; i++) {arch/x86/mm/pageattr.c +5
-0@@ -1184,8 +1210,19 @@ err_out: } return -EINVAL; } + +int set_pages_array_uc (struct page **pages,int addrinarray) +{ + return _set_pages_array(pages, addrinarray, _PAGE_CACHE_UC_MINUS); +} EXPORT_SYMBOL(set_pages_array_uc);
Quoted 14 lines
Quoted 10 lines
arch/x86/mm/pageattr.c +6
-1+int set_pages_array_wc (struct page **pages,int addrinarray) +{ + return _set_pages_array(pages, addrinarray, _PAGE_CACHE_WC); +} +EXPORT_SYMBOL (set_pages_array_wc); + int set_pages_wb(struct page *page, int numpages) { unsigned long addr = (unsigned long)page_address(page); --Quoted 3 lines
1.6.3.3
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
Re: [PATCH 6/7] arch/x86: Add array variants for setting memory to wc caching.
Quoted 6 lines · Pauli Nieminen
Quoted 5 lines · Dave Airlie
Quoted 3 lines · Pauli Nieminen
Setting single memory pages at a time to wc takes a lot time in cache flush. To reduce number of cache flush
set_pages_array_wcandset_memory_array_wccan be used to set multiple pages to WC with single cache flush.
Quoted 6 lines
Quoted 6 lines
I don't think this is correct, I've cc'ed Suresh and Venki who looked at this before,
but I think there is an array in the x86 code that stores the state and it only has a
single bit in it, it needs to be expanded in order to at WC support.
hmm, I thought we had the array variants for the WC already. I don't think there is any explicit reason for not having it. Correcting Venki's address, to see if he has anything to add.
Quoted 7 lines
Quoted 7 lines
arch/x86/mm/pageattr.c int set_pages_uc(struct page *page, int numpages) · line 1198
Quoted 7 lines
arch/x86/mm/pageattr.c +7
-0+ ret =cpa_set_pages_array (pages, addrinarray, +__pgprot (_PAGE_CACHE_UC_MINUS)); +if (!ret && new_type == _PAGE_CACHE_WC) + ret =change_page_attr_set_clr (NULL , addrinarray, +__pgprot (_PAGE_CACHE_WC), +__pgprot (_PAGE_CACHE_MASK), +0 ,CPA_PAGES_ARRAY , pages);
Pauli, Is there any reason for not using cpa_set_pages_array() here aswell?
thanks,
suresh
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
Re: [PATCH 6/7] arch/x86: Add array variants for setting memory to wc caching.
Quoted 26 lines · Suresh Siddha
Quoted 6 lines · Pauli Nieminen
Quoted 5 lines · Dave Airlie
Quoted 3 lines · Pauli Nieminen
Setting single memory pages at a time to wc takes a lot time in cache flush. To reduce number of cache flush
set_pages_array_wcandset_memory_array_wccan be used to set multiple pages to WC with single cache flush.Quoted 6 lines
Quoted 6 lines
I don't think this is correct, I've cc'ed Suresh and Venki who looked at this before,
but I think there is an array in the x86 code that stores the state and it only has a
single bit in it, it needs to be expanded in order to at WC support.hmm, I thought we had the array variants for the WC already. I don't think there is any explicit reason for not having it. Correcting Venki's address, to see if he has anything to add.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
+ ret = cpa_set_pages_array(pages, addrinarray,
+__pgprot(_PAGE_CACHE_UC_MINUS));
+ if (!ret &&new_type==_PAGE_CACHE_WC)
+ ret = change_page_attr_set_clr(NULL, addrinarray,
+ __pgprot(_PAGE_CACHE_WC),
+ __pgprot(_PAGE_CACHE_MASK),
+ 0,CPA_PAGES_ARRAY, pages);
Quoted 3 lines
Pauli, Is there any reason for not using
cpa_set_pages_array()here aswell?
cpa_set_pages_array is setting mask parameter to zero while _set_memory_wc is passing the mask parameter. I didn't look deep enough to the code to know why _set_memory_wc works that way.
Quoted 4 lines
thanks,
suresh
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
A message that is not in the archive (sent off-list or not yet received): [email protected]
2010/3/19 Thomas Hellström <[email protected]>:
Quoted 8 lines
Pauli, Dave and Jerome,
Before reviewing this, Could you describe a bit how this interfaces with the TTM memory accounting. It's important for some systems to be able to set a limit beyond which TTM may not pin any pages.
Am I right in assuming that TTM memory accounting is kicked in only when TTM allocs and frees pages from the pool?
yes.
TTM memory accounting is still handled in ttm_tt.c so pool is outside of it. But I can move calls to memory accounting into pool if that is preferred place.
With current implementation pool can hold about 512 pages more memory than what is the TTM limit.
Quoted 2 lines
Can the system reclaim *all* pages not used by TTM through a shrink mechanism?
Not with current version but I can modify patch so that system can reclaim all pages. Current lowest limit is 16 pages in pool because that avoids refills for 2D only desktop use.
The limit is already changed in runtime so making it scale to zero sized pool is only minor change. But what should happen for pool refill if system just before forced the pool size to zero?
Quoted 5 lines
In the long run, I'd like to have a pool of non-kernel-mapped pages instead of a pool of uncached / write-combined pages, because then we'd have quite fast transition from write-combined to write-back, but I guess that will be something for the future.
I think this can be simulated with multiple pools if free logic is changed from handling single pool at a time to combine multiple pools to a single wb transition operation.
Trouble in making very large cache transition operations is that allocating large continues arrays in kernel is problematic. Current code is limiting the size of single cache transition operation to avoid possible memory allocation problems.
Quoted 44 lines
/Thomas
Quoted 33 lines · Pauli Nieminen
When allocating wc/uc pages cache state transition requires cache flush which
is expensive operation. To avoid cache flushes allocation of wc/uc pages should
be done in large groups when only single cache flush is required for whole group
of pages.In some cases drivers need t oallocate and deallocate many pages in a short time
frame. In this case we can avoid cache flushes if we keep pages in the pool before
actually freeing them later.
arch/x86was missingset_pages_array_wcandset_memory_array_wc. Patch 6 and 7 add
missing functions and hooksset_pages_array_wcto the pool allocator.------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
_______________________________________________
Dri-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dri-devel
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--
Quoted 12 lines · Pauli Nieminen
2010/3/19 Thomas Hellström <[email protected]>:
Quoted 9 lines
Pauli, Dave and Jerome,
Before reviewing this, Could you describe a bit how this interfaces with the TTM memory accounting. It's important for some systems to be able to set a limit beyond which TTM may not pin any pages.
Am I right in assuming that TTM memory accounting is kicked in only when TTM allocs and frees pages from the pool?
Quoted 10 lines
yes.
TTM memory accounting is still handled in
ttm_tt.cso pool is outside of it. But I can move calls to memory accounting into pool if that is preferred place.With current implementation pool can hold about 512 pages more memory than what is the TTM limit.
In that case I think we should keep the pool outside of TTM memory accounting.
Since the system can (or should be able to) reclaim the pool pages through the shrink mechanism, the
pool is sort of transparent from a memory accounting point of view although shrinking comes with a performance penalty.
Quoted 4 lines
Quoted 3 lines
Can the system reclaim *all* pages not used by TTM through a shrink mechanism?
Quoted 8 lines
Not with current version but I can modify patch so that system can reclaim all pages. Current lowest limit is 16 pages in pool because that avoids refills for 2D only desktop use.
The limit is already changed in runtime so making it scale to zero sized pool is only minor change. But what should happen for pool refill if system just before forced the pool size to zero?
16 pages sounds OK to me.
Perhaps we'd want to be able to change both this limit and the upper pool size limit from a sysfs interface, within ttm/memory_accounting ?
Quoted 7 lines
Quoted 6 lines
In the long run, I'd like to have a pool of non-kernel-mapped pages instead of a pool of uncached / write-combined pages, because then we'd have quite fast transition from write-combined to write-back, but I guess that will be something for the future.
Quoted 9 lines
I think this can be simulated with multiple pools if free logic is changed from handling single pool at a time to combine multiple pools to a single wb transition operation.
Trouble in making very large cache transition operations is that allocating large continues arrays in kernel is problematic. Current code is limiting the size of single cache transition operation to avoid possible memory allocation problems.
I think that's a good strategy.
However, the use-case I was thinking of was not fast freeing, but fast reading of GPU memory, like so:
*) Unbind from AGP.
*) Mark pages write-back() while the kernel map is still 'unmapped' (just like highmem pages).
*) Read
*) Cache flush pages and Mark pages write-combined(), while the kernel map is still 'unmapped'.
The operation will not involve any 'set_pages_x' , because the kernel map is not affected.
However, there is no exported api to unmap pages from the linear kernel map (yet).
/Thomas
Quoted 27 lines
Quoted 27 lines
/Thomas
Quoted 20 lines · Pauli Nieminen
When allocating wc/uc pages cache state transition requires cache flush which
is expensive operation. To avoid cache flushes allocation of wc/uc pages should
be done in large groups when only single cache flush is required for whole group
of pages.In some cases drivers need t oallocate and deallocate many pages in a short time
frame. In this case we can avoid cache flushes if we keep pages in the pool before
actually freeing them later.
arch/x86was missingset_pages_array_wcandset_memory_array_wc. Patch 6 and 7 add
missing functions and hooksset_pages_array_wcto the pool allocator.
Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.Quoted 11 lines
Quoted 8 lines
http://p.sf.net/sfu/intel-sw-dev
--
_______________________________________________
Dri-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dri-devel
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
--