From: Jerome Glisse Subject: Re: [RFC] drm/ttm: add pool wc/uc page allocator Date: Wed, 3 Mar 2010 15:50:46 +0100 Message-ID: <20100303145046.GD2078@barney.localdomain> References: <1267392603-8915-1-git-send-email-suokkos@gmail.com> <548cdfc21003011432o3c6103dbj3e9d29d8f42609c3@mail.gmail.com> <20100303133300.GB2078@barney.localdomain> <548cdfc21003030623t2ec356bbob6609d8b86fd13c0@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Return-path: Content-Disposition: inline In-Reply-To: <548cdfc21003030623t2ec356bbob6609d8b86fd13c0@mail.gmail.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.sourceforge.net To: Pauli Nieminen Cc: Jerome Glisse , dri-devel@lists.sourceforge.net List-Id: dri-devel@lists.freedesktop.org On Wed, Mar 03, 2010 at 04:23:08PM +0200, Pauli Nieminen wrote: > On Wed, Mar 3, 2010 at 3:33 PM, Jerome Glisse wr= ote: > > On Tue, Mar 02, 2010 at 12:32:54AM +0200, Pauli Nieminen wrote: > >> On Sun, Feb 28, 2010 at 11:30 PM, Pauli Nieminen w= rote: > >> > On AGP system we might allocate/free routinely uncached or wc memory, > >> > changing page from cached (wb) to uc or wc is very expensive and inv= olves > >> > a lot of flushing. To improve performance this allocator use a pool > >> > of uc,wc pages. > >> > > >> > Pools are linked lists of pages. Ordered so that first is the latest= adition > >> > to the pool and last is the oldest page. Old pages are periodicaly f= reed from > >> > pools to keep memory use at minimum. > >> > > >> > Pools are protected with spinlocks to allow multiple threads to allo= cate pages > >> > simultanously. Expensive operations must not hold the spinlock to ma= ximise > >> > performance for multiple threads. > >> > > >> > Based on Jerome Glisse's and Dave Airlie's pool allocator. > >> > > >> > Signed-off-by: Jerome Glisse > >> > Signed-off-by: Dave Airlie > >> > Signed-off-by: Pauli Nieminen > > > > I think it's overdesigned, by trying to be to clever we often endup > > with code more complex and less efficient in the end. While the idea > > to free page only after some amount of time is the way to go, i think > > a simpler approach will perform a better job. For instance: > > > > pool { > > =A0npages // npages in pool > > =A0page *pages[enoughttohold16Morsomethingconfigurable] > > =A0unsigned long next_free_time; > > =A0lock > > } > > filllocked(pool,npages) > > { > > =A0 =A0 =A0 =A0if (npages > MAXPAGESPERPOOL) > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0npages =3D MAXPAGESPERPOOL; > > =A0 =A0 =A0 =A0for i, npages : allocpage > > } > > alloc(apages, npages) > > { > > =A0 =A0 =A0 =A0do { > > =A0 =A0 =A0 =A0lock(pool) > > =A0 =A0 =A0 =A0pool->next_free_time =3D jiffies + timeout; > > =A0 =A0 =A0 =A0if (npages > pool->npages) { > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0npages -=3D pool->npages; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0copypageptr(apages, pool->pages, pool->n= pages) > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pool->npages =3D 0; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0fillpool(pool, npages) > > =A0 =A0 =A0 =A0} else { > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0pool->npages -=3D npages; > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0copypageptr(apages, pool->pages, npages) > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0npages =3D 0; > > =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0unlock(pool) > > =A0 =A0 =A0 =A0} while (npages) > > } > > poolshrinkcallbacktimer() > > { > > =A0 =A0 =A0 =A0for i in npools { > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (trylock(pool[i].lock) { > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (timeafter(jiffies, p= ool[i].nextfreetime)) { > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0free bun= ch of pages for instance 1M at a time > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0unlock(pool[i].lock) > > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > > =A0 =A0 =A0 =A0} > > } > = > I would need to use split buffer for that big arrays. My aim was to > have FILO queue pages so knowing which pages can be freed now is > simple because the oldest pages is always in the head of the list. I > know that FILO is easy to implement on top of deque but requirement to > split underlying array sounded like more complex than using linked > list. But as I noted in IRC. Requesting for N pages from the linked > list is doomed to have O(N) performance which is bad. So better > implementation would be switching to an array. The design i outlined above drop the concept of recording time per page so no need for a FILO, you just fill the pool and you empty it 1M or less at a time every Nmsec this is a lot simpler no complex spliting or list walking. One simple optimization is to make the amount of page we free dependent of how much the pool is fill so if there is 16M of memory in the pool free 1M at a time, but if there 64K then free one page at a time. > > > > General remark i don't think you handle jiffies wrap around properly. > = > Only place where jiffies are read is ttm_page_pool_free. There > difference is calculate with timeafter call so I think it should > handle the wrapping. > = No doesn't work, for instance (assuming jiffies to be 16 bits) page get it's private set to 0xFFF0, now ttm_page_pool_free jiffies wrap around and are at 0x0 the test will evaluate to false and page will be free prematuraly, i think better would have been to do: page->private =3D jiffies + timeout in ttm_put_pages and do time_after(jiffie, page->private) in ttm_page_pool_free that would have avoided the wrap around. Cheers, Jerome > > Also the whole lock dropping in middle of list traversal looks very > > fragile to me, right now we don't have much concurency in radeon so > > i don't think you well tested the effect of concurently using the > > allocator. Anyway see bellow for more comments on the code. > > > > Btw thanks for looking into this, this isn't a sexy area but it > > definitly needs love. > > > > Cheers, > > Jerome > > > > > >> > --- > >> > =A0drivers/gpu/drm/ttm/Makefile =A0 =A0 =A0 =A0 | =A0 =A02 +- > >> > =A0drivers/gpu/drm/ttm/ttm_memory.c =A0 =A0 | =A0 =A05 +- > >> > =A0drivers/gpu/drm/ttm/ttm_page_alloc.c | =A0514 +++++++++++++++++++= +++++++++++++++ > >> > =A0drivers/gpu/drm/ttm/ttm_page_alloc.h | =A0 38 +++ > >> > =A0drivers/gpu/drm/ttm/ttm_tt.c =A0 =A0 =A0 =A0 | =A0 47 ++-- > >> > =A05 files changed, 580 insertions(+), 26 deletions(-) > >> > =A0create mode 100644 drivers/gpu/drm/ttm/ttm_page_alloc.c > >> > =A0create mode 100644 drivers/gpu/drm/ttm/ttm_page_alloc.h > >> > > >> > diff --git a/drivers/gpu/drm/ttm/Makefile b/drivers/gpu/drm/ttm/Make= file > >> > index 1e138f5..4256e20 100644 > >> > --- a/drivers/gpu/drm/ttm/Makefile > >> > +++ b/drivers/gpu/drm/ttm/Makefile > >> > @@ -4,6 +4,6 @@ > >> > =A0ccflags-y :=3D -Iinclude/drm > >> > =A0ttm-y :=3D ttm_agp_backend.o ttm_memory.o ttm_tt.o ttm_bo.o \ > >> > =A0 =A0 =A0 =A0ttm_bo_util.o ttm_bo_vm.o ttm_module.o ttm_global.o \ > >> > - =A0 =A0 =A0 ttm_object.o ttm_lock.o ttm_execbuf_util.o > >> > + =A0 =A0 =A0 ttm_object.o ttm_lock.o ttm_execbuf_util.o ttm_page_al= loc.o > >> > > >> > =A0obj-$(CONFIG_DRM_TTM) +=3D ttm.o > >> > diff --git a/drivers/gpu/drm/ttm/ttm_memory.c b/drivers/gpu/drm/ttm/= ttm_memory.c > >> > index f5245c0..d587fbe 100644 > >> > --- a/drivers/gpu/drm/ttm/ttm_memory.c > >> > +++ b/drivers/gpu/drm/ttm/ttm_memory.c > >> > @@ -32,6 +32,7 @@ > >> > =A0#include > >> > =A0#include > >> > =A0#include > >> > +#include "ttm_page_alloc.h" > >> > > >> > =A0#define TTM_MEMORY_ALLOC_RETRIES 4 > >> > > >> > @@ -394,6 +395,7 @@ int ttm_mem_global_init(struct ttm_mem_global *g= lob) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 "Zone %7s: Available gra= phics memory: %llu kiB.\n", > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 zone->name, (unsigned lo= ng long) zone->max_mem >> 10); > >> > =A0 =A0 =A0 =A0} > >> > + =A0 =A0 =A0 ttm_page_alloc_init(); > >> > =A0 =A0 =A0 =A0return 0; > >> > =A0out_no_zone: > >> > =A0 =A0 =A0 =A0ttm_mem_global_release(glob); > >> > @@ -413,9 +415,10 @@ void ttm_mem_global_release(struct ttm_mem_glob= al *glob) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0zone =3D glob->zones[i]; > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0kobject_del(&zone->kobj); > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0kobject_put(&zone->kobj); > >> > - =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > =A0 =A0 =A0 =A0kobject_del(&glob->kobj); > >> > =A0 =A0 =A0 =A0kobject_put(&glob->kobj); > >> > + =A0 =A0 =A0 ttm_page_alloc_fini(); > >> > =A0} > >> > =A0EXPORT_SYMBOL(ttm_mem_global_release); > >> > > >> > diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.c b/drivers/gpu/drm/= ttm/ttm_page_alloc.c > >> > new file mode 100644 > >> > index 0000000..de6576c > >> > --- /dev/null > >> > +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.c > >> > @@ -0,0 +1,514 @@ > >> > +/* > >> > + * Copyright (c) Red Hat Inc. > >> > + > >> > + * Permission is hereby granted, free of charge, to any person obta= ining a > >> > + * copy of this software and associated documentation files (the "S= oftware"), > >> > + * to deal in the Software without restriction, including without l= imitation > >> > + * the rights to use, copy, modify, merge, publish, distribute, sub= license, > >> > + * and/or sell copies of the Software, and to permit persons to who= m the > >> > + * Software is furnished to do so, subject to the following conditi= ons: > >> > + * > >> > + * The above copyright notice and this permission notice (including= the > >> > + * next paragraph) shall be included in all copies or substantial p= ortions > >> > + * of the Software. > >> > + * > >> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, = EXPRESS OR > >> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANT= ABILITY, > >> > + * FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVE= NT SHALL > >> > + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGE= S OR OTHER > >> > + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, = ARISING > >> > + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OT= HER > >> > + * DEALINGS IN THE SOFTWARE. > >> > + * > >> > + * Authors: Dave Airlie > >> > + * =A0 =A0 =A0 =A0 =A0Jerome Glisse > >> > + * =A0 =A0 =A0 =A0 =A0Pauli Nienmien > >> > + */ > >> > + > >> > +/* simple list based uncached page pool > >> > + * - Pool collects resently freed pages for reuse > >> > + * - Use page->lru to keep a free list > >> > + * - doesn't track currently in use pages > >> > + */ > >> > +#include > >> > +#include > >> > +#include > >> > +#include > >> > +#include > >> > +#include > >> > + > >> > +#include > >> > +#include "ttm/ttm_bo_driver.h" > >> > +#include "ttm_page_alloc.h" > >> > + > >> > +/* Number of 4k pages to add at once */ > >> > +#define NUM_PAGES_TO_ADD =A0 =A0 =A0 =A0 =A0 =A0 =A0 64 > >> > +/* times are in msecs */ > >> > +#define TIME_TO_KEEP_PAGE_IN_POOL =A0 =A0 =A015000 > >> > +#define PAGE_FREE_INTERVAL =A0 =A0 =A0 =A0 =A0 =A0 1000 > >> > + > >> > +/** > >> > + * list is a FILO stack. > >> > + * All pages pushed into it must go to begin while end of list must= hold the > >> > + * least recently used pages. When pages in end of the list are not= recently > >> > + * use they are freed. > >> > + * > >> > + * All expensive operations must be done outside of pool lock to pr= event > >> > + * contention on lock. > >> > + */ > >> > +struct page_pool { > >> > + =A0 =A0 =A0 struct list_head =A0 =A0 =A0 =A0list; > >> > + =A0 =A0 =A0 int =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags; > >> > + =A0 =A0 =A0 unsigned =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0npages; > >> > + =A0 =A0 =A0 spinlock_t =A0 =A0 =A0 =A0 =A0 =A0 =A0lock; > >> > +}; > >> > + > >> > +# define MAX_NUM_POOLS 4 > >> > + > >> > +struct pool_manager { > >> > + =A0 =A0 =A0 bool =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0page_alloc= _inited; > >> > + =A0 =A0 =A0 unsigned long =A0 =A0 =A0 =A0 =A0 time_to_wait; > >> > + > >> > + =A0 =A0 =A0 struct timer_list =A0 =A0 =A0 timer; > >> > + > >> > + =A0 =A0 =A0 union { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0 =A0pools[= MAX_NUM_POOLS]; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 = =A0 =A0 =A0wc_pool; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 = =A0 =A0 =A0uc_pool; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 = =A0 =A0 =A0wc_pool_dma32; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 = =A0 =A0 =A0uc_pool_dma32; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } ; > >> > + =A0 =A0 =A0 }; > >> > +}; > >> > + > >> > + > >> > +static struct pool_manager _manager; > >> > + > >> > +#ifdef CONFIG_X86 > >> > +/* TODO: add this to x86 like _uc, this version here is inefficient= */ > >> > +static int set_pages_array_wc(struct page **pages, int addrinarray) > >> > +{ > >> > + =A0 =A0 =A0 int i; > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 set_memory_wc((unsigned long)page_addr= ess(pages[i]), 1); > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > +#else > >> > +static int set_pages_array_wb(struct page **pages, int addrinarray) > >> > +{ > >> > +#ifdef TTM_HAS_AGP > >> > + =A0 =A0 =A0 int i; > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 unmap_page_from_agp(pages[i]); > >> > +#endif > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > + > >> > +static int set_pages_array_wc(struct page **pages, int addrinarray) > >> > +{ > >> > +#ifdef TTM_HAS_AGP > >> > + =A0 =A0 =A0 int i; > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 map_page_into_agp(pages[i]); > >> > +#endif > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > + > >> > +static int set_pages_array_uc(struct page **pages, int addrinarray) > >> > +{ > >> > +#ifdef TTM_HAS_AGP > >> > + =A0 =A0 =A0 int i; > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 map_page_into_agp(pages[i]); > >> > +#endif > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > +#endif > >> > + > >> > + > >> > +static void ttm_page_pool_init_locked(struct page_pool *pool) > >> > +{ > >> > + =A0 =A0 =A0 spin_lock_init(&pool->lock); > >> > + =A0 =A0 =A0 INIT_LIST_HEAD(&pool->list); > >> > +} > >> > + > >> > +static struct page_pool *ttm_get_pool(int flags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate) > >> > +{ > >> > + =A0 =A0 =A0 int pool_index; > >> > + > >> > + =A0 =A0 =A0 if (cstate =3D=3D tt_cached) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return NULL; > >> > + > >> > + =A0 =A0 =A0 if (cstate =3D=3D tt_wc) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool_index =3D 0x0; > >> > + =A0 =A0 =A0 else > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool_index =3D 0x1; > >> > + > >> > + =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_DMA32) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool_index |=3D 0x2; > >> > + > >> > + =A0 =A0 =A0 return &_manager.pools[pool_index]; > >> > +} > >> > + > >> > +static void ttm_pages_put(struct page *pages[], unsigned npages) > >> > +{ > >> > + =A0 =A0 =A0 unsigned i; > >> > + =A0 =A0 =A0 set_pages_array_wb(pages, npages); > >> > + =A0 =A0 =A0 for (i =3D 0; i < npages; ++i) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(pages[i]); > >> > +} > >> > + > >> > +/** > >> > + * Free pages from pool. > >> > + * If limit_with_time is set to true pages that wil lbe have to hav= e been > >> > + * in pool for time specified by pool_manager.time_to_wait. > >> > + * > >> > + * Pages are freed in NUM_PAES_TO_ADD chunks to prevent freeing from > >> > + * blocking others operations on pool. > >> > + * > >> > + * This is called from the timer handler. > >> > + **/ > >> > +static void ttm_page_pool_free(struct page_pool *pool, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 const bool limit_with_time) > >> > +{ > >> > + =A0 =A0 =A0 struct page *page; > >> > + =A0 =A0 =A0 struct page *pages_to_free[NUM_PAGES_TO_ADD]; > >> > + =A0 =A0 =A0 unsigned freed_pages; > >> > + =A0 =A0 =A0 unsigned long now =3D jiffies + _manager.time_to_wait; > >> > + =A0 =A0 =A0 unsigned long irq_flags; > >> > + > >> > +restart: > >> > + =A0 =A0 =A0 freed_pages =3D 0; > >> > + =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> > + > >> > + =A0 =A0 =A0 { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_for_each_entry_reverse(page, &poo= l->list, lru) { > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (limit_with_time > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 && tim= e_after(now, page->private)) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pages_to_free[freed_pa= ges++] =3D page; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages >=3D N= UM_PAGES_TO_ADD) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Lim= it the maximum time that we hold the lock > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* n= ot to block more important threads */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __list= _del(page->lru.prev, &pool->list); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool->= npages -=3D freed_pages; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_u= nlock_irqrestore(&pool->lock, irq_flags); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_pa= ges_put(pages_to_free, freed_pages); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 goto r= estart; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool->npages -=3D freed_pages; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __list_del(&page->lru,= &pool->list); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, ir= q_flags); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_pages_put(pages_to= _free, freed_pages); > >> > + =A0 =A0 =A0 } > >> > +} > >> > + > > > > One useless brace level > > > >> > +/* periodicaly free unused pages from pool */ > >> > +static void ttm_page_pool_timer_handler(unsigned long man_addr) > >> > +{ > >> > + =A0 =A0 =A0 int i; > >> > + =A0 =A0 =A0 struct pool_manager *m =3D (struct pool_manager *)man_= addr; > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < MAX_NUM_POOLS; ++i) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_page_pool_free(&_manager.pools[i],= true); > >> > + > >> > + =A0 =A0 =A0 mod_timer(&m->timer, jiffies + msecs_to_jiffies(PAGE_F= REE_INTERVAL)); > >> > +} > >> > + > >> > +static void ttm_page_pool_timer_create(struct pool_manager *m) > >> > +{ > >> > + =A0 =A0 =A0 /* Create a timer that is fired every second to clean > >> > + =A0 =A0 =A0 =A0* the free list */ > >> > + =A0 =A0 =A0 init_timer(&m->timer); > >> > + =A0 =A0 =A0 m->timer.expires =3D jiffies + msecs_to_jiffies(PAGE_F= REE_INTERVAL); > >> > + =A0 =A0 =A0 m->timer.function =3D ttm_page_pool_timer_handler; > >> > + =A0 =A0 =A0 m->timer.data =3D (unsigned long)m; > >> > +} > >> > + > >> > +static void ttm_page_pool_timer_free(struct pool_manager *m) > >> > +{ > >> > + =A0 =A0 =A0 del_timer_sync(&m->timer); > >> > +} > >> > + > >> > +static void ttm_set_pages_caching(struct page **pages, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigne= d cpages) > >> > +{ > >> > + =A0 =A0 =A0 /* Set page caching */ > >> > + =A0 =A0 =A0 switch (cstate) { > >> > + =A0 =A0 =A0 case tt_uncached: > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 set_pages_array_uc(pages, cpages); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + =A0 =A0 =A0 case tt_wc: > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 set_pages_array_wc(pages, cpages); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + =A0 =A0 =A0 default: > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + =A0 =A0 =A0 } > >> > +} > >> > +/** > >> > + * Fill new pages to array. > >> > + */ > >> > +static int ttm_pages_pool_fill_locked(struct page_pool *pool, int f= lags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigne= d count, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned long *irq_flags) > >> > +{ > >> > + =A0 =A0 =A0 unsigned pages_to_alloc =3D count - pool->npages; > >> > + =A0 =A0 =A0 struct page **pages; > >> > + =A0 =A0 =A0 struct page *page; > >> > + =A0 =A0 =A0 struct list_head h; > >> > + =A0 =A0 =A0 unsigned i, cpages; > >> > + =A0 =A0 =A0 unsigned max_cpages =3D min(pages_to_alloc, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (unsigned)(PAGE_SIZE/s= izeof(*page))); > >> > + > >> > + =A0 =A0 =A0 /* reserve allready allocated pages from pool */ > >> > + =A0 =A0 =A0 INIT_LIST_HEAD(&h); > >> > + =A0 =A0 =A0 list_splice_init(&pool->list, &h); > >> > + > >> > + =A0 =A0 =A0 /* We are now safe to allow others users to modify the= pool so unlock > >> > + =A0 =A0 =A0 =A0* it. Allocating pages is expensive operation. */ > >> > + =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, *irq_flags); > >> > + > >> > + =A0 =A0 =A0 pages =3D kmalloc(pages_to_alloc*sizeof(*page), GFP_KE= RNEL); > >> > + > > > > page_to_alloc might be negative if pool->npages > count, i think it can > > happen because there is lock dropping in the page alloc function below. > > also pages_to_alloc*sizeof(*page) might be to big for kmalloc and we > > might need to use vmalloc (this is why i think a design with a static > > allocated on table is better) > = > page_to_alloc can't be negative because lock is held untl later into > pool allocator. > = > Locks are only dropped: > - When allocation succeed > - When returning failure. > - After entering ttm_pages_pool_fill_locked. Before droping lock > there code reserves pages from pool to private list and sets pool to > empty state. > = > ttm_pages_pool_fill_locked had 2 errors in the first version. One of > them was wrong count paramter to kmalloc. 2nd version fixes that we > never alloc more than PAGE_SIZES memory. Second error was that > pool->npages wasn't set to zero when taking the pages from pool. > = > Basicaly fill_pool operates only in private variables when it is > outside of pool locks which makes it reentrant for unlocked part. > = > > > >> > + =A0 =A0 =A0 if (!pages) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm] unable to alloc= ate table for new pages."); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_fl= ags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> > + =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0, cpages =3D 0; i < pages_to_alloc; ++i) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 page =3D alloc_page(pool->gfp_flags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!page) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm]= unable to get page %d\n", i); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_put_pages(&h, flag= s, cstate); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&poo= l->lock, *irq_flags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > +#ifdef CONFIG_HIGHMEM > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* gfp flags of highmem page should ne= ver be dma32 so we > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* we should be fine in such case > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!PageHighMem(page)) > >> > +#endif > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pages[cpages++] =3D pa= ge; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (cpages =3D=3D max_= cpages) { > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_se= t_pages_caching(pages, cstate, cpages); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpages= =3D 0; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&page->lru, &h); > >> > + =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 if (cpages) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_set_pages_caching(pages, cstate, c= pages); > >> > + > >> > + =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_flags); > >> > + > >> > + =A0 =A0 =A0 list_splice(&h, &pool->list); > >> > + > >> > + =A0 =A0 =A0 pool->npages +=3D pages_to_alloc; > >> > + > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > + > >> > +/* > >> > + * On success pages list will hold count number of correctly > >> > + * cached pages. > >> > + */ > >> > +int ttm_get_pages(struct list_head *pages, int flags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigne= d count) > >> > +{ > >> > + =A0 =A0 =A0 struct page_pool *pool =3D ttm_get_pool(flags, cstate); > >> > + =A0 =A0 =A0 struct page *page =3D NULL; > >> > + =A0 =A0 =A0 unsigned long irq_flags; > >> > + =A0 =A0 =A0 int gfp_flags =3D 0; > >> > + =A0 =A0 =A0 int retries =3D 1; > >> > + =A0 =A0 =A0 int r; > >> > + > >> > + =A0 =A0 =A0 /* No pool for cached pages */ > >> > + =A0 =A0 =A0 if (cstate =3D=3D tt_cached) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_Z= ERO; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_DMA32) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D GFP_DMA= 32; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 else > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_H= IGHMEM; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* fallback in case of problems with p= ool allocation */ > >> > +alloc_pages: > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 for (r =3D 0; r < count; ++r) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 page =3D alloc_page(gf= p_flags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!page) { > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk= (KERN_ERR "[ttm] unable to allocate page."); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return= -ENOMEM; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&page->lru, p= ages); > > > > this doesn't alloc page with proper caching > > > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> > + =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 gfp_flags =3D pool->gfp_flags; > >> > + > >> > + =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> > + =A0 =A0 =A0 do { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* First we look if pool has the page = we want */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (pool->npages >=3D count) { > >> > + > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Find the cut locati= on */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (count <=3D pool->n= pages/2) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsign= ed c =3D 0; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_f= or_each_entry(page, &pool->list, lru) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 if (++c >=3D count) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* We = have to point to the last pages cut */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsign= ed c =3D 0; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsign= ed rcount =3D pool->npages - count + 1; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_f= or_each_entry_reverse(page, &pool->list, lru) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 if (++c >=3D rcount) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_cut_position(page= s, &pool->list, &page->lru); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool->npages -=3D coun= t; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore= (&pool->lock, irq_flags); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_F= LAG_ZERO_ALLOC) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_f= or_each_entry(page, pages, lru) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 clear_page(page_address(page)); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (retries =3D=3D 0) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 --retries; > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 r =3D ttm_pages_pool_fill_locked(pool,= flags, cstate, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 count,= &irq_flags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (r) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore= (&pool->lock, irq_flags); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return r; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 } while (1); > >> > + > >> > + =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, irq_flags); > >> > + > >> > + =A0 =A0 =A0 /* =A0Pool allocator failed without errors. we try to = allocate new pages > >> > + =A0 =A0 =A0 =A0* =A0instead. */ > >> > + =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_ZERO; > >> > + > >> > + =A0 =A0 =A0 goto alloc_pages; > >> > +} > > > > This function is quite complex, especialy the failing filling pool path > > that looks supicious, idea of splitting work is not to have to do the > > work in other function so if pool filling fails the alloc should fail > > and shouldn't try to fill the pool on its own this kind of code > > duplication is a recipie for failure i think. > = > yes. Failing path should just return -ENOMEM. I will fix that. > = > > > >> > + > >> > +/* Put all pages in pages list to correct pool to wait for reuse */ > >> > +void ttm_put_pages(struct list_head *pages, int flags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate) > >> > +{ > >> > + =A0 =A0 =A0 struct page_pool *pool =3D ttm_get_pool(flags, cstate); > >> > + =A0 =A0 =A0 struct page *p, *tmp; > >> > + =A0 =A0 =A0 unsigned page_count =3D 0; > >> > + =A0 =A0 =A0 unsigned long irq_flags; > >> > + =A0 =A0 =A0 unsigned long now =3D jiffies; > >> > + > >> > + =A0 =A0 =A0 if (pool =3D=3D NULL) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_for_each_entry_safe(p, tmp, pages= , lru) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(p); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* We took the ownership -> clear the = pages list*/ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 INIT_LIST_HEAD(pages); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return; > >> > + =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 list_for_each_entry_safe(p, tmp, pages, lru) { > >> > + > >> > +#ifdef CONFIG_HIGHMEM > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* we don't have pool for highmem -> f= ree them */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (PageHighMem(p)) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_del(&p->lru); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(p); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else > >> > +#endif > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 p->private =3D now; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ++page_count; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 } > >> > + > >> > + =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> > + =A0 =A0 =A0 list_splice(pages, &pool->list); > >> > + =A0 =A0 =A0 pool->npages +=3D page_count; > >> > + =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, irq_flags); > >> > + > >> > + =A0 =A0 =A0 /* We took the ownership -> clear the pages list*/ > >> > + =A0 =A0 =A0 INIT_LIST_HEAD(pages); > >> > +} > >> > + > >> > + > >> > +int ttm_page_alloc_init(void) > >> > +{ > >> > + =A0 =A0 =A0 if (_manager.page_alloc_inited) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> > + > >> > + =A0 =A0 =A0 ttm_page_pool_init_locked(&_manager.wc_pool); > >> > + =A0 =A0 =A0 _manager.wc_pool.gfp_flags =3D GFP_HIGHUSER; > >> > + =A0 =A0 =A0 ttm_page_pool_init_locked(&_manager.uc_pool); > >> > + =A0 =A0 =A0 _manager.uc_pool.gfp_flags =3D GFP_HIGHUSER; > >> > + =A0 =A0 =A0 ttm_page_pool_init_locked(&_manager.wc_pool_dma32); > >> > + =A0 =A0 =A0 _manager.wc_pool_dma32.gfp_flags =3D GFP_USER | GFP_DM= A32; > >> > + =A0 =A0 =A0 ttm_page_pool_init_locked(&_manager.uc_pool_dma32); > >> > + =A0 =A0 =A0 _manager.uc_pool_dma32.gfp_flags =3D GFP_USER | GFP_DM= A32; > >> > + > >> > + =A0 =A0 =A0 _manager.time_to_wait =3D msecs_to_jiffies(TIME_TO_KEE= P_PAGE_IN_POOL); > >> > + =A0 =A0 =A0 ttm_page_pool_timer_create(&_manager); > >> > + =A0 =A0 =A0 _manager.page_alloc_inited =3D 1; > >> > + =A0 =A0 =A0 return 0; > >> > +} > >> > + > >> > +void ttm_page_alloc_fini(void) > >> > +{ > >> > + =A0 =A0 =A0 int i; > >> > + > >> > + =A0 =A0 =A0 if (!_manager.page_alloc_inited) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return; > >> > + > >> > + =A0 =A0 =A0 ttm_page_pool_timer_free(&_manager); > >> > + > >> > + =A0 =A0 =A0 for (i =3D 0; i < MAX_NUM_POOLS; ++i) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_page_pool_free(&_manager.pools[i],= false); > >> > + > >> > + =A0 =A0 =A0 _manager.page_alloc_inited =3D 0; > >> > +} > >> > diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.h b/drivers/gpu/drm/= ttm/ttm_page_alloc.h > >> > new file mode 100644 > >> > index 0000000..eb0ddf4 > >> > --- /dev/null > >> > +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.h > >> > @@ -0,0 +1,38 @@ > >> > +/* > >> > + * Copyright (c) Red Hat Inc. > >> > + > >> > + * Permission is hereby granted, free of charge, to any person obta= ining a > >> > + * copy of this software and associated documentation files (the "S= oftware"), > >> > + * to deal in the Software without restriction, including without l= imitation > >> > + * the rights to use, copy, modify, merge, publish, distribute, sub= license, > >> > + * and/or sell copies of the Software, and to permit persons to who= m the > >> > + * Software is furnished to do so, subject to the following conditi= ons: > >> > + * > >> > + * The above copyright notice and this permission notice (including= the > >> > + * next paragraph) shall be included in all copies or substantial p= ortions > >> > + * of the Software. > >> > + * > >> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, = EXPRESS OR > >> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANT= ABILITY, > >> > + * FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVE= NT SHALL > >> > + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGE= S OR OTHER > >> > + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, = ARISING > >> > + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OT= HER > >> > + * DEALINGS IN THE SOFTWARE. > >> > + * > >> > + * Authors: Dave Airlie > >> > + * =A0 =A0 =A0 =A0 =A0Jerome Glisse > >> > + */ > >> > +#ifndef TTM_PAGE_ALLOC > >> > +#define TTM_PAGE_ALLOC > >> > + > >> > +#include "ttm/ttm_bo_driver.h" > >> > + > >> > +void ttm_put_pages(struct list_head *pages, int flags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate); > >> > +int ttm_get_pages(struct list_head *pages, int flags, > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigne= d count); > >> > +int ttm_page_alloc_init(void); > >> > +void ttm_page_alloc_fini(void); > >> > + > >> > +#endif > >> > diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_= tt.c > >> > index 3d47a2c..d4e20f6 100644 > >> > --- a/drivers/gpu/drm/ttm/ttm_tt.c > >> > +++ b/drivers/gpu/drm/ttm/ttm_tt.c > >> > @@ -38,6 +38,7 @@ > >> > =A0#include "ttm/ttm_module.h" > >> > =A0#include "ttm/ttm_bo_driver.h" > >> > =A0#include "ttm/ttm_placement.h" > >> > +#include "ttm_page_alloc.h" > >> > > >> > =A0static int ttm_tt_swapin(struct ttm_tt *ttm); > >> > > >> > @@ -72,21 +73,6 @@ static void ttm_tt_free_page_directory(struct ttm= _tt *ttm) > >> > =A0 =A0 =A0 =A0ttm->pages =3D NULL; > >> > =A0} > >> > > >> > -static struct page *ttm_tt_alloc_page(unsigned page_flags) > >> > -{ > >> > - =A0 =A0 =A0 gfp_t gfp_flags =3D GFP_USER; > >> > - > >> > - =A0 =A0 =A0 if (page_flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_ZERO; > >> > - > >> > - =A0 =A0 =A0 if (page_flags & TTM_PAGE_FLAG_DMA32) > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_DMA32; > >> > - =A0 =A0 =A0 else > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_HIGHMEM; > >> > - > >> > - =A0 =A0 =A0 return alloc_page(gfp_flags); > >> > -} > >> > - > >> > =A0static void ttm_tt_free_user_pages(struct ttm_tt *ttm) > >> > =A0{ > >> > =A0 =A0 =A0 =A0int write; > >> > @@ -127,15 +113,21 @@ static void ttm_tt_free_user_pages(struct ttm_= tt *ttm) > >> > =A0static struct page *__ttm_tt_get_page(struct ttm_tt *ttm, int ind= ex) > >> > =A0{ > >> > =A0 =A0 =A0 =A0struct page *p; > >> > + =A0 =A0 =A0 struct list_head h; > >> > =A0 =A0 =A0 =A0struct ttm_mem_global *mem_glob =3D ttm->glob->mem_gl= ob; > >> > =A0 =A0 =A0 =A0int ret; > >> > > >> > =A0 =A0 =A0 =A0while (NULL =3D=3D (p =3D ttm->pages[index])) { > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 p =3D ttm_tt_alloc_page(ttm->page_flag= s); > >> > > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!p) > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 INIT_LIST_HEAD(&h); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D ttm_get_pages(&h, ttm->page_fl= ags, ttm->caching_state, 1); > >> > + > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (ret !=3D 0) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return NULL; > >> > > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 p =3D list_first_entry(&h, struct page= , lru); > >> > + > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ret =3D ttm_mem_global_alloc_page(mem= _glob, p, false, false); > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (unlikely(ret !=3D 0)) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0goto out_err; > >> > @@ -244,10 +236,10 @@ static int ttm_tt_set_caching(struct ttm_tt *t= tm, > >> > =A0 =A0 =A0 =A0if (ttm->caching_state =3D=3D c_state) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return 0; > >> > > >> > - =A0 =A0 =A0 if (c_state !=3D tt_cached) { > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D ttm_tt_populate(ttm); > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (unlikely(ret !=3D 0)) > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return ret; > >> > + =A0 =A0 =A0 if (ttm->state =3D=3D tt_unpopulated) { > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Change caching but don't populate */ > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm->caching_state =3D c_state; > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> > =A0 =A0 =A0 =A0} > >> > > >> > =A0 =A0 =A0 =A0if (ttm->caching_state =3D=3D tt_cached) > >> > @@ -298,25 +290,32 @@ EXPORT_SYMBOL(ttm_tt_set_placement_caching); > >> > =A0static void ttm_tt_free_alloced_pages(struct ttm_tt *ttm) > >> > =A0{ > >> > =A0 =A0 =A0 =A0int i; > >> > + =A0 =A0 =A0 unsigned count =3D 0; > >> > + =A0 =A0 =A0 struct list_head h; > >> > =A0 =A0 =A0 =A0struct page *cur_page; > >> > =A0 =A0 =A0 =A0struct ttm_backend *be =3D ttm->be; > >> > > >> > + =A0 =A0 =A0 INIT_LIST_HEAD(&h); > >> > + > >> > =A0 =A0 =A0 =A0if (be) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0be->func->clear(be); > >> > - =A0 =A0 =A0 (void)ttm_tt_set_caching(ttm, tt_cached); > >> > =A0 =A0 =A0 =A0for (i =3D 0; i < ttm->num_pages; ++i) { > >> > + > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0cur_page =3D ttm->pages[i]; > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ttm->pages[i] =3D NULL; > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (cur_page) { > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (page_count(cur_pa= ge) !=3D 1) > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0print= k(KERN_ERR TTM_PFX > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 "Erroneous page count. " > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0"Leaking pages.\n"); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0"Leaking pages (%d/%d).\n", > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0page_count(cur_page), count); > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ttm_mem_global_free_p= age(ttm->glob->mem_glob, > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 cur_page); > >> > - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(cur_page); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&cur_page->lr= u, &h); > >> > + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 count++; > >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} > >> > =A0 =A0 =A0 =A0} > >> > + =A0 =A0 =A0 ttm_put_pages(&h, ttm->page_flags, ttm->caching_state); > >> > =A0 =A0 =A0 =A0ttm->state =3D tt_unpopulated; > >> > =A0 =A0 =A0 =A0ttm->first_himem_page =3D ttm->num_pages; > >> > =A0 =A0 =A0 =A0ttm->last_lomem_page =3D -1; > >> > -- > >> > 1.6.3.3 > >> > > >> > > >> > >> While no comments yet I did notice huge errors in > >> ttm_pages_pool_fill_locked. But I still would like to comments on > >> overall design. > >> > >> Is it correct to use page->private in pool allocator? > >> > >> bo allocation is still too expensive operation for dma buffers in > >> classic mesa. It takes quite a lot cpu time in bind memory (agp > >> system) and ttm_mem_global functions. Would it be possible to move > >> some parts those expensive operations into pool fill and pool free > >> code? > >> > >> Also using single call to allocate all pages for buffer would improve > >> situation when pool is not large enough. I can change ttm_tt_populate > >> to allocate all pages with single call if it sounds ok. > >> > >> I will still look at how to integrate pool allocator to shinker. > >> > >> > >> Attached fixed full path and here is fixed version of > >> ttm_pages_pool_fill_locked : > >> > >> +static int ttm_pages_pool_fill_locked(struct page_pool *pool, int fla= gs, > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigned = count, > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned long *irq_flags) > >> +{ > >> + =A0 =A0 =A0 unsigned pages_to_alloc =3D count - pool->npages; > >> + =A0 =A0 =A0 struct page **pages; > >> + =A0 =A0 =A0 struct page *page; > >> + =A0 =A0 =A0 struct list_head h; > >> + =A0 =A0 =A0 unsigned i, cpages; > >> + =A0 =A0 =A0 unsigned max_cpages =3D min(pages_to_alloc, > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (unsigned)(PAGE_SIZE/siz= eof(*page))); > >> + > >> + =A0 =A0 =A0 /* reserve allready allocated pages from pool */ > >> + =A0 =A0 =A0 INIT_LIST_HEAD(&h); > >> + =A0 =A0 =A0 list_splice_init(&pool->list, &h); > >> + =A0 =A0 =A0 i =3D pool->npages; > >> + =A0 =A0 =A0 pages_to_alloc =3D count; > >> + =A0 =A0 =A0 pool->npages =3D 0; > >> + > >> + =A0 =A0 =A0 /* We are now safe to allow others users to modify the p= ool so unlock > >> + =A0 =A0 =A0 =A0* it. Allocating pages is expensive operation. */ > >> + =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, *irq_flags); > >> + > >> + =A0 =A0 =A0 pages =3D kmalloc(max_cpages*sizeof(*page), GFP_KERNEL); > >> + > >> + =A0 =A0 =A0 if (!pages) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm] unable to allocat= e table for > >> new pages."); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_flag= s); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> + =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 /* i is set to number of pages already in pool */ > >> + =A0 =A0 =A0 for (cpages =3D 0; i < pages_to_alloc; ++i) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 page =3D alloc_page(pool->gfp_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!page) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm] u= nable to get page %u\n", i); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_put_pages(&h, flags,= cstate); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&pool-= >lock, *irq_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> +#ifdef CONFIG_HIGHMEM > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* gfp flags of highmem page should neve= r be dma32 so we > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* we should be fine in such case > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!PageHighMem(page)) > >> +#endif > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pages[cpages++] =3D page; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (cpages =3D=3D max_cp= ages) { > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_set_= pages_caching(pages, cstate, cpages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpages = =3D 0; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&page->lru, &h); > >> + =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 if (cpages) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_set_pages_caching(pages, cstate, cpa= ges); > >> + > >> + =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_flags); > >> + > >> + =A0 =A0 =A0 list_splice(&h, &pool->list); > >> + > >> + =A0 =A0 =A0 pool->npages +=3D pages_to_alloc; > >> + > >> + =A0 =A0 =A0 return 0; > >> +} > > > >> From 8340b84a4752830feff6b77f3c23707dbd69f757 Mon Sep 17 00:00:00 2001 > >> From: Jerome Glisse > >> Date: Wed, 23 Sep 2009 18:48:56 +1000 > >> Subject: [PATCH] drm/ttm: add pool wc/uc page allocator > >> > >> On AGP system we might allocate/free routinely uncached or wc memory, > >> changing page from cached (wb) to uc or wc is very expensive and invol= ves > >> a lot of flushing. To improve performance this allocator use a pool > >> of uc,wc pages. > >> > >> Pools are linked lists of pages. Ordered so that first is the latest a= dition > >> to the pool and last is the oldest page. Old pages are periodicaly fre= ed from > >> pools to keep memory use at minimum. > >> > >> Pools are protected with spinlocks to allow multiple threads to alloca= te pages > >> simultanously. Expensive operations must not hold the spinlock to maxi= mise > >> performance for multiple threads. > >> > >> V2: Fix ttm_page_pool_fill_locked. > >> =A0- max_cpages should be used for kmalloc instead of pages_to_alloc. > >> =A0- Set pool->npages to zero after taking all pages from the pool. > >> > >> Based on Jerome Glisse's and Dave Airlie's pool allocator. > >> > >> Signed-off-by: Jerome Glisse > >> Signed-off-by: Dave Airlie > >> Signed-off-by: Pauli Nieminen > >> --- > >> =A0drivers/gpu/drm/ttm/Makefile =A0 =A0 =A0 =A0 | =A0 =A02 +- > >> =A0drivers/gpu/drm/ttm/ttm_memory.c =A0 =A0 | =A0 =A05 +- > >> =A0drivers/gpu/drm/ttm/ttm_page_alloc.c | =A0518 +++++++++++++++++++++= +++++++++++++ > >> =A0drivers/gpu/drm/ttm/ttm_page_alloc.h | =A0 38 +++ > >> =A0drivers/gpu/drm/ttm/ttm_tt.c =A0 =A0 =A0 =A0 | =A0 47 ++-- > >> =A05 files changed, 584 insertions(+), 26 deletions(-) > >> =A0create mode 100644 drivers/gpu/drm/ttm/ttm_page_alloc.c > >> =A0create mode 100644 drivers/gpu/drm/ttm/ttm_page_alloc.h > >> > >> diff --git a/drivers/gpu/drm/ttm/Makefile b/drivers/gpu/drm/ttm/Makefi= le > >> index 1e138f5..4256e20 100644 > >> --- a/drivers/gpu/drm/ttm/Makefile > >> +++ b/drivers/gpu/drm/ttm/Makefile > >> @@ -4,6 +4,6 @@ > >> =A0ccflags-y :=3D -Iinclude/drm > >> =A0ttm-y :=3D ttm_agp_backend.o ttm_memory.o ttm_tt.o ttm_bo.o \ > >> =A0 =A0 =A0 ttm_bo_util.o ttm_bo_vm.o ttm_module.o ttm_global.o \ > >> - =A0 =A0 ttm_object.o ttm_lock.o ttm_execbuf_util.o > >> + =A0 =A0 ttm_object.o ttm_lock.o ttm_execbuf_util.o ttm_page_alloc.o > >> > >> =A0obj-$(CONFIG_DRM_TTM) +=3D ttm.o > >> diff --git a/drivers/gpu/drm/ttm/ttm_memory.c b/drivers/gpu/drm/ttm/tt= m_memory.c > >> index f5245c0..d587fbe 100644 > >> --- a/drivers/gpu/drm/ttm/ttm_memory.c > >> +++ b/drivers/gpu/drm/ttm/ttm_memory.c > >> @@ -32,6 +32,7 @@ > >> =A0#include > >> =A0#include > >> =A0#include > >> +#include "ttm_page_alloc.h" > >> > >> =A0#define TTM_MEMORY_ALLOC_RETRIES 4 > >> > >> @@ -394,6 +395,7 @@ int ttm_mem_global_init(struct ttm_mem_global *glo= b) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0"Zone %7s: Available graphi= cs memory: %llu kiB.\n", > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0zone->name, (unsigned long = long) zone->max_mem >> 10); > >> =A0 =A0 =A0 } > >> + =A0 =A0 ttm_page_alloc_init(); > >> =A0 =A0 =A0 return 0; > >> =A0out_no_zone: > >> =A0 =A0 =A0 ttm_mem_global_release(glob); > >> @@ -413,9 +415,10 @@ void ttm_mem_global_release(struct ttm_mem_global= *glob) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 zone =3D glob->zones[i]; > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 kobject_del(&zone->kobj); > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 kobject_put(&zone->kobj); > >> - =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> =A0 =A0 =A0 kobject_del(&glob->kobj); > >> =A0 =A0 =A0 kobject_put(&glob->kobj); > >> + =A0 =A0 ttm_page_alloc_fini(); > >> =A0} > >> =A0EXPORT_SYMBOL(ttm_mem_global_release); > >> > >> diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.c b/drivers/gpu/drm/tt= m/ttm_page_alloc.c > >> new file mode 100644 > >> index 0000000..e269a10 > >> --- /dev/null > >> +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.c > >> @@ -0,0 +1,518 @@ > >> +/* > >> + * Copyright (c) Red Hat Inc. > >> + > >> + * Permission is hereby granted, free of charge, to any person obtain= ing a > >> + * copy of this software and associated documentation files (the "Sof= tware"), > >> + * to deal in the Software without restriction, including without lim= itation > >> + * the rights to use, copy, modify, merge, publish, distribute, sub l= icense, > >> + * and/or sell copies of the Software, and to permit persons to whom = the > >> + * Software is furnished to do so, subject to the following condition= s: > >> + * > >> + * The above copyright notice and this permission notice (including t= he > >> + * next paragraph) shall be included in all copies or substantial por= tions > >> + * of the Software. > >> + * > >> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EX= PRESS OR > >> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTAB= ILITY, > >> + * FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT= SHALL > >> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES = OR OTHER > >> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, AR= ISING > >> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER > >> + * DEALINGS IN THE SOFTWARE. > >> + * > >> + * Authors: Dave Airlie > >> + * =A0 =A0 =A0 =A0 =A0Jerome Glisse > >> + * =A0 =A0 =A0 =A0 =A0Pauli Nieminen > >> + */ > >> + > >> +/* simple list based uncached page pool > >> + * - Pool collects resently freed pages for reuse > >> + * - Use page->lru to keep a free list > >> + * - doesn't track currently in use pages > >> + */ > >> +#include > >> +#include > >> +#include > >> +#include > >> +#include > >> +#include > >> + > >> +#include > >> +#include "ttm/ttm_bo_driver.h" > >> +#include "ttm_page_alloc.h" > >> + > >> +/* Number of 4k pages to add at once */ > >> +#define NUM_PAGES_TO_ADD =A0 =A0 =A0 =A0 =A0 =A0 64 > >> +/* times are in msecs */ > >> +#define TIME_TO_KEEP_PAGE_IN_POOL =A0 =A015000 > >> +#define PAGE_FREE_INTERVAL =A0 =A0 =A0 =A0 =A0 1000 > >> + > >> +/** > >> + * list is a FILO stack. > >> + * All pages pushed into it must go to begin while end of list must h= old the > >> + * least recently used pages. When pages in end of the list are not r= ecently > >> + * use they are freed. > >> + * > >> + * All expensive operations must be done outside of pool lock to prev= ent > >> + * contention on lock. > >> + */ > >> +struct page_pool { > >> + =A0 =A0 struct list_head =A0 =A0 =A0 =A0list; > >> + =A0 =A0 int =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags; > >> + =A0 =A0 unsigned =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0npages; > >> + =A0 =A0 spinlock_t =A0 =A0 =A0 =A0 =A0 =A0 =A0lock; > >> +}; > >> + > >> +# define MAX_NUM_POOLS 4 > >> + > >> +struct pool_manager { > >> + =A0 =A0 bool =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0page_alloc_inite= d; > >> + =A0 =A0 unsigned long =A0 =A0 =A0 =A0 =A0 time_to_wait; > >> + > >> + =A0 =A0 struct timer_list =A0 =A0 =A0 timer; > >> + > >> + =A0 =A0 union { > >> + =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0 =A0pools[MAX_NU= M_POOLS]; > >> + =A0 =A0 =A0 =A0 =A0 =A0 struct { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0= =A0wc_pool; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0= =A0uc_pool; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0= =A0wc_pool_dma32; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 struct page_pool =A0 =A0 =A0= =A0uc_pool_dma32; > >> + =A0 =A0 =A0 =A0 =A0 =A0 } ; > >> + =A0 =A0 }; > >> +}; > >> + > >> + > >> +static struct pool_manager _manager; > >> + > >> +#ifdef CONFIG_X86 > >> +/* TODO: add this to x86 like _uc, this version here is inefficient */ > >> +static int set_pages_array_wc(struct page **pages, int addrinarray) > >> +{ > >> + =A0 =A0 int i; > >> + > >> + =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> + =A0 =A0 =A0 =A0 =A0 =A0 set_memory_wc((unsigned long)page_address(pa= ges[i]), 1); > >> + =A0 =A0 return 0; > >> +} > >> +#else > >> +static int set_pages_array_wb(struct page **pages, int addrinarray) > >> +{ > >> +#ifdef TTM_HAS_AGP > >> + =A0 =A0 int i; > >> + > >> + =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> + =A0 =A0 =A0 =A0 =A0 =A0 unmap_page_from_agp(pages[i]); > >> +#endif > >> + =A0 =A0 return 0; > >> +} > >> + > >> +static int set_pages_array_wc(struct page **pages, int addrinarray) > >> +{ > >> +#ifdef TTM_HAS_AGP > >> + =A0 =A0 int i; > >> + > >> + =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> + =A0 =A0 =A0 =A0 =A0 =A0 map_page_into_agp(pages[i]); > >> +#endif > >> + =A0 =A0 return 0; > >> +} > >> + > >> +static int set_pages_array_uc(struct page **pages, int addrinarray) > >> +{ > >> +#ifdef TTM_HAS_AGP > >> + =A0 =A0 int i; > >> + > >> + =A0 =A0 for (i =3D 0; i < addrinarray; i++) > >> + =A0 =A0 =A0 =A0 =A0 =A0 map_page_into_agp(pages[i]); > >> +#endif > >> + =A0 =A0 return 0; > >> +} > >> +#endif > >> + > >> + > >> +static void ttm_page_pool_init_locked(struct page_pool *pool) > >> +{ > >> + =A0 =A0 spin_lock_init(&pool->lock); > >> + =A0 =A0 INIT_LIST_HEAD(&pool->list); > >> +} > >> + > >> +static struct page_pool *ttm_get_pool(int flags, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate) > >> +{ > >> + =A0 =A0 int pool_index; > >> + > >> + =A0 =A0 if (cstate =3D=3D tt_cached) > >> + =A0 =A0 =A0 =A0 =A0 =A0 return NULL; > >> + > >> + =A0 =A0 if (cstate =3D=3D tt_wc) > >> + =A0 =A0 =A0 =A0 =A0 =A0 pool_index =3D 0x0; > >> + =A0 =A0 else > >> + =A0 =A0 =A0 =A0 =A0 =A0 pool_index =3D 0x1; > >> + > >> + =A0 =A0 if (flags & TTM_PAGE_FLAG_DMA32) > >> + =A0 =A0 =A0 =A0 =A0 =A0 pool_index |=3D 0x2; > >> + > >> + =A0 =A0 return &_manager.pools[pool_index]; > >> +} > >> + > >> +static void ttm_pages_put(struct page *pages[], unsigned npages) > >> +{ > >> + =A0 =A0 unsigned i; > >> + =A0 =A0 set_pages_array_wb(pages, npages); > >> + =A0 =A0 for (i =3D 0; i < npages; ++i) > >> + =A0 =A0 =A0 =A0 =A0 =A0 __free_page(pages[i]); > >> +} > >> + > >> +/** > >> + * Free pages from pool. > >> + * If limit_with_time is set to true pages that wil lbe have to have = been > >> + * in pool for time specified by pool_manager.time_to_wait. > >> + * > >> + * Pages are freed in NUM_PAES_TO_ADD chunks to prevent freeing from > >> + * blocking others operations on pool. > >> + * > >> + * This is called from the timer handler. > >> + **/ > >> +static void ttm_page_pool_free(struct page_pool *pool, > >> + =A0 =A0 =A0 =A0 =A0 =A0 const bool limit_with_time) > >> +{ > >> + =A0 =A0 struct page *page; > >> + =A0 =A0 struct page *pages_to_free[NUM_PAGES_TO_ADD]; > >> + =A0 =A0 unsigned freed_pages; > >> + =A0 =A0 unsigned long now =3D jiffies + _manager.time_to_wait; > >> + =A0 =A0 unsigned long irq_flags; > >> + > >> +restart: > >> + =A0 =A0 freed_pages =3D 0; > >> + =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> + > >> + =A0 =A0 { > >> + =A0 =A0 =A0 =A0 =A0 =A0 list_for_each_entry_reverse(page, &pool->lis= t, lru) { > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (limit_with_time > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 && time_afte= r(now, page->private)) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pages_to_free[freed_pages++]= =3D page; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages >=3D NUM_PAG= ES_TO_ADD) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Limit the= maximum time that we hold the lock > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* not to = block more important threads */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __list_del(p= age->lru.prev, &pool->list); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool->npages= -=3D freed_pages; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_= irqrestore(&pool->lock, irq_flags); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_pages_pu= t(pages_to_free, freed_pages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 goto restart; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 pool->npages -=3D freed_pages; > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __list_del(&page->lru, &pool= ->list); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore(&pool->lock, irq_flag= s); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (freed_pages) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_pages_put(pages_to_free,= freed_pages); > >> + =A0 =A0 } > >> +} > >> + > >> +/* periodicaly free unused pages from pool */ > >> +static void ttm_page_pool_timer_handler(unsigned long man_addr) > >> +{ > >> + =A0 =A0 int i; > >> + =A0 =A0 struct pool_manager *m =3D (struct pool_manager *)man_addr; > >> + > >> + =A0 =A0 for (i =3D 0; i < MAX_NUM_POOLS; ++i) > >> + =A0 =A0 =A0 =A0 =A0 =A0 ttm_page_pool_free(&_manager.pools[i], true); > >> + > >> + =A0 =A0 mod_timer(&m->timer, jiffies + msecs_to_jiffies(PAGE_FREE_IN= TERVAL)); > >> +} > >> + > >> +static void ttm_page_pool_timer_create(struct pool_manager *m) > >> +{ > >> + =A0 =A0 /* Create a timer that is fired every second to clean > >> + =A0 =A0 =A0* the free list */ > >> + =A0 =A0 init_timer(&m->timer); > >> + =A0 =A0 m->timer.expires =3D jiffies + msecs_to_jiffies(PAGE_FREE_IN= TERVAL); > >> + =A0 =A0 m->timer.function =3D ttm_page_pool_timer_handler; > >> + =A0 =A0 m->timer.data =3D (unsigned long)m; > >> +} > >> + > >> +static void ttm_page_pool_timer_free(struct pool_manager *m) > >> +{ > >> + =A0 =A0 del_timer_sync(&m->timer); > >> +} > >> + > >> +static void ttm_set_pages_caching(struct page **pages, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigned cpag= es) > >> +{ > >> + =A0 =A0 /* Set page caching */ > >> + =A0 =A0 switch (cstate) { > >> + =A0 =A0 case tt_uncached: > >> + =A0 =A0 =A0 =A0 =A0 =A0 set_pages_array_uc(pages, cpages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 break; > >> + =A0 =A0 case tt_wc: > >> + =A0 =A0 =A0 =A0 =A0 =A0 set_pages_array_wc(pages, cpages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 break; > >> + =A0 =A0 default: > >> + =A0 =A0 =A0 =A0 =A0 =A0 break; > >> + =A0 =A0 } > >> +} > >> +/** > >> + * Fill new pages to array. > >> + */ > >> +static int ttm_pages_pool_fill_locked(struct page_pool *pool, int fla= gs, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigned coun= t, > >> + =A0 =A0 =A0 =A0 =A0 =A0 unsigned long *irq_flags) > >> +{ > >> + =A0 =A0 unsigned pages_to_alloc =3D count - pool->npages; > >> + =A0 =A0 struct page **pages; > >> + =A0 =A0 struct page *page; > >> + =A0 =A0 struct list_head h; > >> + =A0 =A0 unsigned i, cpages; > >> + =A0 =A0 unsigned max_cpages =3D min(pages_to_alloc, > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 (unsigned)(PAGE_SIZE/sizeof(= *page))); > >> + > >> + =A0 =A0 /* reserve allready allocated pages from pool */ > >> + =A0 =A0 INIT_LIST_HEAD(&h); > >> + =A0 =A0 list_splice_init(&pool->list, &h); > >> + =A0 =A0 i =3D pool->npages; > >> + =A0 =A0 pages_to_alloc =3D count; > >> + =A0 =A0 pool->npages =3D 0; > >> + > >> + =A0 =A0 /* We are now safe to allow others users to modify the pool = so unlock > >> + =A0 =A0 =A0* it. Allocating pages is expensive operation. */ > >> + =A0 =A0 spin_unlock_irqrestore(&pool->lock, *irq_flags); > >> + > >> + =A0 =A0 pages =3D kmalloc(max_cpages*sizeof(*page), GFP_KERNEL); > >> + > >> + =A0 =A0 if (!pages) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm] unable to allocate ta= ble for new pages."); > >> + =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> + =A0 =A0 } > >> + > >> + =A0 =A0 /* i is set to number of pages already in pool */ > >> + =A0 =A0 for (cpages =3D 0; i < pages_to_alloc; ++i) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 page =3D alloc_page(pool->gfp_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (!page) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_ERR "[ttm] unabl= e to get page %u\n", i); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_put_pages(&h, flags, cst= ate); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_lock_irqsave(&pool->loc= k, *irq_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOMEM; > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> +#ifdef CONFIG_HIGHMEM > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* gfp flags of highmem page should never be= dma32 so we > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0* we should be fine in such case > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (!PageHighMem(page)) > >> +#endif > >> + =A0 =A0 =A0 =A0 =A0 =A0 { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pages[cpages++] =3D page; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (cpages =3D=3D max_cpages= ) { > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_set_page= s_caching(pages, cstate, cpages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 cpages =3D 0; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 list_add(&page->lru, &h); > >> + =A0 =A0 } > >> + > >> + =A0 =A0 if (cpages) > >> + =A0 =A0 =A0 =A0 =A0 =A0 ttm_set_pages_caching(pages, cstate, cpages); > >> + > >> + =A0 =A0 spin_lock_irqsave(&pool->lock, *irq_flags); > >> + > >> + =A0 =A0 list_splice(&h, &pool->list); > >> + > >> + =A0 =A0 pool->npages +=3D pages_to_alloc; > >> + > >> + =A0 =A0 return 0; > >> +} > >> + > >> +/* > >> + * On success pages list will hold count number of correctly > >> + * cached pages. > >> + */ > >> +int ttm_get_pages(struct list_head *pages, int flags, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigned coun= t) > >> +{ > >> + =A0 =A0 struct page_pool *pool =3D ttm_get_pool(flags, cstate); > >> + =A0 =A0 struct page *page =3D NULL; > >> + =A0 =A0 unsigned long irq_flags; > >> + =A0 =A0 int gfp_flags =3D 0; > >> + =A0 =A0 int retries =3D 1; > >> + =A0 =A0 int r; > >> + > >> + =A0 =A0 /* No pool for cached pages */ > >> + =A0 =A0 if (cstate =3D=3D tt_cached) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_ZERO; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_DMA32) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D GFP_DMA32; > >> + =A0 =A0 =A0 =A0 =A0 =A0 else > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_HIGHMEM; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* fallback in case of problems with pool al= location */ > >> +alloc_pages: > >> + =A0 =A0 =A0 =A0 =A0 =A0 for (r =3D 0; r < count; ++r) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 page =3D alloc_page(gfp_flag= s); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (!page) { > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KERN_= ERR "[ttm] unable to allocate page."); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return -ENOM= EM; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&page->lru, pages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> + =A0 =A0 } > >> + > >> + =A0 =A0 gfp_flags =3D pool->gfp_flags; > >> + > >> + =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> + =A0 =A0 do { > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* First we look if pool has the page we wan= t */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (pool->npages >=3D count) { > >> + > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* Find the cut location */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (count <=3D pool->npages/= 2) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned c = =3D 0; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_for_eac= h_entry(page, &pool->list, lru) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 if (++c >=3D count) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 break; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } else { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* We have t= o point to the last pages cut */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned c = =3D 0; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 unsigned rco= unt =3D pool->npages - count + 1; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_for_eac= h_entry_reverse(page, &pool->list, lru) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 if (++c >=3D rcount) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 break; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_cut_position(pages, &po= ol->list, &page->lru); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pool->npages -=3D count; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore(&pool= ->lock, irq_flags); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (flags & TTM_PAGE_FLAG_ZE= RO_ALLOC) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_for_eac= h_entry(page, pages, lru) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 clear_page(page_address(page)); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (retries =3D=3D 0) > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 break; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 --retries; > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 r =3D ttm_pages_pool_fill_locked(pool, flags= , cstate, > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 count, &irq_= flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (r) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 spin_unlock_irqrestore(&pool= ->lock, irq_flags); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return r; > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 } while (1); > >> + > >> + =A0 =A0 spin_unlock_irqrestore(&pool->lock, irq_flags); > >> + > >> + =A0 =A0 /* =A0Pool allocator failed without errors. we try to alloca= te new pages > >> + =A0 =A0 =A0* =A0instead. */ > >> + =A0 =A0 if (flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> + =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_ZERO; > >> + > >> + =A0 =A0 goto alloc_pages; > >> +} > >> + > >> +/* Put all pages in pages list to correct pool to wait for reuse */ > >> +void ttm_put_pages(struct list_head *pages, int flags, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate) > >> +{ > >> + =A0 =A0 struct page_pool *pool =3D ttm_get_pool(flags, cstate); > >> + =A0 =A0 struct page *p, *tmp; > >> + =A0 =A0 unsigned page_count =3D 0; > >> + =A0 =A0 unsigned long irq_flags; > >> + =A0 =A0 unsigned long now =3D jiffies; > >> + > >> + =A0 =A0 if (pool =3D=3D NULL) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 list_for_each_entry_safe(p, tmp, pages, lru)= { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(p); > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* We took the ownership -> clear the pages = list*/ > >> + =A0 =A0 =A0 =A0 =A0 =A0 INIT_LIST_HEAD(pages); > >> + =A0 =A0 =A0 =A0 =A0 =A0 return; > >> + =A0 =A0 } > >> + > >> + =A0 =A0 list_for_each_entry_safe(p, tmp, pages, lru) { > >> + > >> +#ifdef CONFIG_HIGHMEM > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* we don't have pool for highmem -> free th= em */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (PageHighMem(p)) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_del(&p->lru); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(p); > >> + =A0 =A0 =A0 =A0 =A0 =A0 } else > >> +#endif > >> + =A0 =A0 =A0 =A0 =A0 =A0 { > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 p->private =3D now; > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ++page_count; > >> + =A0 =A0 =A0 =A0 =A0 =A0 } > >> + > >> + =A0 =A0 } > >> + > >> + =A0 =A0 spin_lock_irqsave(&pool->lock, irq_flags); > >> + =A0 =A0 list_splice(pages, &pool->list); > >> + =A0 =A0 pool->npages +=3D page_count; > >> + =A0 =A0 spin_unlock_irqrestore(&pool->lock, irq_flags); > >> + > >> + =A0 =A0 /* We took the ownership -> clear the pages list*/ > >> + =A0 =A0 INIT_LIST_HEAD(pages); > >> +} > >> + > >> + > >> +int ttm_page_alloc_init(void) > >> +{ > >> + =A0 =A0 if (_manager.page_alloc_inited) > >> + =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> + > >> + =A0 =A0 ttm_page_pool_init_locked(&_manager.wc_pool); > >> + =A0 =A0 _manager.wc_pool.gfp_flags =3D GFP_HIGHUSER; > >> + =A0 =A0 ttm_page_pool_init_locked(&_manager.uc_pool); > >> + =A0 =A0 _manager.uc_pool.gfp_flags =3D GFP_HIGHUSER; > >> + =A0 =A0 ttm_page_pool_init_locked(&_manager.wc_pool_dma32); > >> + =A0 =A0 _manager.wc_pool_dma32.gfp_flags =3D GFP_USER | GFP_DMA32; > >> + =A0 =A0 ttm_page_pool_init_locked(&_manager.uc_pool_dma32); > >> + =A0 =A0 _manager.uc_pool_dma32.gfp_flags =3D GFP_USER | GFP_DMA32; > >> + > >> + =A0 =A0 _manager.time_to_wait =3D msecs_to_jiffies(TIME_TO_KEEP_PAGE= _IN_POOL); > >> + =A0 =A0 ttm_page_pool_timer_create(&_manager); > >> + =A0 =A0 _manager.page_alloc_inited =3D 1; > >> + =A0 =A0 return 0; > >> +} > >> + > >> +void ttm_page_alloc_fini(void) > >> +{ > >> + =A0 =A0 int i; > >> + > >> + =A0 =A0 if (!_manager.page_alloc_inited) > >> + =A0 =A0 =A0 =A0 =A0 =A0 return; > >> + > >> + =A0 =A0 ttm_page_pool_timer_free(&_manager); > >> + > >> + =A0 =A0 for (i =3D 0; i < MAX_NUM_POOLS; ++i) > >> + =A0 =A0 =A0 =A0 =A0 =A0 ttm_page_pool_free(&_manager.pools[i], false= ); > >> + > >> + =A0 =A0 _manager.page_alloc_inited =3D 0; > >> +} > >> diff --git a/drivers/gpu/drm/ttm/ttm_page_alloc.h b/drivers/gpu/drm/tt= m/ttm_page_alloc.h > >> new file mode 100644 > >> index 0000000..eb0ddf4 > >> --- /dev/null > >> +++ b/drivers/gpu/drm/ttm/ttm_page_alloc.h > >> @@ -0,0 +1,38 @@ > >> +/* > >> + * Copyright (c) Red Hat Inc. > >> + > >> + * Permission is hereby granted, free of charge, to any person obtain= ing a > >> + * copy of this software and associated documentation files (the "Sof= tware"), > >> + * to deal in the Software without restriction, including without lim= itation > >> + * the rights to use, copy, modify, merge, publish, distribute, sub l= icense, > >> + * and/or sell copies of the Software, and to permit persons to whom = the > >> + * Software is furnished to do so, subject to the following condition= s: > >> + * > >> + * The above copyright notice and this permission notice (including t= he > >> + * next paragraph) shall be included in all copies or substantial por= tions > >> + * of the Software. > >> + * > >> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EX= PRESS OR > >> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTAB= ILITY, > >> + * FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT= SHALL > >> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES = OR OTHER > >> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, AR= ISING > >> + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER > >> + * DEALINGS IN THE SOFTWARE. > >> + * > >> + * Authors: Dave Airlie > >> + * =A0 =A0 =A0 =A0 =A0Jerome Glisse > >> + */ > >> +#ifndef TTM_PAGE_ALLOC > >> +#define TTM_PAGE_ALLOC > >> + > >> +#include "ttm/ttm_bo_driver.h" > >> + > >> +void ttm_put_pages(struct list_head *pages, int flags, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate); > >> +int ttm_get_pages(struct list_head *pages, int flags, > >> + =A0 =A0 =A0 =A0 =A0 =A0 enum ttm_caching_state cstate, unsigned coun= t); > >> +int ttm_page_alloc_init(void); > >> +void ttm_page_alloc_fini(void); > >> + > >> +#endif > >> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt= .c > >> index 3d47a2c..d4e20f6 100644 > >> --- a/drivers/gpu/drm/ttm/ttm_tt.c > >> +++ b/drivers/gpu/drm/ttm/ttm_tt.c > >> @@ -38,6 +38,7 @@ > >> =A0#include "ttm/ttm_module.h" > >> =A0#include "ttm/ttm_bo_driver.h" > >> =A0#include "ttm/ttm_placement.h" > >> +#include "ttm_page_alloc.h" > >> > >> =A0static int ttm_tt_swapin(struct ttm_tt *ttm); > >> > >> @@ -72,21 +73,6 @@ static void ttm_tt_free_page_directory(struct ttm_t= t *ttm) > >> =A0 =A0 =A0 ttm->pages =3D NULL; > >> =A0} > >> > >> -static struct page *ttm_tt_alloc_page(unsigned page_flags) > >> -{ > >> - =A0 =A0 gfp_t gfp_flags =3D GFP_USER; > >> - > >> - =A0 =A0 if (page_flags & TTM_PAGE_FLAG_ZERO_ALLOC) > >> - =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_ZERO; > >> - > >> - =A0 =A0 if (page_flags & TTM_PAGE_FLAG_DMA32) > >> - =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_DMA32; > >> - =A0 =A0 else > >> - =A0 =A0 =A0 =A0 =A0 =A0 gfp_flags |=3D __GFP_HIGHMEM; > >> - > >> - =A0 =A0 return alloc_page(gfp_flags); > >> -} > >> - > >> =A0static void ttm_tt_free_user_pages(struct ttm_tt *ttm) > >> =A0{ > >> =A0 =A0 =A0 int write; > >> @@ -127,15 +113,21 @@ static void ttm_tt_free_user_pages(struct ttm_tt= *ttm) > >> =A0static struct page *__ttm_tt_get_page(struct ttm_tt *ttm, int index) > >> =A0{ > >> =A0 =A0 =A0 struct page *p; > >> + =A0 =A0 struct list_head h; > >> =A0 =A0 =A0 struct ttm_mem_global *mem_glob =3D ttm->glob->mem_glob; > >> =A0 =A0 =A0 int ret; > >> > >> =A0 =A0 =A0 while (NULL =3D=3D (p =3D ttm->pages[index])) { > >> - =A0 =A0 =A0 =A0 =A0 =A0 p =3D ttm_tt_alloc_page(ttm->page_flags); > >> > >> - =A0 =A0 =A0 =A0 =A0 =A0 if (!p) > >> + =A0 =A0 =A0 =A0 =A0 =A0 INIT_LIST_HEAD(&h); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 ret =3D ttm_get_pages(&h, ttm->page_flags, t= tm->caching_state, 1); > >> + > >> + =A0 =A0 =A0 =A0 =A0 =A0 if (ret !=3D 0) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return NULL; > >> > >> + =A0 =A0 =A0 =A0 =A0 =A0 p =3D list_first_entry(&h, struct page, lru); > >> + > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D ttm_mem_global_alloc_page(mem_glob= , p, false, false); > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (unlikely(ret !=3D 0)) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 goto out_err; > >> @@ -244,10 +236,10 @@ static int ttm_tt_set_caching(struct ttm_tt *ttm, > >> =A0 =A0 =A0 if (ttm->caching_state =3D=3D c_state) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> > >> - =A0 =A0 if (c_state !=3D tt_cached) { > >> - =A0 =A0 =A0 =A0 =A0 =A0 ret =3D ttm_tt_populate(ttm); > >> - =A0 =A0 =A0 =A0 =A0 =A0 if (unlikely(ret !=3D 0)) > >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 return ret; > >> + =A0 =A0 if (ttm->state =3D=3D tt_unpopulated) { > >> + =A0 =A0 =A0 =A0 =A0 =A0 /* Change caching but don't populate */ > >> + =A0 =A0 =A0 =A0 =A0 =A0 ttm->caching_state =3D c_state; > >> + =A0 =A0 =A0 =A0 =A0 =A0 return 0; > >> =A0 =A0 =A0 } > >> > >> =A0 =A0 =A0 if (ttm->caching_state =3D=3D tt_cached) > >> @@ -298,25 +290,32 @@ EXPORT_SYMBOL(ttm_tt_set_placement_caching); > >> =A0static void ttm_tt_free_alloced_pages(struct ttm_tt *ttm) > >> =A0{ > >> =A0 =A0 =A0 int i; > >> + =A0 =A0 unsigned count =3D 0; > >> + =A0 =A0 struct list_head h; > >> =A0 =A0 =A0 struct page *cur_page; > >> =A0 =A0 =A0 struct ttm_backend *be =3D ttm->be; > >> > >> + =A0 =A0 INIT_LIST_HEAD(&h); > >> + > >> =A0 =A0 =A0 if (be) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 be->func->clear(be); > >> - =A0 =A0 (void)ttm_tt_set_caching(ttm, tt_cached); > >> =A0 =A0 =A0 for (i =3D 0; i < ttm->num_pages; ++i) { > >> + > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 cur_page =3D ttm->pages[i]; > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm->pages[i] =3D NULL; > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (cur_page) { > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (page_count(cur_page) != =3D 1) > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 printk(KER= N_ERR TTM_PFX > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0"Erroneous page count. " > >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0"Leaking pages.\n"); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0"Leaking pages (%d/%d).\n", > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0page_count(cur_page), count); > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ttm_mem_global_free_page(t= tm->glob->mem_glob, > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0cur_page); > >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 __free_page(cur_page); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 list_add(&cur_page->lru, &h); > >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 count++; > >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 } > >> =A0 =A0 =A0 } > >> + =A0 =A0 ttm_put_pages(&h, ttm->page_flags, ttm->caching_state); > >> =A0 =A0 =A0 ttm->state =3D tt_unpopulated; > >> =A0 =A0 =A0 ttm->first_himem_page =3D ttm->num_pages; > >> =A0 =A0 =A0 ttm->last_lomem_page =3D -1; > >> -- > >> 1.6.3.3 > >> > > > >> ----------------------------------------------------------------------= -------- > >> Download Intel® Parallel Studio Eval > >> Try the new software tools for yourself. Speed compiling, find bugs > >> proactively, and fine-tune applications for parallel performance. > >> See why Intel Parallel Studio got high marks during beta. > >> http://p.sf.net/sfu/intel-sw-dev > > > >> -- > >> _______________________________________________ > >> Dri-devel mailing list > >> Dri-devel@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/dri-devel > > > > > = ---------------------------------------------------------------------------= --- Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev --