improve reuse implementation
cover improve reuse implementation
From: "Xiong, James" <[email protected]>
With gem_reuse enabled, when a buffer size is different than the sizes of buckets, it is aligned to the next bucket's size, which means about 25% more memory than the requested is allocated in the worst senario. For example:
Orignal size Actual
32KB+1Byte 40KB
.
.
.
8MB+1Byte 10MB
.
.
.
96MB+1Byte 112MB
This is very memory expensive and make the reuse feature less favorable than it deserves to be.
This series aligns the reuse buffer size on page size instead to save memory. Performed gfxbench tests on Gen9 without LLC, the performances and reuse ratioes (reuse count/allocation count) were same as before, saved memory usage by 1% ~ 7%(gl_manhattan: peak allocated memory size was reduced from 448401408 to 419078144).
v2: split the patch to a series of small functional changes (Chris)
Xiong, James (5):
drm: add a macro DRMLISTFOREACHENTRYREVERSE
intel: reorganize internal function
intel: get a cached bucket by size
intel: get a cached buffer by size for reuse
intel: purge cached bucket when its cached buffer is evicted
intel/intel_bufmgr_gem.c | 180 +++++++++++++++++++++++++---------------------- libdrm_lists.h | 6 ++ 2 files changed, 100 insertions(+), 86 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH 1/5 drm: add a macro DRMLISTFOREACHENTRYREVERSE
From: "Xiong, James" <[email protected]>
it goes through DRMLIST in a reverse order
--- libdrm_lists.h | 6 ++++++ 1 file changed, 6 insertions(+)
libdrm_lists.h +6 -0
@@ -101,6 +101,12 @@ typedef struct _drmMMListHead (__item) = DRMLISTENTRY(typeof(*__item), \ (__item)->__head.next, __head)) +#define DRMLISTFOREACHENTRYREVERSE (__item, __list, __head) \ + for ((__item) = DRMLISTENTRY(typeof(*__item), (__list)->prev, __head); \ + &(__item)->__head != (__list); \ + (__item) = DRMLISTENTRY(typeof(*__item), \ + (__item)->__head.prev, __head)) +#define DRMLISTFOREACHENTRYSAFE (__item, __temp, __list, __head) \ for ((__item) = DRMLISTENTRY(typeof(*__item), (__list)->next, __head), \ (__temp) = DRMLISTENTRY(typeof(*__item), \
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH 2/5 intel: reorganize internal function
In reply to James XiongFrom: "Xiong, James" <[email protected]>
split drm_intel_gem_bo_alloc_internal, and add a function to search for a suitable buffer for given size for reuse.
--- intel/intel_bufmgr_gem.c | 141 ++++++++++++++++++++++++----------------------- 1 file changed, 73 insertions(+), 68 deletions(-)
intel/intel_bufmgr_gem.c +73 -68
@@ -704,6 +704,71 @@ drm_intel_gem_bo_cache_purge_bucket(drm_intel_bufmgr_gem *bufmgr_gem, } } +static drm_intel_bo_gem * +drm_intel_gem_bo_cached_for_size (drm_intel_bufmgr_gem *bufmgr_gem, +unsigned long size, +uint32_t tiling_mode, +unsigned long stride, +unsigned long alignment, +bool for_render) +{ +struct drm_intel_gem_bo_bucket *bucket = +drm_intel_gem_bo_bucket_for_size (bufmgr_gem, size); + +if (bucket !=NULL ) { +drm_intel_bo_gem *bo_gem, *temp_bo_gem; +retry: + bo_gem =NULL ; +if (for_render) { +/* Allocate new render-target BOs from the tail (MRU) +* of the list, as it will likely be hot in the GPU +* cache and in the aperture for us. +*/ + bo_gem =DRMLISTENTRY (drm_intel_bo_gem, + bucket->head.prev, head); +DRMLISTDEL (&bo_gem->head); + bo_gem->bo.align = alignment; + }else { +assert (alignment ==0 ); +/* For non-render-target BOs (where we're probably +* going to map it first thing in order to fill it +* with data), check if the last BO in the cache is +* unbusy, and only reuse in that case. Otherwise, +* allocating a new buffer is probably faster than +* waiting for the GPU to finish. +*/ + bo_gem =DRMLISTENTRY (drm_intel_bo_gem, + bucket->head.next, head); +if (!drm_intel_gem_bo_busy (&bo_gem->bo)) { +DRMLISTDEL (&bo_gem->head); + }else { + bo_gem =NULL ; + } + } + +if (bo_gem) { +if (!drm_intel_gem_bo_madvise_internal + (bufmgr_gem, bo_gem,I915_MADV_WILLNEED )) { +drm_intel_gem_bo_free (&bo_gem->bo); +drm_intel_gem_bo_cache_purge_bucket (bufmgr_gem, + bucket); +return NULL ; + } + +if (drm_intel_gem_bo_set_tiling_internal (&bo_gem->bo, + tiling_mode, + stride)) { +drm_intel_gem_bo_free (&bo_gem->bo); + goto retry; + } + } + +return bo_gem; + } + +return NULL ; +} +static drm_intel_bo * drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr, const char *name,
@@ -715,81 +780,21 @@ drm_intel_gem_bo_alloc_internal(drm_intel_bufmgr *bufmgr, {drm_intel_bufmgr_gem *bufmgr_gem = (drm_intel_bufmgr_gem *) bufmgr;drm_intel_bo_gem *bo_gem; -unsigned int page_size =getpagesize ();int ret; -struct drm_intel_gem_bo_bucket *bucket; -bool alloc_from_cache; -unsigned long bo_size;bool for_render = false;if (flags &BO_ALLOC_FOR_RENDER ) for_render = true; -/* Round the allocated size up to a power of two number of pages. */ - bucket =drm_intel_gem_bo_bucket_for_size (bufmgr_gem, size); - -/* If we don't have caching at this size, don't actually round the -* allocation up. -*/ -if (bucket ==NULL ) { - bo_size = size; -if (bo_size < page_size) - bo_size = page_size; - }else { - bo_size = bucket->size; - } +/* first align the size on page boundary */ + size =ALIGN (size,getpagesize ());pthread_mutex_lock (&bufmgr_gem->lock);/* Get a buffer out of the cache if available */ -retry: - alloc_from_cache = false; -if (bucket !=NULL && !DRMLISTEMPTY (&bucket->head)) { -if (for_render) { -/* Allocate new render-target BOs from the tail (MRU) -* of the list, as it will likely be hot in the GPU -* cache and in the aperture for us. -*/ - bo_gem =DRMLISTENTRY (drm_intel_bo_gem, - bucket->head.prev, head); -DRMLISTDEL (&bo_gem->head); - alloc_from_cache = true; - bo_gem->bo.align = alignment; - }else { -assert (alignment ==0 ); -/* For non-render-target BOs (where we're probably -* going to map it first thing in order to fill it -* with data), check if the last BO in the cache is -* unbusy, and only reuse in that case. Otherwise, -* allocating a new buffer is probably faster than -* waiting for the GPU to finish. -*/ - bo_gem =DRMLISTENTRY (drm_intel_bo_gem, - bucket->head.next, head); -if (!drm_intel_gem_bo_busy (&bo_gem->bo)) { - alloc_from_cache = true; -DRMLISTDEL (&bo_gem->head); - } - } + bo_gem =drm_intel_gem_bo_cached_for_size (bufmgr_gem, size, tiling_mode, + stride, alignment, for_render); -if (alloc_from_cache) { -if (!drm_intel_gem_bo_madvise_internal - (bufmgr_gem, bo_gem,I915_MADV_WILLNEED )) { -drm_intel_gem_bo_free (&bo_gem->bo); -drm_intel_gem_bo_cache_purge_bucket (bufmgr_gem, - bucket); - goto retry; - } - -if (drm_intel_gem_bo_set_tiling_internal (&bo_gem->bo, - tiling_mode, - stride)) { -drm_intel_gem_bo_free (&bo_gem->bo); - goto retry; - } - } - } - -if (!alloc_from_cache) { +if (bo_gem ==NULL ) {struct drm_i915_gem_create create; bo_gem =calloc (1 ,sizeof (*bo_gem));
@@ -800,10 +805,10 @@ retry:list (vma_list), so better set the list head here */ DRMINITLISTHEAD (&bo_gem->vma_list); - bo_gem->bo.size = bo_size; + bo_gem->bo.size = size;memclear (create); - create.size = bo_size; + create.size = size; ret = drmIoctl(bufmgr_gem->fd, DRM_IOCTL_I915_GEM_CREATE,
@@ -844,7 +849,7 @@ retry:pthread_mutex_unlock (&bufmgr_gem->lock);DBG ("bo_create: buf %d (%s) %ldb\n" , - bo_gem->gem_handle, bo_gem->name, size); + bo_gem->gem_handle, bo_gem->name, bo_gem->bo.size);return &bo_gem->bo;
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH 3/5 intel: get a cached bucket by size
In reply to James XiongFrom: "Xiong, James" <[email protected]>
cached buckets are sorted by size in increasing order, each now contains cached buffers with different sizes. A buffer with size
Quoted 1 line
= buckets[n].size and < buckets[n+1].size is put in bucket n
for future reuse.
--- intel/intel_bufmgr_gem.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
intel/intel_bufmgr_gem.c +4 -5
@@ -402,11 +402,10 @@ drm_intel_gem_bo_bucket_for_size(drm_intel_bufmgr_gem *bufmgr_gem, {int i; -for (i =0 ; i < bufmgr_gem->num_buckets; i++) { -struct drm_intel_gem_bo_bucket *bucket = - &bufmgr_gem->cache_bucket[i]; -if (bucket->size >= size) { -return bucket; +for (i =0 ; i < bufmgr_gem->num_buckets -1 ; i++) { +if (size >= bufmgr_gem->cache_bucket[i].size && + size < bufmgr_gem->cache_bucket[i+1 ].size) { +return &bufmgr_gem->cache_bucket[i]; } }
Signature · 6 lines
2.7.4 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 4/5 intel: get a cached buffer by size for reuse
In reply to James XiongFrom: "Xiong, James" <[email protected]>
Previously all cached buffers in a given bucket were same sized, when reusing, the MRU buffer at the tail was poped out. With the new implementation, we go through the buffer list in a reverse order to search for a MRU buffer with a suitable size.
--- intel/intel_bufmgr_gem.c | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-)
intel/intel_bufmgr_gem.c +15 -10
@@ -723,10 +723,14 @@ retry:* of the list, as it will likely be hot in the GPU * cache and in the aperture for us. */ - bo_gem =DRMLISTENTRY (drm_intel_bo_gem, - bucket->head.prev, head); -DRMLISTDEL (&bo_gem->head); - bo_gem->bo.align = alignment; + DRMLISTFOREACHENTRYREVERSE(temp_bo_gem, &bucket->head, head) { +if (temp_bo_gem->bo.size >= size) { + bo_gem = temp_bo_gem; +DRMLISTDEL (&bo_gem->head); + bo_gem->bo.align = alignment; +break ; + } + } }else {assert (alignment ==0 );/* For non-render-target BOs (where we're probably
@@ -736,12 +740,13 @@ retry:* allocating a new buffer is probably faster than * waiting for the GPU to finish. */ - bo_gem =DRMLISTENTRY (drm_intel_bo_gem, - bucket->head.next, head); -if (!drm_intel_gem_bo_busy (&bo_gem->bo)) { -DRMLISTDEL (&bo_gem->head); - }else { - bo_gem =NULL ; + DRMLISTFOREACHENTRY(temp_bo_gem, &bucket->head, head) { +if (temp_bo_gem->bo.size >= size && + !drm_intel_gem_bo_busy (&temp_bo_gem->bo)) { + bo_gem = temp_bo_gem; +DRMLISTDEL (&bo_gem->head); +break ; + } } }
Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
PATCH 5/5 intel: purge cached bucket when its cached buffer is evicted
In reply to James XiongFrom: "Xiong, James" <[email protected]>
Previously when a cached MRU buffer was found to be evicted by kernel, the bucket was emptied. The new implementation purged these buffers that were freed before the evicted one.
--- intel/intel_bufmgr_gem.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-)
intel/intel_bufmgr_gem.c +14 -15
@@ -684,22 +684,20 @@ drm_intel_gem_bo_madvise(drm_intel_bo *bo, int madv) madv); } -/* drop the oldest entries that have been purged by the kernel */ +/* drop the entries that are older than the given time */ static void drm_intel_gem_bo_cache_purge_bucket (drm_intel_bufmgr_gem *bufmgr_gem, -struct drm_intel_gem_bo_bucket *bucket) +struct drm_intel_gem_bo_bucket *bucket, +time_t time) { -while (!DRMLISTEMPTY (&bucket->head)) { -drm_intel_bo_gem *bo_gem; - - bo_gem =DRMLISTENTRY (drm_intel_bo_gem, - bucket->head.next, head); -if (drm_intel_gem_bo_madvise_internal - (bufmgr_gem, bo_gem,I915_MADV_DONTNEED )) -break ; - -DRMLISTDEL (&bo_gem->head); -drm_intel_gem_bo_free (&bo_gem->bo); +drm_intel_bo_gem *bo_gem, *temp; + DRMLISTFOREACHENTRYSAFE(bo_gem, temp, &bucket->head, head) { +if (bo_gem->free_time >= time) { +drm_intel_gem_bo_madvise_internal + (bufmgr_gem, bo_gem,I915_MADV_DONTNEED ); +DRMLISTDEL (&bo_gem->head); +drm_intel_gem_bo_free (&bo_gem->bo); + } } }
@@ -753,9 +751,10 @@ retry:if (bo_gem) {if (!drm_intel_gem_bo_madvise_internal (bufmgr_gem, bo_gem,I915_MADV_WILLNEED )) { -drm_intel_gem_bo_free (&bo_gem->bo);drm_intel_gem_bo_cache_purge_bucket (bufmgr_gem, - bucket); + bucket, + bo_gem->free_time); +drm_intel_gem_bo_free (&bo_gem->bo);return NULL ; }
Signature · 6 lines
2.7.4 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 21 lines · James Xiong
From: "Xiong, James" <[email protected]>
With
gem_reuseenabled, when a buffer size is different than the sizes of buckets, it is aligned to the next bucket's size, which means about 25% more memory than the requested is allocated in the worst senario. For example:Orignal size Actual
32KB+1Byte 40KB
.
.
.
8MB+1Byte 10MB
.
.
.
96MB+1Byte 112MBThis is very memory expensive and make the reuse feature less favorable than it deserves to be.
Quoted 7 lines
This series aligns the reuse buffer size on page size instead to save memory. Performed gfxbench tests on Gen9 without LLC, the performances and reuse ratioes (reuse count/allocation count) were same as before, saved memory usage by 1% ~ 7%(gl_manhattan: peak allocated memory size was reduced from 448401408 to 419078144).
v2: split the patch to a series of small functional changes (Chris)
Mesa gen driver stopped using the libdrm buffer allocator. The gen2/3 driver still uses it, but I think that's not the one you benchmarked. The 17.1 release was the first one with that change.
I think you want to port your changes over to mesa to future proof it, merging this to upstream makes little sense.
-Daniel
Quoted 19 lines
Xiong, James (5):
drm: add a macro DRMLISTFOREACHENTRYREVERSE
intel: reorganize internal function
intel: get a cached bucket by size
intel: get a cached buffer by size for reuse
intel: purge cached bucket when its cached buffer is evictedintel/intel_bufmgr_gem.c | 180 +++++++++++++++++++++++++---------------------- libdrm_lists.h | 6 ++ 2 files changed, 100 insertions(+), 86 deletions(-)Signature · 6 lines
2.7.4 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 28 lines · Daniel Vetter
Quoted 27 lines · James Xiong
From: "Xiong, James" <[email protected]>
With
gem_reuseenabled, when a buffer size is different than the sizes of buckets, it is aligned to the next bucket's size, which means about 25% more memory than the requested is allocated in the worst senario. For example:Orignal size Actual
32KB+1Byte 40KB
.
.
.
8MB+1Byte 10MB
.
.
.
96MB+1Byte 112MBThis is very memory expensive and make the reuse feature less favorable than it deserves to be.
This series aligns the reuse buffer size on page size instead to save memory. Performed gfxbench tests on Gen9 without LLC, the performances and reuse ratioes (reuse count/allocation count) were same as before, saved memory usage by 1% ~ 7%(gl_manhattan: peak allocated memory size was reduced from 448401408 to 419078144).
Quoted 8 lines
Quoted 1 line
v2: split the patch to a series of small functional changes (Chris)
Mesa gen driver stopped using the libdrm buffer allocator. The gen2/3 driver still uses it, but I think that's not the one you benchmarked. The 17.1 release was the first one with that change.
I think you want to port your changes over to mesa to future proof it, merging this to upstream makes little sense.
Perhaps it can have in both? After all i915, libva and beignet still make use of libdrm_intel.
The Mesa copy has changed drastically so this series might need serious rework.
-Emil
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Quoted 35 lines · Emil Velikov
Quoted 34 lines · Daniel Vetter
Quoted 21 lines · James Xiong
From: "Xiong, James" <[email protected]>
With
gem_reuseenabled, when a buffer size is different than the sizes of buckets, it is aligned to the next bucket's size, which means about 25% more memory than the requested is allocated in the worst senario. For example:Orignal size Actual
32KB+1Byte 40KB
.
.
.
8MB+1Byte 10MB
.
.
.
96MB+1Byte 112MBThis is very memory expensive and make the reuse feature less favorable than it deserves to be.
Quoted 7 lines
This series aligns the reuse buffer size on page size instead to save memory. Performed gfxbench tests on Gen9 without LLC, the performances and reuse ratioes (reuse count/allocation count) were same as before, saved memory usage by 1% ~ 7%(gl_manhattan: peak allocated memory size was reduced from 448401408 to 419078144).
v2: split the patch to a series of small functional changes (Chris)
Mesa gen driver stopped using the libdrm buffer allocator. The gen2/3 driver still uses it, but I think that's not the one you benchmarked. The 17.1 release was the first one with that change.
Quoted 6 lines
Quoted 2 lines
I think you want to port your changes over to mesa to future proof it, merging this to upstream makes little sense.
Perhaps it can have in both? After all i915, libva and beignet still make use of
libdrm_intel.
The Mesa copy has changed drastically so this series might need serious rework.
Yeah, we can have both, but then the series needs to be updated to have relative improvements for the stacks still using libdrm. Apparently our new media stack has it's own copy of libdrm (like everyone else nowadays), so fairly limited. But I think the platform this was done for is still using the older libva, so would benefit.
Just want to make sure the commit justification reflects actual upstream reality instead of what we had 2 years ago.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 37 lines
-----Original Message-----
From: dri-devel [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Friday, March 16, 2018 1:43 AM
To: Xiong, James <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: [PATCH libdrm 0/5] improve reuse implementationQuoted 27 lines · James Xiong
From: "Xiong, James" <[email protected]>
With
gem_reuseenabled, when a buffer size is different than the sizes of buckets, it is aligned to the next bucket's size, which means about 25% more memory than the requested is allocated in the worst senario. For example:Orignal size Actual
32KB+1Byte 40KB
.
.
.
8MB+1Byte 10MB
.
.
.
96MB+1Byte 112MBThis is very memory expensive and make the reuse feature less favorable than it deserves to be.
This series aligns the reuse buffer size on page size instead to save memory. Performed gfxbench tests on Gen9 without LLC, the performances and reuse ratioes (reuse count/allocation count) were same as before, saved memory usage by 1% ~ 7%(gl_manhattan: peak allocated memory size was reduced from 448401408 to 419078144).
Quoted 5 lines
Quoted 1 line
v2: split the patch to a series of small functional changes (Chris)
Mesa gen driver stopped using the libdrm buffer allocator. The gen2/3 driver still uses it, but I think that's not the one you benchmarked. The 17.1 release was the first one with that change.
Thanks for the tip, Daniel. I tested it on mesa 13.0.5.
Quoted 2 lines
I think you want to port your changes over to mesa to future proof it, merging this to upstream makes little sense. -Daniel
I am not sure about mesa and other components that use embedded libdrm, at least for the project I am currently working on, though we have our owner libdrm repo, we heavily rely on the open source community, and we keep an eye on the upstream and pull in fixes we think might help
with our customers, actually that's the only way we merge a libdrm patch to our repo. I think it would help to merge the patches to upstream(only if the patches make sense of course) so that the component owner can decide themselves whether to take and rebase it to their own, IMHO.
Quoted 21 lines
Quoted 13 lines
Xiong, James (5):
drm: add a macro DRMLISTFOREACHENTRYREVERSE
intel: reorganize internal function
intel: get a cached bucket by size
intel: get a cached buffer by size for reuse
intel: purge cached bucket when its cached buffer is evictedintel/intel_bufmgr_gem.c | 180 +++++++++++++++++++++++++---------------------- libdrm_lists.h | 6 ++ 2 files changed, 100 insertions(+), 86 deletions(-)Quoted 7 lines
--
2.7.4_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Quoted 8 lines
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx