Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
Subsystems (7)
Dmitry Osipenko
maintainerv8 cover Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
This series:
1. Adds common drm-shmem memory shrinker
2. Enables shrinker for VirtIO-GPU driver
3. Switches Panfrost driver to the common shrinker
Related patches:
Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise igt: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvise
https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise
I'll upstream Mesa and igt patches once kernel part will be merged.
Changelog:
v8: - Rebased on top of recent linux-next that now has dma-buf locking
convention patches merged, which was blocking shmem shrinker before.
- Shmem shrinker now uses new drm_gem_lru helper.
- Dropped Steven Price t-b from the Panfrost patch because code changed significantly since v6 and should be re-tested.
v7: - dma-buf locking convention
v6: https://lore.kernel.org/dri-devel/[email protected]/
Dmitry Osipenko (7):
drm/msm/gem: Prevent blocking within shrinker loop drm/shmem-helper: Don't use vmap_use_count for dma-bufs drm/shmem-helper: Switch to reservation lock drm/shmem-helper: Add memory shrinker
drm/gem: Add drm_gem_pin_unlocked() drm/virtio: Support memory shrinking drm/panfrost: Switch to generic memory shrinker
drivers/gpu/drm/drm_gem.c | 38 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 648 ++++++++++++++---- drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/msm/msm_gem_shrinker.c | 8 +- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 26 +- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 - .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 122 ---- drivers/gpu/drm/panfrost/panfrost_job.c | 18 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 18 +- drivers/gpu/drm/virtio/virtgpu_gem.c | 55 ++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 + drivers/gpu/drm/virtio/virtgpu_kms.c | 9 + drivers/gpu/drm/virtio/virtgpu_object.c | 138 +++- drivers/gpu/drm/virtio/virtgpu_plane.c | 22 +- drivers/gpu/drm/virtio/virtgpu_vq.c | 40 ++ include/drm/drm_device.h | 4 + include/drm/drm_gem.h | 7 +- include/drm/drm_gem_shmem_helper.h | 99 ++- include/uapi/drm/virtgpu_drm.h | 14 + 23 files changed, 1003 insertions(+), 374 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
Signature · 1 line
2.37.3
Dmitry Osipenko
maintainerPATCH v8 1/7 drm/msm/gem: Prevent blocking within shrinker loop
Consider this scenario:
1. APP1 continuously creates lots of small GEMs
2. APP2 triggers drop_caches
3. Shrinker starts to evict APP1 GEMs, while APP1 produces new purgeable GEMs
4. msm_gem_shrinker_scan() returns non-zero number of freed pages and causes shrinker to try shrink more
5. msm_gem_shrinker_scan() returns non-zero number of freed pages again, goto 4
6. The APP2 is blocked in drop_caches until APP1 stops producing purgeable GEMs
To prevent this blocking scenario, check number of remaining pages that GPU shrinker couldn't release due to a GEM locking contention. If there are no remaining pages left to shrink, then there is no need to free up more pages and shrinker may break out from the loop.
This problem was uncovered during shrinker/madvise IOCTL testing of virtio-gpu driver. The MSM driver is affected in the same way.
--- drivers/gpu/drm/drm_gem.c | 9 +++++++-- drivers/gpu/drm/msm/msm_gem_shrinker.c | 8 ++++++-- include/drm/drm_gem.h | 4 +++- 3 files changed, 16 insertions(+), 5 deletions(-)
drivers/gpu/drm/drm_gem.c +7 -2
@@ -1375,10 +1375,13 @@ EXPORT_SYMBOL(drm_gem_lru_move_tail);* * @lru: The LRU to scan * @nr_to_scan: The number of pages to try to reclaim +* @remaining: The number of pages left to reclaim * @shrink: Callback to try to shrink/reclaim the object. */ unsigned long -drm_gem_lru_scan (struct drm_gem_lru *lru,unsigned nr_to_scan, +drm_gem_lru_scan (struct drm_gem_lru *lru, +unsigned int nr_to_scan, +unsigned long *remaining,bool (*shrink)(struct drm_gem_object *obj)) {struct drm_gem_lru still_in_lru;
@@ -1417,8 +1420,10 @@ drm_gem_lru_scan(struct drm_gem_lru *lru, unsigned nr_to_scan,* hit shrinker in response to trying to get backing pages * for this obj (ie. while it's lock is already held) */ -if (!dma_resv_trylock (obj->resv)) +if (!dma_resv_trylock (obj->resv)) { + *remaining += obj->size >>PAGE_SHIFT ; goto tail; + }if (shrink (obj)) { freed += obj->size >>PAGE_SHIFT ;
drivers/gpu/drm/msm/msm_gem_shrinker.c +6 -2
@@ -116,12 +116,14 @@ msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) };long nr = sc->nr_to_scan;unsigned long freed =0 ; +unsigned long remaining =0 ;for (unsigned i =0 ; (nr >0 ) && (i <ARRAY_SIZE (stages)); i++) {if (!stages[i].cond)continue ; stages[i].freed = -drm_gem_lru_scan (stages[i].lru, nr, stages[i].shrink); +drm_gem_lru_scan (stages[i].lru, nr, &remaining, + stages[i].shrink); nr -= stages[i].freed; freed += stages[i].freed; }
@@ -132,7 +134,7 @@ msm_gem_shrinker_scan(struct shrinker *shrinker, struct shrink_control *sc) stages[3].freed); } -return (freed >0 ) ? freed :SHRINK_STOP ; +return (freed >0 && remaining >0 ) ? freed :SHRINK_STOP ; }#ifdef CONFIG_DEBUG_FS
@@ -182,10 +184,12 @@ msm_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr) NULL, };unsigned idx, unmapped =0 ; +unsigned long remaining =0 ;for (idx =0 ; lrus[idx] && unmapped < vmap_shrink_limit; idx++) { unmapped +=drm_gem_lru_scan (lrus[idx], vmap_shrink_limit - unmapped, + &remaining, vmap_shrink); }
include/drm/drm_gem.h +3 -1
@@ -475,7 +475,9 @@ int drm_gem_dumb_map_offset(struct drm_file *file, struct drm_device *dev,void drm_gem_lru_init (struct drm_gem_lru *lru,struct mutex *lock);void drm_gem_lru_remove (struct drm_gem_object *obj);void drm_gem_lru_move_tail (struct drm_gem_lru *lru,struct drm_gem_object *obj); -unsigned long drm_gem_lru_scan (struct drm_gem_lru *lru,unsigned nr_to_scan, +unsigned long drm_gem_lru_scan (struct drm_gem_lru *lru, +unsigned int nr_to_scan, +unsigned long *remaining,bool (*shrink)(struct drm_gem_object *obj));#endif /* __DRM_GEM_H__ */
Signature · 1 line
2.37.3
Dmitry Osipenko
maintainerPATCH v8 2/7 drm/shmem-helper: Don't use vmap_use_count for dma-bufs
In reply to Dmitry OsipenkoDMA-buf core has its own refcounting of vmaps, use it instead of drm-shmem counting. This change prepares drm-shmem for addition of memory shrinker support where drm-shmem will use a single dma-buf reservation lock for all operations performed over dma-bufs.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 35 +++++++++++++++----------- 1 file changed, 20 insertions(+), 15 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +20 -15
@@ -293,24 +293,22 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,struct drm_gem_object *obj = &shmem->base;int ret =0 ; -if (shmem->vmap_use_count++ >0 ) { -iosys_map_set_vaddr (map, shmem->vaddr); -return 0 ; - } -if (obj->import_attach) { ret =dma_buf_vmap (obj->import_attach->dmabuf, map);if (!ret) {if (WARN_ON (map->is_iomem)) {dma_buf_vunmap (obj->import_attach->dmabuf, map); - ret = -EIO ; - goto err_put_pages; +return -EIO ; } - shmem->vaddr = map->vaddr; } }else {pgprot_t prot =PAGE_KERNEL ; +if (shmem->vmap_use_count++ >0 ) { +iosys_map_set_vaddr (map, shmem->vaddr); +return 0 ; + } + ret =drm_gem_shmem_get_pages (shmem);if (ret) goto err_zero_use;
@@ -376,15 +374,15 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, {struct drm_gem_object *obj = &shmem->base; -if (WARN_ON_ONCE (!shmem->vmap_use_count)) -return ; - -if (--shmem->vmap_use_count >0 ) -return ; -if (obj->import_attach) {dma_buf_vunmap (obj->import_attach->dmabuf, map); }else { +if (WARN_ON_ONCE (!shmem->vmap_use_count)) +return ; + +if (--shmem->vmap_use_count >0 ) +return ; +vunmap (shmem->vaddr);drm_gem_shmem_put_pages (shmem); }
@@ -646,7 +644,14 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem, struct drm_printer *p, unsigned int indent) { drm_printf_indent(p, indent, "pages_use_count=%u\n", shmem->pages_use_count); -drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); + +if (shmem->base.import_attach) +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->base.dma_buf->vmapping_counter); +else +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->vmap_use_count); +drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }EXPORT_SYMBOL (drm_gem_shmem_print_info);
Signature · 1 line
2.37.3
Thomas Zimmermann
maintainerRe: [PATCH v8 2/7] drm/shmem-helper: Don't use vmap_use_count for dma-bufs
Hi,
I have a few comments that are not really something for this patch. I'll comment it here anyway to have them posted.
Am 06.11.22 um 00:27 schrieb Dmitry Osipenko:
Quoted 15 lines
DMA-buf core has its own refcounting of vmaps, use it instead of drm-shmem counting. This change prepares drm-shmem for addition of memory shrinker support where drm-shmem will use a single dma-buf reservation lock for all operations performed over dma-bufs.
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 35 +++++++++++++++----------- 1 file changed, 20 insertions(+), 15 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +0
-0
drivers/gpu/drm/drm_gem_shmem_helper.c static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, · line 296
Quoted 9 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +0
-5@@ -293,24 +293,22 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,struct drm_gem_object *obj = &shmem->base;int ret =0 ; -if (shmem->vmap_use_count++ >0 ) { -iosys_map_set_vaddr (map, shmem->vaddr); -return 0 ; - } -if (obj->import_attach) {
We have a number of such branches in various memory managers. At some point we should think about something like a GEM DMA-BUF object (or at least some helpers) that represents an imported buffer. Such a ting might be useful to avoid the duplication and branching that's going on here and in other functions.
drivers/gpu/drm/drm_gem_shmem_helper.c static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, · line 299
Quoted 3 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +0
-0ret =dma_buf_vmap (obj->import_attach->dmabuf, map);if (!ret) {if (WARN_ON (map->is_iomem)) {
I think it's OK to drop this test for is_iomem. It was useful when we did not distinguish between I/O and system memory in most of our helpers. But that should have been resolved by now and whatever code still depends on it should be updated accordingly. All helpers use iosys_map functionality to access the memory.
drivers/gpu/drm/drm_gem_shmem_helper.c static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, · line 309
Quoted 6 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-3dma_buf_vunmap (obj->import_attach->dmabuf, map); - ret = -EIO ; - goto err_put_pages; +return -EIO ; } - shmem->vaddr = map->vaddr;
To resolve the is_iomap issue, we'd have to store vaddr as struct iosysm_map value.
Quoted 37 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +11
-6} }else {pgprot_t prot =PAGE_KERNEL ; +if (shmem->vmap_use_count++ >0 ) { +iosys_map_set_vaddr (map, shmem->vaddr); +return 0 ; + } + ret =drm_gem_shmem_get_pages (shmem);if (ret) goto err_zero_use;@@ -376,15 +374,15 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, {struct drm_gem_object *obj = &shmem->base; -if (WARN_ON_ONCE (!shmem->vmap_use_count)) -return ; - -if (--shmem->vmap_use_count >0 ) -return ; -if (obj->import_attach) {dma_buf_vunmap (obj->import_attach->dmabuf, map); }else { +if (WARN_ON_ONCE (!shmem->vmap_use_count)) +return ; + +if (--shmem->vmap_use_count >0 ) +return ; +vunmap (shmem->vaddr);drm_gem_shmem_put_pages (shmem); }@@ -646,7 +644,14 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem, struct drm_printer *p, unsigned int indent) {
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem, · line 654
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +8
-1drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count); -drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); + +if (shmem->base.import_attach) +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->base.dma_buf->vmapping_counter); +else +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->vmap_use_count); +
Here's another case where a GEM DMA-BUF object might become helpful.
Best regards
Thomas
Quoted 3 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +0
-0drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }EXPORT_SYMBOL (drm_gem_shmem_print_info);
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
Dmitry Osipenko
maintainerPATCH v8 3/7 drm/shmem-helper: Switch to reservation lock
In reply to Dmitry OsipenkoReplace all drm-shmem locks with a GEM reservation lock. This makes locks consistent with dma-buf locking convention where importers are responsible for holding reservation lock for all operations performed over dma-bufs, preventing deadlock between dma-buf importers and exporters.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 184 +++++++----------- drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- include/drm/drm_gem_shmem_helper.h | 14 +- 6 files changed, 95 insertions(+), 143 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +71 -113
@@ -86,8 +86,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)if (ret) goto err_release; -mutex_init (&shmem->pages_lock); -mutex_init (&shmem->vmap_lock);INIT_LIST_HEAD (&shmem->madv_list);if (!private) {
@@ -139,11 +137,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -WARN_ON (shmem->vmap_use_count); -if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { +dma_resv_lock (shmem->base.resv,NULL ); + +WARN_ON (shmem->vmap_use_count); +if (shmem->sgt) {dma_unmap_sgtable (obj->dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );
@@ -152,18 +152,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }if (shmem->pages)drm_gem_shmem_put_pages (shmem); - } -WARN_ON (shmem->pages_use_count); +WARN_ON (shmem->pages_use_count); + +dma_resv_unlock (shmem->base.resv); + }drm_gem_object_release (obj); -mutex_destroy (&shmem->pages_lock); -mutex_destroy (&shmem->vmap_lock);kfree (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages;
@@ -194,35 +194,17 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) }/* -* drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object +* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object * -* This function makes sure that backing pages exists for the shmem GEM object -* and increases the use count. -* -* Returns: -* 0 on success or a negative error code on failure. +* This function decreases the use count and puts the backing pages when use drops to zero. */ -int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) -{ -int ret; - -WARN_ON (shmem->base.import_attach); - - ret =mutex_lock_interruptible (&shmem->pages_lock); -if (ret) -return ret; - ret =drm_gem_shmem_get_pages_locked (shmem); -mutex_unlock (&shmem->pages_lock); - -return ret; -} -EXPORT_SYMBOL (drm_gem_shmem_get_pages); - -static void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; +dma_resv_assert_held (shmem->base.resv); +if (WARN_ON_ONCE (!shmem->pages_use_count))return ;
@@ -239,19 +221,6 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } - -/* -* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object -* @shmem: shmem GEM object -* -* This function decreases the use count and puts the backing pages when use drops to zero. -*/ -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) -{ -mutex_lock (&shmem->pages_lock); -drm_gem_shmem_put_pages_locked (shmem); -mutex_unlock (&shmem->pages_lock); -}EXPORT_SYMBOL (drm_gem_shmem_put_pages);/**
@@ -266,6 +235,8 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +dma_resv_assert_held (shmem->base.resv); +WARN_ON (shmem->base.import_attach);return drm_gem_shmem_get_pages (shmem);
@@ -281,14 +252,31 @@ EXPORT_SYMBOL(drm_gem_shmem_pin);*/ void drm_gem_shmem_unpin (struct drm_gem_shmem_object *shmem) { +dma_resv_assert_held (shmem->base.resv); +WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); }EXPORT_SYMBOL (drm_gem_shmem_unpin); -static int drm_gem_shmem_vmap_locked (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +/* +* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object +* @shmem: shmem GEM object +* @map: Returns the kernel virtual address of the SHMEM GEM object's backing +* store. +* +* This function makes sure that a contiguous kernel virtual address mapping +* exists for the buffer backing the shmem GEM object. It hides the differences +* between dma-buf imported and natively allocated objects. +* +* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap(). +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;int ret =0 ;
@@ -304,6 +292,8 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, }else {pgprot_t prot =PAGE_KERNEL ; +dma_resv_assert_held (shmem->base.resv); +if (shmem->vmap_use_count++ >0 ) {iosys_map_set_vaddr (map, shmem->vaddr);return 0 ;
@@ -338,45 +328,30 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,return ret; } +EXPORT_SYMBOL (drm_gem_shmem_vmap);/* -* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object +* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object * @shmem: shmem GEM object -* @map: Returns the kernel virtual address of the SHMEM GEM object's backing -* store. -* -* This function makes sure that a contiguous kernel virtual address mapping -* exists for the buffer backing the shmem GEM object. It hides the differences -* between dma-buf imported and natively allocated objects. +* @map: Kernel virtual address where the SHMEM GEM object was mapped * -* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap(). +* This function cleans up a kernel virtual address mapping acquired by +* drm_gem_shmem_vmap(). The mapping is only removed when the use count drops to +* zero. * -* Returns: -* 0 on success or a negative error code on failure. +* This function hides the differences between dma-buf imported and natively +* allocated objects. */ -int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) -{ -int ret; - - ret =mutex_lock_interruptible (&shmem->vmap_lock); -if (ret) -return ret; - ret =drm_gem_shmem_vmap_locked (shmem, map); -mutex_unlock (&shmem->vmap_lock); - -return ret; -} -EXPORT_SYMBOL (drm_gem_shmem_vmap); - -static void drm_gem_shmem_vunmap_locked (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;if (obj->import_attach) {dma_buf_vunmap (obj->import_attach->dmabuf, map); }else { +dma_resv_assert_held (shmem->base.resv); +if (WARN_ON_ONCE (!shmem->vmap_use_count))return ;
@@ -389,26 +364,6 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, shmem->vaddr =NULL ; } - -/* -* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object -* @shmem: shmem GEM object -* @map: Kernel virtual address where the SHMEM GEM object was mapped -* -* This function cleans up a kernel virtual address mapping acquired by -* drm_gem_shmem_vmap(). The mapping is only removed when the use count drops to -* zero. -* -* This function hides the differences between dma-buf imported and natively -* allocated objects. -*/ -void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) -{ -mutex_lock (&shmem->vmap_lock); -drm_gem_shmem_vunmap_locked (shmem, map); -mutex_unlock (&shmem->vmap_lock); -}EXPORT_SYMBOL (drm_gem_shmem_vunmap); static struct drm_gem_shmem_object *
@@ -441,24 +396,24 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,*/ int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) { -mutex_lock (&shmem->pages_lock); +dma_resv_assert_held (shmem->base.resv);if (shmem->madv >=0 ) shmem->madv = madv; madv = shmem->madv; -mutex_unlock (&shmem->pages_lock); -return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev; +dma_resv_assert_held (shmem->base.resv); +WARN_ON (!drm_gem_shmem_is_purgeable (shmem));dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );
@@ -466,7 +421,7 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); shmem->sgt =NULL ; -drm_gem_shmem_put_pages_locked (shmem); +drm_gem_shmem_put_pages (shmem); shmem->madv =-1 ;
@@ -482,17 +437,6 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); } -EXPORT_SYMBOL (drm_gem_shmem_purge_locked); - -bool drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) -{ -if (!mutex_trylock (&shmem->pages_lock)) -return false; -drm_gem_shmem_purge_locked (shmem); -mutex_unlock (&shmem->pages_lock); - -return true; -}EXPORT_SYMBOL (drm_gem_shmem_purge);/**
@@ -548,7 +492,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; -mutex_lock (&shmem->pages_lock); +dma_resv_lock (shmem->base.resv,NULL ); if (page_offset >= num_pages || WARN_ON_ONCE(!shmem->pages) ||
@@ -560,7 +504,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf) ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } -mutex_unlock (&shmem->pages_lock); +dma_resv_unlock (shmem->base.resv);return ret; }
@@ -573,8 +517,10 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)WARN_ON (shmem->base.import_attach); +dma_resv_lock (shmem->base.resv,NULL ); ret =drm_gem_shmem_get_pages (shmem);WARN_ON_ONCE (ret !=0 ); +dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma); }
@@ -584,7 +530,10 @@ static void drm_gem_shmem_vm_close(struct vm_area_struct *vma)struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +dma_resv_lock (shmem->base.resv,NULL );drm_gem_shmem_put_pages (shmem); +dma_resv_unlock (shmem->base.resv); +drm_gem_vm_close (vma); }
@@ -619,7 +568,10 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structreturn dma_buf_mmap (obj->dma_buf, vma,0 ); } +dma_resv_lock (shmem->base.resv,NULL ); ret =drm_gem_shmem_get_pages (shmem); +dma_resv_unlock (shmem->base.resv); +if (ret) {drm_gem_vm_close (vma);return ret;
@@ -707,9 +659,11 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)WARN_ON (obj->import_attach); +dma_resv_lock (shmem->base.resv,NULL ); + ret =drm_gem_shmem_get_pages (shmem);if (ret) -return ERR_PTR (ret); + goto err_unlock; sgt =drm_gem_shmem_get_sg_table (shmem);if (IS_ERR (sgt)) {
@@ -723,6 +677,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +dma_resv_unlock (shmem->base.resv); +return sgt; err_free_sgt:
@@ -730,6 +686,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)kfree (sgt); err_put_pages:drm_gem_shmem_put_pages (shmem); +err_unlock: +dma_resv_unlock (shmem->base.resv);return ERR_PTR (ret); }EXPORT_SYMBOL_GPL (drm_gem_shmem_get_pages_sgt);
drivers/gpu/drm/lima/lima_gem.c +4 -4
@@ -34,7 +34,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) new_size =min (new_size, bo->base.base.size); -mutex_lock (&bo->base.pages_lock); +dma_resv_lock (bo->base.base.resv,NULL );if (bo->base.pages) { pages = bo->base.pages;
@@ -42,7 +42,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) pages =kvmalloc_array (bo->base.base.size >>PAGE_SHIFT ,sizeof (*pages),GFP_KERNEL | __GFP_ZERO);if (!pages) { -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv);return -ENOMEM ; }
@@ -56,13 +56,13 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)struct page *page =shmem_read_mapping_page (mapping, i);if (IS_ERR (page)) { -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv);return PTR_ERR (page); } pages[i] = page; } -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv); ret =sg_alloc_table_from_pages (&sgt, pages, i,0 , new_size,GFP_KERNEL );
drivers/gpu/drm/panfrost/panfrost_drv.c +6 -1
@@ -405,6 +405,10 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, bo =to_panfrost_bo (gem_obj); + ret =dma_resv_lock_interruptible (bo->base.base.resv,NULL ); +if (ret) + goto out_put_object; +mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {
@@ -442,7 +446,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, out_unlock_mappings:mutex_unlock (&bo->mappings.lock);mutex_unlock (&pfdev->shrinker_lock); - +dma_resv_unlock (bo->base.base.resv); +out_put_object:drm_gem_object_put (gem_obj);return ret; }
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +3 -3
@@ -48,14 +48,14 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)if (!mutex_trylock (&bo->mappings.lock))return false; -if (!mutex_trylock (&shmem->pages_lock)) +if (!dma_resv_trylock (shmem->base.resv)) goto unlock_mappings;panfrost_gem_teardown_mappings_locked (bo); -drm_gem_shmem_purge_locked (&bo->base); +drm_gem_shmem_purge (&bo->base); ret = true; -mutex_unlock (&shmem->pages_lock); +dma_resv_unlock (shmem->base.resv); unlock_mappings:mutex_unlock (&bo->mappings.lock);
drivers/gpu/drm/panfrost/panfrost_mmu.c +10 -9
@@ -434,6 +434,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,struct panfrost_gem_mapping *bomapping;struct panfrost_gem_object *bo;struct address_space *mapping; +struct drm_gem_object *obj;pgoff_t page_offset;struct sg_table *sgt;struct page **pages;
@@ -456,15 +457,16 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, page_offset = addr >>PAGE_SHIFT ; page_offset -= bomapping->mmnode.start; -mutex_lock (&bo->base.pages_lock); + obj = &bo->base.base; + +dma_resv_lock (obj->resv,NULL );if (!bo->base.pages) { bo->sgts =kvmalloc_array (bo->base.base.size /SZ_2M ,sizeof (struct sg_table ),GFP_KERNEL | __GFP_ZERO);if (!bo->sgts) { -mutex_unlock (&bo->base.pages_lock); ret = -ENOMEM ; - goto err_bo; + goto err_unlock; } pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
@@ -472,9 +474,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,if (!pages) {kvfree (bo->sgts); bo->sgts =NULL ; -mutex_unlock (&bo->base.pages_lock); ret = -ENOMEM ; - goto err_bo; + goto err_unlock; } bo->base.pages = pages; bo->base.pages_use_count =1 ;
@@ -482,7 +483,6 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, pages = bo->base.pages;if (pages[page_offset]) {/* Pages are already mapped, bail out. */ -mutex_unlock (&bo->base.pages_lock); goto out; } }
@@ -493,14 +493,11 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,for (i = page_offset; i < page_offset +NUM_FAULT_PAGES ; i++) { pages[i] =shmem_read_mapping_page (mapping, i);if (IS_ERR (pages[i])) { -mutex_unlock (&bo->base.pages_lock); ret =PTR_ERR (pages[i]); goto err_pages; } } -mutex_unlock (&bo->base.pages_lock); - sgt = &bo->sgts[page_offset / (SZ_2M /PAGE_SIZE )]; ret =sg_alloc_table_from_pages (sgt, pages + page_offset,NUM_FAULT_PAGES ,0 ,SZ_2M ,GFP_KERNEL );
@@ -519,6 +516,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,dev_dbg (pfdev->dev,"mapped page fault @ AS%d %llx" , as, addr); out: +dma_resv_unlock (obj->resv); +panfrost_gem_mapping_put (bomapping);return 0 ;
@@ -527,6 +526,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,sg_free_table (sgt); err_pages:drm_gem_shmem_put_pages (&bo->base); +err_unlock: +dma_resv_unlock (obj->resv); err_bo:panfrost_gem_mapping_put (bomapping);return ret;
include/drm/drm_gem_shmem_helper.h +1 -13
@@ -26,11 +26,6 @@ struct drm_gem_shmem_object {*/ struct drm_gem_object base; -/** -* @pages_lock: Protects the page table and use count -*/ -struct mutex pages_lock; -/** * @pages: Page table */
@@ -79,11 +74,6 @@ struct drm_gem_shmem_object {*/ struct sg_table *sgt; -/** -* @vmap_lock: Protects the vmap address and use count -*/ -struct mutex vmap_lock; -/** * @vaddr: Kernel virtual address of the backing memory */
@@ -109,7 +99,6 @@ struct drm_gem_shmem_object {struct drm_gem_shmem_object *drm_gem_shmem_create (struct drm_device *dev,size_t size);void drm_gem_shmem_free (struct drm_gem_shmem_object *shmem); -int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem);int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_unpin (struct drm_gem_shmem_object *shmem);
@@ -128,8 +117,7 @@ static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem !shmem->base.dma_buf && !shmem->base.import_attach; } -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem); -bool drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);
Signature · 1 line
2.37.3
Dmitry Osipenko
maintainerPATCH v8 4/7 drm/shmem-helper: Add memory shrinker
In reply to Dmitry OsipenkoIntroduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement evict() callback of shmem object where driver should check whether object is purgeable or evictable and perform shrinking action
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device)
3. Use drm_gem_shmem_set_purgeable(shmem) and alike API functions to activate shrinking of shmem GEMs
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 493 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 +++- 4 files changed, 539 insertions(+), 54 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +445 -48
@@ -126,6 +126,54 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static void drm_gem_shmem_resv_assert_held (struct drm_gem_shmem_object *shmem) +{ +/* +* Destroying the object is a special case.. drm_gem_shmem_free() +* calls many things that WARN_ON if the obj lock is not held. But +* acquiring the obj lock in drm_gem_shmem_free() can cause a locking +* order inversion between reservation_ww_class_mutex and fs_reclaim. +* +* This deadlock is not actually possible, because no one should +* be already holding the lock when msm_gem_free_object() is called. +* Unfortunately lockdep is not aware of this detail. So when the +* refcount drops to zero, we pretend it is already locked. +*/ +if (kref_read (&shmem->base.refcount)) +dma_resv_assert_held (shmem->base.resv); +} + +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +drm_gem_shmem_resv_assert_held (shmem); + +if (!gem_shrinker || obj->import_attach) +return ; + +if (drm_gem_shmem_is_evictable (shmem) ||drm_gem_shmem_is_purgeable (shmem)) +drm_gem_lru_move_tail (&gem_shrinker->lru_evictable, &shmem->base); +else if (shmem->madv <0 ) +drm_gem_lru_remove (&shmem->base); +else if (shmem->evicted) +drm_gem_lru_move_tail (&gem_shrinker->lru_evicted, &shmem->base); +else if (!shmem->pages) +drm_gem_lru_remove (&shmem->base); +else +drm_gem_lru_move_tail (&gem_shrinker->lru_pinned, &shmem->base); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free
@@ -140,7 +188,8 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { -dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 );WARN_ON (shmem->vmap_use_count);
@@ -150,12 +199,10 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count); - -dma_resv_unlock (shmem->base.resv); }drm_gem_object_release (obj);
@@ -163,18 +210,86 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) { +dma_resv_unlock (shmem->base.resv); +return -ENOMEM ; + } + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) { +dma_resv_unlock (shmem->base.resv); +return -ENOMEM ; + } + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }
@@ -193,23 +308,46 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } -/* -* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object -* @shmem: shmem GEM object -* -* This function decreases the use count and puts the backing pages when use drops to zero. -*/ -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; +int err;dma_resv_assert_held (shmem->base.resv); -if (WARN_ON_ONCE (!shmem->pages_use_count)) -return ; +if (shmem->madv <0 ) +return -ENOMEM ; -if (--shmem->pages_use_count >0 ) +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 );return ; + }#ifdef CONFIG_X86 if (shmem->map_wc)
@@ -221,6 +359,27 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } + +/* +* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object +* @shmem: shmem GEM object +* +* This function decreases the use count and puts the backing pages when use drops to zero. +*/ +void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +{ +drm_gem_shmem_resv_assert_held (shmem); + +if (WARN_ON_ONCE (!shmem->pages_use_count)) +return ; + +if (--shmem->pages_use_count >0 ) +return ; + +drm_gem_shmem_release_pages (shmem); + +drm_gem_shmem_update_pages_state (shmem); +}EXPORT_SYMBOL (drm_gem_shmem_put_pages);/**
@@ -235,11 +394,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);
@@ -257,6 +422,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);
@@ -299,7 +466,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;
@@ -322,7 +489,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;
@@ -359,7 +526,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;
@@ -396,48 +563,84 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,*/ int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) { -dma_resv_assert_held (shmem->base.resv); +drm_gem_shmem_resv_assert_held (shmem);if (shmem->madv >=0 ) shmem->madv = madv; madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
@@ -488,22 +691,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;
@@ -513,13 +727,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);
@@ -595,6 +811,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)
@@ -604,7 +822,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);
@@ -677,6 +897,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;
@@ -727,6 +949,181 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +unsigned long count = gem_shrinker->lru_evictable.count; + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static bool drm_gem_shmem_shrinker_evict (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < obj->size >>PAGE_SHIFT ) +return false; + + err = shmem->evict (shmem); +if (err) +return false; + +return true; +} + +static bool drm_gem_shmem_shrinker_purge (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_purgeable (shmem)) +return false; + + err = shmem->evict (shmem); +if (err) +return false; + +return true; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +unsigned long nr_to_scan = sc->nr_to_scan; +unsigned long remaining =0 ; +unsigned long freed =0 ; + +/* purge as many objects as we can */ + freed +=drm_gem_lru_scan (&gem_shrinker->lru_evictable, + nr_to_scan, &remaining, + drm_gem_shmem_shrinker_purge); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_lru_scan (&gem_shrinker->lru_evictable, + nr_to_scan - freed, &remaining, + drm_gem_shmem_shrinker_evict); + +return (freed >0 && remaining >0 ) ? freed :SHRINK_STOP ; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev,const char *name) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +mutex_init (&gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_evictable, &gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_evicted, &gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_pinned, &gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base, name); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable.list)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted.list)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned.list)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8 -1
@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {
@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }
include/drm/drm_device.h +4 -0
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;
@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: *
include/drm/drm_gem_shmem_helper.h +82 -5
132 lines not on this page · raw message
Signature · 1 line
2.37.3
Thomas Zimmermann
maintainerRe: [PATCH v8 4/7] drm/shmem-helper: Add memory shrinker
Hi
Am 06.11.22 um 00:27 schrieb Dmitry Osipenko:
Quoted 7 lines
Introduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement
evict()callback of shmem object where driver should check whether object is purgeable or evictable and perform shrinking action
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device)
I've long wanted to add an init call for GEM SHMEM helpers; mostly for shrinking. So this could be called drmm_gem_shmem_init() and do all kinds of initialization. Cleanup should be managed. It can still be optional, so you don't have to go through all the SHMEM drivers.
It should also take the something like
struct drm_gem_shmem_funcs {
evict()
};
that contains the evict callback, as there's no point in having a per-object evict callback. The struct can then be declared static const within the driver.
Alternatively, and that might be preferable in the long term, the evict hook could go into struct drm_gem_object_funcs and we could converge the various memory managers onto the interface.
Quoted 2 lines
3. Use
drm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs
The shrinking code will allow us to aviod changes to the page tables during vunmap. That is important for many driver's performance. So the long-term goal should be to active all this automatically and filter out the cases where eviction/purging is not wanted.
Quoted 105 lines
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 493 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 +++- 4 files changed, 539 insertions(+), 54 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +58
-5@@ -126,6 +126,54 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static void drm_gem_shmem_resv_assert_held (struct drm_gem_shmem_object *shmem) +{ +/* +* Destroying the object is a special case.. drm_gem_shmem_free() +* calls many things that WARN_ON if the obj lock is not held. But +* acquiring the obj lock in drm_gem_shmem_free() can cause a locking +* order inversion between reservation_ww_class_mutex and fs_reclaim. +* +* This deadlock is not actually possible, because no one should +* be already holding the lock when msm_gem_free_object() is called. +* Unfortunately lockdep is not aware of this detail. So when the +* refcount drops to zero, we pretend it is already locked. +*/ +if (kref_read (&shmem->base.refcount)) +dma_resv_assert_held (shmem->base.resv); +} + +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +drm_gem_shmem_resv_assert_held (shmem); + +if (!gem_shrinker || obj->import_attach) +return ; + +if (drm_gem_shmem_is_evictable (shmem) ||drm_gem_shmem_is_purgeable (shmem)) +drm_gem_lru_move_tail (&gem_shrinker->lru_evictable, &shmem->base); +else if (shmem->madv <0 ) +drm_gem_lru_remove (&shmem->base); +else if (shmem->evicted) +drm_gem_lru_move_tail (&gem_shrinker->lru_evicted, &shmem->base); +else if (!shmem->pages) +drm_gem_lru_remove (&shmem->base); +else +drm_gem_lru_move_tail (&gem_shrinker->lru_pinned, &shmem->base); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -140,7 +188,8 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { -dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 );WARN_ON (shmem->vmap_use_count);@@ -150,12 +199,10 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count); - -dma_resv_unlock (shmem->base.resv); }drm_gem_object_release (obj);@@ -163,18 +210,86 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +*
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 229
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) { +dma_resv_unlock (shmem->base.resv); +return -ENOMEM ;
ENOMEM is not right here. It's for failed memory allocation. ENODEV seems more appropriate.
But why do we need an error here anyway? Why not just fail transparently?
Quoted 17 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +17
-0+ } + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +*
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 256
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) { +dma_resv_unlock (shmem->base.resv); +return -ENOMEM ;
Same as above.
Quoted 13 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +13
-0+ } + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 278
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-1+drm_gem_shmem_acquire_pages(struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages);
drm_WARN_ON(), here and everywhere else. The file is doesn't use the new drm logging much, but it's time to switch.
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +4
-0+return -ENOMEM ; + } + +if (shmem->pages) {
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 292
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-0+WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages));
drm_dbg_kms() here and everywhere else.
Quoted 556 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +323
-42- shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,23 +308,46 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } -/* -* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object -* @shmem: shmem GEM object -* -* This function decreases the use count and puts the backing pages when use drops to zero. -*/ -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; +int err;dma_resv_assert_held (shmem->base.resv); -if (WARN_ON_ONCE (!shmem->pages_use_count)) -return ; +if (shmem->madv <0 ) +return -ENOMEM ; -if (--shmem->pages_use_count >0 ) +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 );return ; + }#ifdef CONFIG_X86 if (shmem->map_wc)@@ -221,6 +359,27 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } + +/* +* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object +* @shmem: shmem GEM object +* +* This function decreases the use count and puts the backing pages when use drops to zero. +*/ +void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +{ +drm_gem_shmem_resv_assert_held (shmem); + +if (WARN_ON_ONCE (!shmem->pages_use_count)) +return ; + +if (--shmem->pages_use_count >0 ) +return ; + +drm_gem_shmem_release_pages (shmem); + +drm_gem_shmem_update_pages_state (shmem); +}EXPORT_SYMBOL (drm_gem_shmem_put_pages);/** @@ -235,11 +394,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +422,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +466,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +489,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +526,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -396,48 +563,84 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,*/ int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) { -dma_resv_assert_held (shmem->base.resv); +drm_gem_shmem_resv_assert_held (shmem);if (shmem->madv >=0 ) shmem->madv = madv; madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +691,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +727,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -595,6 +811,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -604,7 +822,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -677,6 +897,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -727,6 +949,181 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +unsigned long count = gem_shrinker->lru_evictable.count; + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static bool drm_gem_shmem_shrinker_evict (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < obj->size >>PAGE_SHIFT ) +return false; + + err = shmem->evict (shmem); +if (err) +return false; + +return true; +} + +static bool drm_gem_shmem_shrinker_purge (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_purgeable (shmem)) +return false; + + err = shmem->evict (shmem); +if (err) +return false; + +return true; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +unsigned long nr_to_scan = sc->nr_to_scan; +unsigned long remaining =0 ; +unsigned long freed =0 ; + +/* purge as many objects as we can */ + freed +=drm_gem_lru_scan (&gem_shrinker->lru_evictable, + nr_to_scan, &remaining, + drm_gem_shmem_shrinker_purge); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_lru_scan (&gem_shrinker->lru_evictable, + nr_to_scan - freed, &remaining, + drm_gem_shmem_shrinker_evict); + +return (freed >0 && remaining >0 ) ? freed :SHRINK_STOP ; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev,const char *name) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +mutex_init (&gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_evictable, &gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_evicted, &gem_shrinker->lock); +drm_gem_lru_init (&gem_shrinker->lru_pinned, &gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base, name); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable.list)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted.list)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned.list)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }include/drm/drm_device.h +1
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;
include/drm/drm_device.h struct drm_device { · line 283
Quoted 6 lines
include/drm/drm_device.h +3
-0@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +
As mentioned above, that could be GEM SHMEM state instead of just a shrinker. You can put this into a union with vram_mm. No driver will use both memory managers at the same time.
Quoted 62 lines
include/drm/drm_device.h +0
-0/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +24
-1@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages allowed to be evicted by memory shrinker +* only when the count is zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +*
include/drm/drm_gem_shmem_helper.h struct drm_gem_shmem_object { · line 126
Quoted 10 lines
include/drm/drm_gem_shmem_helper.h +10
-0+* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted;
Maybe turn those booleans, including map_wc, into 1-bit bit fields.
Best regards
Thomas
Quoted 73 lines
include/drm/drm_gem_shmem_helper.h +48
-4+ +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. Drivers using +* drm-shmem shrinker must implement this callback. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct drm_gem_lru lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct drm_gem_lru lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct drm_gem_lru lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev,const char *name); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
Dmitry Osipenko
maintainerRe: [PATCH v8 4/7] drm/shmem-helper: Add memory shrinker
Hello Thomas,
Quoted 10 lines · Thomas Zimmermann
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 229
Quoted 7 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +7
-0+int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ + dma_resv_lock(shmem->base.resv, NULL); + + if (shmem->madv < 0) { + dma_resv_unlock(shmem->base.resv); + return -ENOMEM;ENOMEM is not right here. It's for failed memory allocation. ENODEV seems more appropriate.
Had the same thought about ENOMEM and at one point was considering ENOENT, but in the end decided it's not much better than ENOMEM.
Quoted 1 line
But why do we need an error here anyway? Why not just fail transparently?
I added the error handling everywhere for consistency. Perhaps indeed will be better to fail transparently for now since nobody cares about such errors and likely won't in the future.
The rest of the comments are also good to me, will start preparing the v9. Thank you for the review!
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerPATCH v8 5/7 drm/gem: Add drm_gem_pin_unlocked()
In reply to Dmitry OsipenkoAdd unlocked variants of drm_gem_un/pin() functions. These new helpers will take care of GEM dma-reservation locking for DRM drivers.
VirtIO-GPU driver will use these helpers to pin shmem framebuffers, preventing them from eviction during scanout.
--- drivers/gpu/drm/drm_gem.c | 29 +++++++++++++++++++++++++++++ include/drm/drm_gem.h | 3 +++ 2 files changed, 32 insertions(+)
drivers/gpu/drm/drm_gem.c +29 -0
@@ -1154,6 +1154,35 @@ void drm_gem_unpin(struct drm_gem_object *obj) obj->funcs->unpin (obj); } +int drm_gem_pin_unlocked (struct drm_gem_object *obj) +{ +int ret; + +if (!obj->funcs->pin) +return 0 ; + + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret) +return ret; + + ret = obj->funcs->pin (obj); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL (drm_gem_pin_unlocked); + +void drm_gem_unpin_unlocked (struct drm_gem_object *obj) +{ +if (!obj->funcs->unpin) +return ; + +dma_resv_lock (obj->resv,NULL ); + obj->funcs->unpin (obj); +dma_resv_unlock (obj->resv); +} +EXPORT_SYMBOL (drm_gem_unpin_unlocked); +int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map) {int ret;
include/drm/drm_gem.h +3 -0
@@ -480,4 +480,7 @@ unsigned long drm_gem_lru_scan(struct drm_gem_lru *lru,unsigned long *remaining, bool (*shrink)(struct drm_gem_object *obj)); +int drm_gem_pin_unlocked (struct drm_gem_object *obj); +void drm_gem_unpin_unlocked (struct drm_gem_object *obj); +#endif /* __DRM_GEM_H__ */
Signature · 1 line
2.37.3
Dmitry Osipenko
maintainerPATCH v8 6/7 drm/virtio: Support memory shrinking
In reply to Dmitry OsipenkoSupport generic drm-shmem memory shrinker and add new madvise IOCTL to the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as "don't need" using the new IOCTL to let shrinker purge the marked BOs on OOM, the shrinker will also evict unpurgeable shmem BOs from memory if guest supports SWAP file or partition.
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 18 +++- drivers/gpu/drm/virtio/virtgpu_gem.c | 55 ++++++++++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 +++++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 9 ++ drivers/gpu/drm/virtio/virtgpu_object.c | 138 +++++++++++++++++++----- drivers/gpu/drm/virtio/virtgpu_plane.c | 22 +++- drivers/gpu/drm/virtio/virtgpu_vq.c | 40 +++++++ include/uapi/drm/virtgpu_drm.h | 14 +++ 8 files changed, 303 insertions(+), 30 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +17 -1
@@ -90,6 +90,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;
@@ -275,7 +276,7 @@ struct virtio_gpu_fpriv { };/* virtgpu_ioctl.c */ -#define DRM_VIRTIO_NUM_IOCTLS 12 +#define DRM_VIRTIO_NUM_IOCTLS 13extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ];void virtio_gpu_create_context (struct drm_device *dev,struct drm_file *file);
@@ -311,6 +312,10 @@ void virtio_gpu_array_put_free(struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_delayed (struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_work (struct work_struct *work); +int virtio_gpu_array_prepare (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object_array *objs); +int virtio_gpu_gem_host_mem_release (struct virtio_gpu_object *bo); +int virtio_gpu_gem_madvise (struct virtio_gpu_object *obj,int madv);/* virtgpu_vq.c */ int virtio_gpu_alloc_vbufs (struct virtio_gpu_device *vgdev);
@@ -322,6 +327,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev, struct virtio_gpu_fence *fence);void virtio_gpu_cmd_unref_resource (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo); +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo); void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev, uint64_t offset, uint32_t width, uint32_t height,
@@ -342,6 +349,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,struct virtio_gpu_object *obj, struct virtio_gpu_mem_entry *ents, unsigned int nents); +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence);int virtio_gpu_attach_status_page (struct virtio_gpu_device *vgdev);int virtio_gpu_detach_status_page (struct virtio_gpu_device *vgdev); void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -454,6 +464,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */
@@ -484,4 +496,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,struct sg_table *sgt, enum dma_data_direction dir); +/* virtgpu_gem_shrinker.c */ +int virtio_gpu_gem_shrinker_init (struct virtio_gpu_device *vgdev); +void virtio_gpu_gem_shrinker_fini (struct virtio_gpu_device *vgdev); +#endif
drivers/gpu/drm/virtio/virtgpu_gem.c +55 -0
58 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_ioctl.c +37 -0
@@ -217,6 +217,10 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data, ret =virtio_gpu_array_lock_resv (buflist);if (ret) goto out_memdup; + + ret =virtio_gpu_array_prepare (vgdev, buflist); +if (ret) + goto out_unresv; } out_fence =virtio_gpu_fence_alloc (vgdev, fence_ctx, ring_idx);
@@ -423,6 +427,10 @@ static int virtio_gpu_transfer_from_host_ioctl(struct drm_device *dev,if (ret !=0 ) goto err_put_free; + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 );if (!fence) { ret = -ENOMEM ;
@@ -482,6 +490,10 @@ static int virtio_gpu_transfer_to_host_ioctl(struct drm_device *dev, void *data,if (ret !=0 ) goto err_put_free; + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + ret = -ENOMEM ; fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 );
@@ -838,6 +850,28 @@ static int virtio_gpu_context_init_ioctl(struct drm_device *dev,return ret; } +static int virtio_gpu_madvise_ioctl (struct drm_device *dev, +void *data, +struct drm_file *file) +{ +struct drm_virtgpu_madvise *args = data; +struct virtio_gpu_object *bo; +struct drm_gem_object *obj; + +if (args->madv >VIRTGPU_MADV_DONTNEED ) +return -EOPNOTSUPP ; + + obj =drm_gem_object_lookup (file, args->bo_handle); +if (!obj) +return -ENOENT ; + + bo =gem_to_virtio_gpu_obj (obj); + args->retained =virtio_gpu_gem_madvise (bo, args->madv); +drm_gem_object_put (obj); + +return 0 ; +} +struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ] = {DRM_IOCTL_DEF_DRV (VIRTGPU_MAP , virtio_gpu_map_ioctl,DRM_RENDER_ALLOW ),
@@ -877,4 +911,7 @@ struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS] = {DRM_IOCTL_DEF_DRV (VIRTGPU_CONTEXT_INIT , virtio_gpu_context_init_ioctl,DRM_RENDER_ALLOW ), + +DRM_IOCTL_DEF_DRV (VIRTGPU_MADVISE , virtio_gpu_madvise_ioctl, + DRM_RENDER_ALLOW), };
drivers/gpu/drm/virtio/virtgpu_kms.c +9 -0
@@ -240,6 +240,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) goto err_scanouts; } + ret =drm_gem_shmem_shrinker_register (dev,"virtio-gpu-shrinker" ); +if (ret) { +DRM_ERROR ("shrinker init failed\n" ); + goto err_modeset; + } +virtio_device_ready (vgdev->vdev);if (num_capsets)
@@ -252,6 +258,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) 5 * HZ);return 0 ; +err_modeset: +virtio_gpu_modeset_fini (vgdev); err_scanouts:virtio_gpu_free_vbufs (vgdev); err_vbufs:
@@ -290,6 +298,7 @@ void virtio_gpu_release(struct drm_device *dev)if (!vgdev)return ; +drm_gem_shmem_shrinker_unregister (dev);virtio_gpu_modeset_fini (vgdev);virtio_gpu_free_vbufs (vgdev);virtio_gpu_cleanup_cap_cache (vgdev);
drivers/gpu/drm/virtio/virtgpu_object.c +111 -27
172 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_plane.c +20 -2
@@ -238,20 +238,32 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,struct virtio_gpu_device *vgdev = dev->dev_private;struct virtio_gpu_framebuffer *vgfb;struct virtio_gpu_object *bo; +int err;if (!new_state->fb)return 0 ; vgfb =to_virtio_gpu_framebuffer (new_state->fb); bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); -if (!bo || (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)) + +if (virtio_gpu_is_shmem (bo)) { + err =drm_gem_pin_unlocked (&bo->base.base); +if (err) +return err; + } + +if (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)return 0 ;if (bo->dumb && (plane->state->fb != new_state->fb)) { vgfb->fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 ); -if (!vgfb->fence) +if (!vgfb->fence) { +if (virtio_gpu_is_shmem (bo)) +drm_gem_unpin_unlocked (&bo->base.base); +return -ENOMEM ; + } }return 0 ;
@@ -261,15 +273,21 @@ static void virtio_gpu_plane_cleanup_fb(struct drm_plane *plane, struct drm_plane_state *state) {struct virtio_gpu_framebuffer *vgfb; +struct virtio_gpu_object *bo;if (!state->fb)return ; vgfb =to_virtio_gpu_framebuffer (state->fb); + bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); +if (vgfb->fence) {dma_fence_put (&vgfb->fence->f); vgfb->fence =NULL ; } + +if (virtio_gpu_is_shmem (bo)) +drm_gem_unpin_unlocked (&bo->base.base); } static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
drivers/gpu/drm/virtio/virtgpu_vq.c +40 -0
@@ -538,6 +538,21 @@ void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,virtio_gpu_cleanup_object (bo); } +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo) +{ +struct virtio_gpu_resource_unref *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_UNREF ); + cmd_p->resource_id =cpu_to_le32 (bo->hw_res_handle); + +return virtio_gpu_queue_ctrl_buffer (vgdev, vbuf); +} + void virtio_gpu_cmd_set_scanout(struct virtio_gpu_device *vgdev, uint32_t scanout_id, uint32_t resource_id, uint32_t width, uint32_t height,
@@ -638,6 +653,23 @@ virtio_gpu_cmd_resource_attach_backing(struct virtio_gpu_device *vgdev,virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); } +static void +virtio_gpu_cmd_resource_detach_backing (struct virtio_gpu_device *vgdev, +u32 resource_id, +struct virtio_gpu_fence *fence) +{ +struct virtio_gpu_resource_attach_backing *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING ); + cmd_p->resource_id =cpu_to_le32 (resource_id); + +virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); +} +static void virtio_gpu_cmd_get_display_info_cb (struct virtio_gpu_device *vgdev,struct virtio_gpu_vbuffer *vbuf) {
@@ -1101,6 +1133,14 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev, ents, nents, NULL); } +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence) +{ +virtio_gpu_cmd_resource_detach_backing (vgdev, obj->hw_res_handle, + fence); +} +void virtio_gpu_cursor_ping (struct virtio_gpu_device *vgdev,struct virtio_gpu_output *output) {
include/uapi/drm/virtgpu_drm.h +14 -0
@@ -48,6 +48,7 @@ extern "C" {#define DRM_VIRTGPU_GET_CAPS 0x09#define DRM_VIRTGPU_RESOURCE_CREATE_BLOB 0x0a#define DRM_VIRTGPU_CONTEXT_INIT 0x0b +#define DRM_VIRTGPU_MADVISE 0x0c#define VIRTGPU_EXECBUF_FENCE_FD_IN 0x01#define VIRTGPU_EXECBUF_FENCE_FD_OUT 0x02
@@ -196,6 +197,15 @@ struct drm_virtgpu_context_init {__u64 ctx_set_params; }; +#define VIRTGPU_MADV_WILLNEED 0 +#define VIRTGPU_MADV_DONTNEED 1 +struct drm_virtgpu_madvise { +__u32 bo_handle; +__u32 retained;/* out, non-zero if BO can be used */ +__u32 madv; +__u32 pad; +}; +/* * Event code that's given when VIRTGPU_CONTEXT_PARAM_POLL_RINGS_MASK is in * effect. The event size is sizeof(drm_event), since there is no additional
@@ -246,6 +256,10 @@ struct drm_virtgpu_context_init {DRM_IOWR (DRM_COMMAND_BASE +DRM_VIRTGPU_CONTEXT_INIT , \ struct drm_virtgpu_context_init) +#define DRM_IOCTL_VIRTGPU_MADVISE \ + DRM_IOWR(DRM_COMMAND_BASE + DRM_VIRTGPU_MADVISE, \ + struct drm_virtgpu_madvise) +#if defined(__cplusplus) }#endif
Signature · 1 line
2.37.3
Dmitry Osipenko
maintainerPATCH v8 7/7 drm/panfrost: Switch to generic memory shrinker
In reply to Dmitry OsipenkoReplace Panfrost's custom memory shrinker with a common drm-shmem memory shrinker.
--- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 19 +-- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +++-- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 -- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 ------------------ drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++- 7 files changed, 42 insertions(+), 171 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
drivers/gpu/drm/panfrost/Makefile +0 -1
@@ -5,7 +5,6 @@ panfrost-y := \ panfrost_device.o \ panfrost_devfreq.o \ panfrost_gem.o \ - panfrost_gem_shrinker.o \ panfrost_gpu.o \ panfrost_job.o \ panfrost_mmu.o \
drivers/gpu/drm/panfrost/panfrost_device.h +0 -4
@@ -115,10 +115,6 @@ struct panfrost_device {atomic_t pending; } reset; -struct mutex shrinker_lock; -struct list_head shrinker_list; -struct shrinker shrinker; -struct panfrost_devfreq pfdevfreq; };
drivers/gpu/drm/panfrost/panfrost_drv.c +2 -17
@@ -160,7 +160,6 @@ panfrost_lookup_bos(struct drm_device *dev,break ; } -atomic_inc (&bo->gpu_usecount); job->mappings[i] = mapping; }
@@ -392,7 +391,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, {struct panfrost_file_priv *priv = file_priv->driver_priv;struct drm_panfrost_madvise *args = data; -struct panfrost_device *pfdev = dev->dev_private;struct drm_gem_object *gem_obj;struct panfrost_gem_object *bo;int ret =0 ;
@@ -409,7 +407,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (ret) goto out_put_object; -mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {struct panfrost_gem_mapping *first;
@@ -435,17 +432,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, args->retained =drm_gem_shmem_madvise (&bo->base, args->madv); -if (args->retained) { -if (args->madv ==PANFROST_MADV_DONTNEED ) -list_move_tail (&bo->base.madv_list, - &pfdev->shrinker_list); -else if (args->madv ==PANFROST_MADV_WILLNEED ) -list_del_init (&bo->base.madv_list); - } - out_unlock_mappings:mutex_unlock (&bo->mappings.lock); -mutex_unlock (&pfdev->shrinker_lock);dma_resv_unlock (bo->base.base.resv); out_put_object:drm_gem_object_put (gem_obj);
@@ -577,9 +565,6 @@ static int panfrost_probe(struct platform_device *pdev) ddev->dev_private = pfdev; pfdev->ddev = ddev; -mutex_init (&pfdev->shrinker_lock); -INIT_LIST_HEAD (&pfdev->shrinker_list); - err =panfrost_device_init (pfdev);if (err) {if (err != -EPROBE_DEFER )
@@ -601,7 +586,7 @@ static int panfrost_probe(struct platform_device *pdev)if (err <0 ) goto err_out1; -panfrost_gem_shrinker_init (ddev); +drm_gem_shmem_shrinker_register (ddev,"panfrost-shrinker" );return 0 ;
@@ -619,8 +604,8 @@ static int panfrost_remove(struct platform_device *pdev)struct panfrost_device *pfdev =platform_get_drvdata (pdev);struct drm_device *ddev = pfdev->ddev; +drm_gem_shmem_shrinker_unregister (ddev);drm_dev_unregister (ddev); -panfrost_gem_shrinker_cleanup (ddev);pm_runtime_get_sync (pfdev->dev);pm_runtime_disable (pfdev->dev);
drivers/gpu/drm/panfrost/panfrost_gem.c +23 -10
@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)struct panfrost_gem_object *bo =to_panfrost_bo (obj);struct panfrost_device *pfdev = obj->dev->dev_private; -/* -* Make sure the BO is no longer inserted in the shrinker list before -* taking care of the destruction itself. If we don't do that we have a -* race condition between this function and what's done in -* panfrost_gem_shrinker_scan(). -*/ -mutex_lock (&pfdev->shrinker_lock); -list_del_init (&bo->base.madv_list); -mutex_unlock (&pfdev->shrinker_lock); -/* * If we still have mappings attached to the BO, there's a problem in * our refcounting.
@@ -209,6 +199,25 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .vm_ops = &drm_gem_shmem_vm_ops, }; +static int panfrost_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +struct panfrost_gem_object *bo =to_panfrost_bo (&shmem->base); + +if (!drm_gem_shmem_is_purgeable (shmem)) +return -EOPNOTSUPP ; + +if (!mutex_trylock (&bo->mappings.lock)) +return -EBUSY ; + +panfrost_gem_teardown_mappings_locked (bo); + +drm_gem_shmem_purge (shmem); + +mutex_unlock (&bo->mappings.lock); + +return 0 ; +} +/** * panfrost_gem_create_object - Implementation of driver->gem_create_object. * @dev: DRM device
@@ -230,6 +239,7 @@ struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_tmutex_init (&obj->mappings.lock); obj->base.base.funcs = &panfrost_gem_funcs; obj->base.map_wc = !pfdev->coherent; + obj->base.evict = panfrost_shmem_evict;return &obj->base.base; }
@@ -266,6 +276,9 @@ panfrost_gem_create_with_handle(struct drm_file *file_priv,if (ret)return ERR_PTR (ret); +if (!bo->is_heap) +drm_gem_shmem_set_purgeable (shmem); +return bo; }
drivers/gpu/drm/panfrost/panfrost_gem.h +0 -9
@@ -30,12 +30,6 @@ struct panfrost_gem_object {struct mutex lock; } mappings; -/* -* Count the number of jobs referencing this BO so we don't let the -* shrinker reclaim this object prematurely. -*/ -atomic_t gpu_usecount; - bool noexec :1; bool is_heap :1; };
@@ -84,7 +78,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); -void panfrost_gem_shrinker_cleanup (struct drm_device *dev); -#endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0 -129
129 lines not on this page · raw message
drivers/gpu/drm/panfrost/panfrost_job.c +17 -1
@@ -272,6 +272,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,dma_resv_add_fence (bos[i]->resv, fence,DMA_RESV_USAGE_WRITE ); } +static int panfrost_objects_prepare (struct drm_gem_object **bos,int bo_count) +{ +struct panfrost_gem_object *bo; +int ret =0 ; + +while (!ret && bo_count--) { + bo =to_panfrost_bo (bos[bo_count]); + ret = bo->base.madv ? -ENOMEM :0 ; + } + +return ret; +} +int panfrost_job_push (struct panfrost_job *job) {struct panfrost_device *pfdev = job->pfdev;
@@ -283,6 +296,10 @@ int panfrost_job_push(struct panfrost_job *job)if (ret)return ret; + ret =panfrost_objects_prepare (job->bos, job->bo_count); +if (ret) + goto unlock; +mutex_lock (&pfdev->sched_lock);drm_sched_job_arm (&job->base);
@@ -324,7 +341,6 @@ static void panfrost_job_cleanup(struct kref *ref)if (!job->mappings[i])break ; -atomic_dec (&job->mappings[i]->obj->gpu_usecount);panfrost_gem_mapping_put (job->mappings[i]); }kvfree (job->mappings);
Signature · 1 line
2.37.3
Steven Price
maintainerRe: [PATCH v8 7/7] drm/panfrost: Switch to generic memory shrinker
Quoted 4 lines · Dmitry Osipenko
Replace Panfrost's custom memory shrinker with a common drm-shmem memory shrinker.
Signed-off-by Dmitry Osipenko <[email protected]>
Sadly this triggers GPU faults under memory pressure - it looks suspiciously like mappings are being freed while the jobs are still running.
I'm not sure I understand how the generic shrinker replicates the "gpu_usecount" atomic that Panfrost currently has, and I'm wondering if that's the cause?
Also just reverting this commit (so just patches 1-6) I can't actually get Panfrost to purge any memory. So I don't think the changes (most likely in patch 4) are quite right either.
At the moment I don't have the time to investigate in detail. But if you've any ideas for something specific I should look at I can run more testing.
Steve
Quoted 381 lines
--- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 19 +-- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +++-- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 -- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 ------------------ drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++- 7 files changed, 42 insertions(+), 171 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.cdrivers/gpu/drm/panfrost/Makefile +0
-1@@ -5,7 +5,6 @@ panfrost-y := \ panfrost_device.o \ panfrost_devfreq.o \ panfrost_gem.o \ - panfrost_gem_shrinker.o \ panfrost_gpu.o \ panfrost_job.o \ panfrost_mmu.o \drivers/gpu/drm/panfrost/panfrost_device.h +0
-4@@ -115,10 +115,6 @@ struct panfrost_device {atomic_t pending; } reset; -struct mutex shrinker_lock; -struct list_head shrinker_list; -struct shrinker shrinker; -struct panfrost_devfreq pfdevfreq; };drivers/gpu/drm/panfrost/panfrost_drv.c +2
-17@@ -160,7 +160,6 @@ panfrost_lookup_bos(struct drm_device *dev,break ; } -atomic_inc (&bo->gpu_usecount); job->mappings[i] = mapping; }@@ -392,7 +391,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, {struct panfrost_file_priv *priv = file_priv->driver_priv;struct drm_panfrost_madvise *args = data; -struct panfrost_device *pfdev = dev->dev_private;struct drm_gem_object *gem_obj;struct panfrost_gem_object *bo;int ret =0 ;@@ -409,7 +407,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (ret) goto out_put_object; -mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {struct panfrost_gem_mapping *first;@@ -435,17 +432,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, args->retained =drm_gem_shmem_madvise (&bo->base, args->madv); -if (args->retained) { -if (args->madv ==PANFROST_MADV_DONTNEED ) -list_move_tail (&bo->base.madv_list, - &pfdev->shrinker_list); -else if (args->madv ==PANFROST_MADV_WILLNEED ) -list_del_init (&bo->base.madv_list); - } - out_unlock_mappings:mutex_unlock (&bo->mappings.lock); -mutex_unlock (&pfdev->shrinker_lock);dma_resv_unlock (bo->base.base.resv); out_put_object:drm_gem_object_put (gem_obj);@@ -577,9 +565,6 @@ static int panfrost_probe(struct platform_device *pdev) ddev->dev_private = pfdev; pfdev->ddev = ddev; -mutex_init (&pfdev->shrinker_lock); -INIT_LIST_HEAD (&pfdev->shrinker_list); - err =panfrost_device_init (pfdev);if (err) {if (err != -EPROBE_DEFER )@@ -601,7 +586,7 @@ static int panfrost_probe(struct platform_device *pdev)if (err <0 ) goto err_out1; -panfrost_gem_shrinker_init (ddev); +drm_gem_shmem_shrinker_register (ddev,"panfrost-shrinker" );return 0 ;@@ -619,8 +604,8 @@ static int panfrost_remove(struct platform_device *pdev)struct panfrost_device *pfdev =platform_get_drvdata (pdev);struct drm_device *ddev = pfdev->ddev; +drm_gem_shmem_shrinker_unregister (ddev);drm_dev_unregister (ddev); -panfrost_gem_shrinker_cleanup (ddev);pm_runtime_get_sync (pfdev->dev);pm_runtime_disable (pfdev->dev);drivers/gpu/drm/panfrost/panfrost_gem.c +23
-10@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)struct panfrost_gem_object *bo =to_panfrost_bo (obj);struct panfrost_device *pfdev = obj->dev->dev_private; -/* -* Make sure the BO is no longer inserted in the shrinker list before -* taking care of the destruction itself. If we don't do that we have a -* race condition between this function and what's done in -* panfrost_gem_shrinker_scan(). -*/ -mutex_lock (&pfdev->shrinker_lock); -list_del_init (&bo->base.madv_list); -mutex_unlock (&pfdev->shrinker_lock); -/* * If we still have mappings attached to the BO, there's a problem in * our refcounting. @@ -209,6 +199,25 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .vm_ops = &drm_gem_shmem_vm_ops, }; +static int panfrost_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +struct panfrost_gem_object *bo =to_panfrost_bo (&shmem->base); + +if (!drm_gem_shmem_is_purgeable (shmem)) +return -EOPNOTSUPP ; + +if (!mutex_trylock (&bo->mappings.lock)) +return -EBUSY ; + +panfrost_gem_teardown_mappings_locked (bo); + +drm_gem_shmem_purge (shmem); + +mutex_unlock (&bo->mappings.lock); + +return 0 ; +} +/** * panfrost_gem_create_object - Implementation of driver->gem_create_object. * @dev: DRM device @@ -230,6 +239,7 @@ struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_tmutex_init (&obj->mappings.lock); obj->base.base.funcs = &panfrost_gem_funcs; obj->base.map_wc = !pfdev->coherent; + obj->base.evict = panfrost_shmem_evict;return &obj->base.base; }@@ -266,6 +276,9 @@ panfrost_gem_create_with_handle(struct drm_file *file_priv,if (ret)return ERR_PTR (ret); +if (!bo->is_heap) +drm_gem_shmem_set_purgeable (shmem); +return bo; }drivers/gpu/drm/panfrost/panfrost_gem.h +0
-9@@ -30,12 +30,6 @@ struct panfrost_gem_object {struct mutex lock; } mappings; -/* -* Count the number of jobs referencing this BO so we don't let the -* shrinker reclaim this object prematurely. -*/ -atomic_t gpu_usecount; - bool noexec :1; bool is_heap :1; };@@ -84,7 +78,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); -void panfrost_gem_shrinker_cleanup (struct drm_device *dev); -#endif /* __PANFROST_GEM_H__ */ drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0
-129@@ -1,129 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* Copyright (C) 2019 Arm Ltd. -* -* Based on msm_gem_freedreno.c: -* Copyright (C) 2016 Red Hat -* Author: Rob Clark <[email protected]> -*/ - -#include <linux/list.h> - -#include <drm/drm_device.h> -#include <drm/drm_gem_shmem_helper.h> - -#include "panfrost_device.h" -#include "panfrost_gem.h" -#include "panfrost_mmu.h" - -static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) -{ -return (shmem->madv >0 ) && - !shmem->pages_pin_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; -} - -static unsigned long -panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) -{ -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); -struct drm_gem_shmem_object *shmem; -unsigned long count =0 ; - -if (!mutex_trylock (&pfdev->shrinker_lock)) -return 0 ; - -list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (panfrost_gem_shmem_is_purgeable (shmem)) - count += shmem->base.size >>PAGE_SHIFT ; - } - -mutex_unlock (&pfdev->shrinker_lock); - -return count; -} - -static bool panfrost_gem_purge (struct drm_gem_object *obj) -{ -struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -struct panfrost_gem_object *bo =to_panfrost_bo (obj); -bool ret = false; - -if (atomic_read (&bo->gpu_usecount)) -return false; - -if (!mutex_trylock (&bo->mappings.lock)) -return false; - -if (!dma_resv_trylock (shmem->base.resv)) - goto unlock_mappings; - -panfrost_gem_teardown_mappings_locked (bo); -drm_gem_shmem_purge (&bo->base); - ret = true; - -dma_resv_unlock (shmem->base.resv); - -unlock_mappings: -mutex_unlock (&bo->mappings.lock); -return ret; -} - -static unsigned long -panfrost_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) -{ -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); -struct drm_gem_shmem_object *shmem, *tmp; -unsigned long freed =0 ; - -if (!mutex_trylock (&pfdev->shrinker_lock)) -return SHRINK_STOP ; - -list_for_each_entry_safe (shmem, tmp, &pfdev->shrinker_list, madv_list) { -if (freed >= sc->nr_to_scan) -break ; -if (drm_gem_shmem_is_purgeable (shmem) && -panfrost_gem_purge (&shmem->base)) { - freed += shmem->base.size >>PAGE_SHIFT ; -list_del_init (&shmem->madv_list); - } - } - -mutex_unlock (&pfdev->shrinker_lock); - -if (freed >0 ) -pr_info_ratelimited ("Purging %lu bytes\n" , freed <<PAGE_SHIFT ); - -return freed; -} - -/** -* panfrost_gem_shrinker_init - Initialize panfrost shrinker -* @dev: DRM device -* -* This function registers and sets up the panfrost shrinker. -*/ -void panfrost_gem_shrinker_init (struct drm_device *dev) -{ -struct panfrost_device *pfdev = dev->dev_private; - pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; - pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; - pfdev->shrinker.seeks =DEFAULT_SEEKS ; -WARN_ON (register_shrinker (&pfdev->shrinker,"drm-panfrost" )); -} - -/** -* panfrost_gem_shrinker_cleanup - Clean up panfrost shrinker -* @dev: DRM device -* -* This function unregisters the panfrost shrinker. -*/ -void panfrost_gem_shrinker_cleanup (struct drm_device *dev) -{ -struct panfrost_device *pfdev = dev->dev_private; - -if (pfdev->shrinker.nr_deferred) { -unregister_shrinker (&pfdev->shrinker); - } -}drivers/gpu/drm/panfrost/panfrost_job.c +17
-1@@ -272,6 +272,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,dma_resv_add_fence (bos[i]->resv, fence,DMA_RESV_USAGE_WRITE ); } +static int panfrost_objects_prepare (struct drm_gem_object **bos,int bo_count) +{ +struct panfrost_gem_object *bo; +int ret =0 ; + +while (!ret && bo_count--) { + bo =to_panfrost_bo (bos[bo_count]); + ret = bo->base.madv ? -ENOMEM :0 ; + } + +return ret; +} +int panfrost_job_push (struct panfrost_job *job) {struct panfrost_device *pfdev = job->pfdev;@@ -283,6 +296,10 @@ int panfrost_job_push(struct panfrost_job *job)if (ret)return ret; + ret =panfrost_objects_prepare (job->bos, job->bo_count); +if (ret) + goto unlock; +mutex_lock (&pfdev->sched_lock);drm_sched_job_arm (&job->base);@@ -324,7 +341,6 @@ static void panfrost_job_cleanup(struct kref *ref)if (!job->mappings[i])break ; -atomic_dec (&job->mappings[i]->obj->gpu_usecount);panfrost_gem_mapping_put (job->mappings[i]); }kvfree (job->mappings);
Dmitry Osipenko
maintainerRe: [PATCH v8 7/7] drm/panfrost: Switch to generic memory shrinker
Hello Steve,
Quoted 13 lines · Steven Price
Quoted 4 lines · Dmitry Osipenko
Replace Panfrost's custom memory shrinker with a common drm-shmem memory shrinker.
Signed-off-by Dmitry Osipenko <[email protected]>Sadly this triggers GPU faults under memory pressure - it looks suspiciously like mappings are being freed while the jobs are still running.
I'm not sure I understand how the generic shrinker replicates the "
gpu_usecount" atomic that Panfrost currently has, and I'm wondering if that's the cause?
Quoted 7 lines
Also just reverting this commit (so just patches 1-6) I can't actually get Panfrost to purge any memory. So I don't think the changes (most likely in patch 4) are quite right either.
At the moment I don't have the time to investigate in detail. But if you've any ideas for something specific I should look at I can run more testing.
Thank you for the testing! It just occurred to me that the shrinker callback lost the dma_resv_test_signaled() in comparison to the previous versions of this patchset. It appeared to me that the drm_gem_lru now checks whether reservation is busy, but it doesn't.
I saw a similar page faults once in a while when was testing the Panfrost driver, but then couldn't reproduce the faults after applying the IOMMU unmap range fix that Robin made recently.
I'll re-add the dma_resv_test_signaled() in v9, it was a luck that I didn't hit it much during my testing.
Signature · 2 lines
Best regards, Dmitry