Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
Subsystems (6)
Dmitry Osipenko
maintainerv17 cover Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
This series:
1. Adds common drm-shmem memory shrinker
2. Enables shrinker for VirtIO-GPU driver
3. Switches Panfrost driver to the common shrinker
4. Fixes bugs and improves drm-shmem code
Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise IGT: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvise
https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise
Changelog:
v17:- Dropped patches that added new drm-shmem sgt flags, fixing dma-buf UAF
in drm-prime error code path and preventing invalid page_count when GEM is freed. Will revist them later on and then factor them out into a seprate patchset.
- Dropped patches that replaced drm_gem_shmem_free() with drm_gem_object_put(), they not needed anymore after changing drm_gem_shmem_free() to not touch reservation lock.
- Addressed review comments from Boris Brezillon:
- Added new patch to clean up error unwinding in drm_gem_shmem_vmap_locked()
- Added new __drm_gem_shmem_put_pages() to let the callers to assert the held reservation lock themselves
- Moved replacement of shmem->pages check with refcount_read() in drm_gem_shmem_free() to the shrinker addition patch
- Improved commit message of the vmap_use_count patch
- Added r-bs from Boris Brezillon that he gave to v16
v16:- Added more comments to the code for the new drm-shmem flags
- Added r-bs from Boris Brezillon
- Fixed typos and made impovements pointed out by Boris Brezillon
- Replaced kref with refcount_t as was suggested by Boris Brezillon
- Corrected placement of got_sgt flag in the Lima driver, also renamed flag to got_pages_sgt
- Removed drm_gem_shmem_resv_assert_held() and made drm_gem_shmem_free() to free pages without a new func that doesn't touch resv lock, as was suggested by Boris Brezillon
- Added pages_pin_count to drm_gem_shmem_print_info()
v15:- Moved drm-shmem reference counters to use kref that allows to
optimize unlocked functions, like was suggested by Boris Brezillon.
- Changed drm/gem/shmem function names to use _locked postfix and dropped the _unlocked, making the naming scheme consistent across DRM code, like was suggested by Boris Brezillon.
- Added patch that fixes UAF in drm-shmem for drivers that import dma-buf and then release buffer in the import error code path.
- Added patch that makes drm-shmem use new flag for SGT's get_pages() refcounting, preventing unbalanced refcounting when GEM is freed.
- Fixed guest blob pinning in virtio-gpu driver that was missed previously in the shrinker patch.
- Moved VC4 and virtio-gpu drivers to use drm_gem_put() in GEM-creation error code paths, which is now required by drm-shmem and was missed in a previous patch versions.
- Virtio-GPU now attaches shmem pages to host on first use and not when BO is created. In older patch versions there was a potential race condition in the BO creation code path where both get_sgt()+object_attach() should've been made under same resv lock, otherwise pages could be evicted before attachment is invoked.
- Virtio-GPU and drm-shmem shrinker patches are split into smaller ones.
v14:- All the prerequisite reservation locking patches landed upstream,
previously were a part of this series in v13 and older.
https://lore.kernel.org/dri-devel/[email protected]/
- Added patches to improve locked/unlocked function names, like was suggested by Boris Brezillon for v13.
- Made all exported drm-shmem symbols GPL, like was previously discussed with Thomas Zimmermann on this series.
- Improved virtio-gpu shrinker patch. Now it won't detach purged BO when userspace closes GEM. Crosvm (and not qemu) checks res_id on CMD_CTX_DETACH_RESOURCE and prints noisy error message if ID is invalid, which wasn't noticed before.
v13:- Updated virtio-gpu shrinker patch to use drm_gem_shmem_object_pin()
directly instead of drm_gem_pin() and dropped patch that exported drm_gem_pin() functions, like was requested by Thomas Zimmermann in v12.
v12:- Fixed the "no previous prototype for function" warning reported by
kernel build bot for v11.
- Fixed the missing reservation lock reported by Intel CI for VGEM driver. Other drivers using drm-shmem were affected similarly to VGEM. The problem was in the dma-buf attachment code path that led to drm-shmem pinning function which assumed the held reservation lock by drm_gem_pin(). In the past that code path was causing trouble for i915 driver and we've changed the locking scheme for the attachment code path in the dma-buf core to let exporters to handle the locking themselves. After a closer investigation, I realized that my assumption about testing of dma-buf export code path using Panfrost driver was incorrect. Now I created additional local test to exrecise the Panfrost export path. I also reproduced the issue reported by the Intel CI for v10. It's all fixed now by making the drm_gem_shmem_pin() to take the resv lock by itself.
- Patches are based on top of drm-tip, CC'd intel-gfx CI for testing.
v11:- Rebased on a recent linux-next. Added new patch as a result:
drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
It's needed by the virtio-gpu driver to swap-in/unevict shmem object, previously get_pages_sgt() didn't use locking.
- Separated the "Add memory shrinker" patch into smaller parts to ease the reviewing, as was requested by Thomas Zimmermann:
drm/shmem-helper: Factor out pages alloc/release from
drm_gem_shmem_get/put_pages()
drm/shmem-helper: Add pages_pin_count field drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin drm/shmem-helper: Factor out unpinning part from drm_gem_shmem_purge()
- Addessed the v10 review comments from Thomas Zimmermann: return errno instead of bool, sort code alphabetically, rename function and etc minor changes.
- Added new patch to remove the "map->is_iomem" from drm-shmem, as was suggested by Thomas Zimmermann.
- Added acks and r-b's that were given to v10.
v10:- Was partially applied to misc-fixes/next.
https://lore.kernel.org/dri-devel/[email protected]/T/
Dmitry Osipenko (18):
drm/gem: Change locked/unlocked postfix of drm_gem_v/unmap() function
names
drm/gem: Add _locked postfix to functions that have unlocked
counterpart
drm/shmem-helper: Make all exported symbols GPL drm/shmem-helper: Refactor locked/unlocked functions drm/shmem-helper: Remove obsoleted is_iomem test drm/shmem-helper: Add and use pages_pin_count drm/shmem-helper: Use refcount_t for pages_use_count drm/shmem-helper: Add and use lockless drm_gem_shmem_get_pages() drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin drm/shmem-helper: Use refcount_t for vmap_use_count drm/shmem-helper: Improve drm_gem_shmem_vmap_locked() error handling drm/shmem-helper: Prepare drm_gem_shmem_free() to shrinker addition drm/shmem-helper: Add memory shrinker
drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked() drm/virtio: Pin display framebuffer BO
drm/virtio: Attach shmem BOs dynamically
drm/virtio: Support memory shrinking
drm/panfrost: Switch to generic memory shrinker
drivers/gpu/drm/drm_client.c | 6 +- drivers/gpu/drm/drm_gem.c | 26 +- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 588 +++++++++++++++--- drivers/gpu/drm/drm_internal.h | 4 +- drivers/gpu/drm/drm_prime.c | 4 +- drivers/gpu/drm/lima/lima_gem.c | 10 +- drivers/gpu/drm/lima/lima_sched.c | 4 +- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 29 +- drivers/gpu/drm/panfrost/panfrost_dump.c | 4 +- drivers/gpu/drm/panfrost/panfrost_gem.c | 36 +- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 - .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 122 ---- drivers/gpu/drm/panfrost/panfrost_job.c | 18 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 4 +- drivers/gpu/drm/panfrost/panfrost_perfcnt.c | 6 +- drivers/gpu/drm/v3d/v3d_bo.c | 4 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 22 +- drivers/gpu/drm/virtio/virtgpu_gem.c | 80 +++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 57 +- drivers/gpu/drm/virtio/virtgpu_kms.c | 8 + drivers/gpu/drm/virtio/virtgpu_object.c | 145 ++++- drivers/gpu/drm/virtio/virtgpu_plane.c | 17 +- drivers/gpu/drm/virtio/virtgpu_submit.c | 15 +- drivers/gpu/drm/virtio/virtgpu_vq.c | 40 ++ include/drm/drm_device.h | 10 +- include/drm/drm_gem.h | 6 +- include/drm/drm_gem_shmem_helper.h | 127 +++- include/uapi/drm/virtgpu_drm.h | 14 + 31 files changed, 1050 insertions(+), 376 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 1/18 drm/gem: Change locked/unlocked postfix of drm_gem_v/unmap() function names
Make drm/gem API function names consistent by having locked function use the _locked postfix in the name, while the unlocked variants don't use the _unlocked postfix. Rename drm_gem_v/unmap() function names to make them consistent with the rest of the API functions.
--- drivers/gpu/drm/drm_client.c | 6 +++--- drivers/gpu/drm/drm_gem.c | 20 ++++++++++---------- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +++--- drivers/gpu/drm/drm_internal.h | 4 ++-- drivers/gpu/drm/drm_prime.c | 4 ++-- drivers/gpu/drm/lima/lima_sched.c | 4 ++-- drivers/gpu/drm/panfrost/panfrost_dump.c | 4 ++-- drivers/gpu/drm/panfrost/panfrost_perfcnt.c | 6 +++--- include/drm/drm_gem.h | 4 ++-- 9 files changed, 29 insertions(+), 29 deletions(-)
drivers/gpu/drm/drm_client.c +3 -3
@@ -265,7 +265,7 @@ void drm_client_dev_restore(struct drm_device *dev)static void drm_client_buffer_delete (struct drm_client_buffer *buffer) {if (buffer->gem) { -drm_gem_vunmap_unlocked (buffer->gem, &buffer->map); +drm_gem_vunmap (buffer->gem, &buffer->map);drm_gem_object_put (buffer->gem); }
@@ -349,7 +349,7 @@ drm_client_buffer_vmap(struct drm_client_buffer *buffer,* fd_install step out of the driver backend hooks, to make that * final step optional for internal users. */ - ret =drm_gem_vmap_unlocked (buffer->gem, map); + ret =drm_gem_vmap (buffer->gem, map);if (ret)return ret;
@@ -371,7 +371,7 @@ void drm_client_buffer_vunmap(struct drm_client_buffer *buffer) {struct iosys_map *map = &buffer->map; -drm_gem_vunmap_unlocked (buffer->gem, map); +drm_gem_vunmap (buffer->gem, map); }EXPORT_SYMBOL (drm_client_buffer_vunmap);
drivers/gpu/drm/drm_gem.c +10 -10
@@ -1173,7 +1173,7 @@ void drm_gem_unpin(struct drm_gem_object *obj) obj->funcs->unpin (obj); } -int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map) +int drm_gem_vmap_locked (struct drm_gem_object *obj,struct iosys_map *map) {int ret;
@@ -1190,9 +1190,9 @@ int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)return 0 ; } -EXPORT_SYMBOL (drm_gem_vmap); +EXPORT_SYMBOL (drm_gem_vmap_locked); -void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map) +void drm_gem_vunmap_locked (struct drm_gem_object *obj,struct iosys_map *map) {dma_resv_assert_held (obj->resv);
@@ -1205,27 +1205,27 @@ void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map)/* Always set the mapping to NULL. Callers may rely on this. */ iosys_map_clear (map); } -EXPORT_SYMBOL (drm_gem_vunmap); +EXPORT_SYMBOL (drm_gem_vunmap_locked); -int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map) {int ret;dma_resv_lock (obj->resv,NULL ); - ret =drm_gem_vmap (obj, map); + ret =drm_gem_vmap_locked (obj, map);dma_resv_unlock (obj->resv);return ret; } -EXPORT_SYMBOL (drm_gem_vmap_unlocked); +EXPORT_SYMBOL (drm_gem_vmap); -void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map) {dma_resv_lock (obj->resv,NULL ); -drm_gem_vunmap (obj, map); +drm_gem_vunmap_locked (obj, map);dma_resv_unlock (obj->resv); } -EXPORT_SYMBOL (drm_gem_vunmap_unlocked); +EXPORT_SYMBOL (drm_gem_vunmap);/** * drm_gem_lock_reservations - Sets up the ww context and acquires
drivers/gpu/drm/drm_gem_framebuffer_helper.c +3 -3
@@ -362,7 +362,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, struct iosys_map *map, ret = -EINVAL ; goto err_drm_gem_vunmap; } - ret =drm_gem_vmap_unlocked (obj, &map[i]); + ret =drm_gem_vmap (obj, &map[i]);if (ret) goto err_drm_gem_vunmap; }
@@ -384,7 +384,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, struct iosys_map *map, obj =drm_gem_fb_get_obj (fb, i);if (!obj)continue ; -drm_gem_vunmap_unlocked (obj, &map[i]); +drm_gem_vunmap (obj, &map[i]); }return ret; }
@@ -411,7 +411,7 @@ void drm_gem_fb_vunmap(struct drm_framebuffer *fb, struct iosys_map *map)continue ;if (iosys_map_is_null (&map[i]))continue ; -drm_gem_vunmap_unlocked (obj, &map[i]); +drm_gem_vunmap (obj, &map[i]); } }EXPORT_SYMBOL (drm_gem_fb_vunmap);
drivers/gpu/drm/drm_internal.h +2 -2
@@ -175,8 +175,8 @@ void drm_gem_print_info(struct drm_printer *p, unsigned int indent,int drm_gem_pin (struct drm_gem_object *obj);void drm_gem_unpin (struct drm_gem_object *obj); -int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map); -void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map); +int drm_gem_vmap_locked (struct drm_gem_object *obj,struct iosys_map *map); +void drm_gem_vunmap_locked (struct drm_gem_object *obj,struct iosys_map *map);/* drm_debugfs.c drm_debugfs_crc.c */ #if defined(CONFIG_DEBUG_FS )
drivers/gpu/drm/drm_prime.c +2 -2
@@ -682,7 +682,7 @@ int drm_gem_dmabuf_vmap(struct dma_buf *dma_buf, struct iosys_map *map) {struct drm_gem_object *obj = dma_buf->priv; -return drm_gem_vmap (obj, map); +return drm_gem_vmap_locked (obj, map); }EXPORT_SYMBOL (drm_gem_dmabuf_vmap);
@@ -698,7 +698,7 @@ void drm_gem_dmabuf_vunmap(struct dma_buf *dma_buf, struct iosys_map *map) {struct drm_gem_object *obj = dma_buf->priv; -drm_gem_vunmap (obj, map); +drm_gem_vunmap_locked (obj, map); }EXPORT_SYMBOL (drm_gem_dmabuf_vunmap);
drivers/gpu/drm/lima/lima_sched.c +2 -2
@@ -371,7 +371,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task) }else { buffer_chunk->size =lima_bo_size (bo); - ret =drm_gem_vmap_unlocked (&bo->base.base, &map); + ret =drm_gem_vmap (&bo->base.base, &map);if (ret) {kvfree (et); goto out;
@@ -379,7 +379,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task)memcpy (buffer_chunk +1 , map.vaddr, buffer_chunk->size); -drm_gem_vunmap_unlocked (&bo->base.base, &map); +drm_gem_vunmap (&bo->base.base, &map); } buffer_chunk = (void *)(buffer_chunk +1 ) + buffer_chunk->size;
drivers/gpu/drm/panfrost/panfrost_dump.c +2 -2
@@ -209,7 +209,7 @@ void panfrost_core_dump(struct panfrost_job *job) goto dump_header; } - ret =drm_gem_vmap_unlocked (&bo->base.base, &map); + ret =drm_gem_vmap (&bo->base.base, &map);if (ret) {dev_err (pfdev->dev,"Panfrost Dump: couldn't map Buffer Object\n" ); iter.hdr->bomap.valid =0 ;
@@ -236,7 +236,7 @@ void panfrost_core_dump(struct panfrost_job *job) vaddr = map.vaddr;memcpy (iter.data, vaddr, bo->base.base.size); -drm_gem_vunmap_unlocked (&bo->base.base, &map); +drm_gem_vunmap (&bo->base.base, &map); iter.hdr->bomap.valid =1 ;
drivers/gpu/drm/panfrost/panfrost_perfcnt.c +3 -3
@@ -106,7 +106,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev, goto err_close_bo; } - ret =drm_gem_vmap_unlocked (&bo->base, &map); + ret =drm_gem_vmap (&bo->base, &map);if (ret) goto err_put_mapping; perfcnt->buf = map.vaddr;
@@ -165,7 +165,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev,return 0 ; err_vunmap: -drm_gem_vunmap_unlocked (&bo->base, &map); +drm_gem_vunmap (&bo->base, &map); err_put_mapping:panfrost_gem_mapping_put (perfcnt->mapping); err_close_bo:
@@ -195,7 +195,7 @@ static int panfrost_perfcnt_disable_locked(struct panfrost_device *pfdev, GPU_PERFCNT_CFG_MODE(GPU_PERFCNT_CFG_MODE_OFF)); perfcnt->user =NULL ; -drm_gem_vunmap_unlocked (&perfcnt->mapping->obj->base.base, &map); +drm_gem_vunmap (&perfcnt->mapping->obj->base.base, &map); perfcnt->buf =NULL ;panfrost_gem_close (&perfcnt->mapping->obj->base.base, file_priv);panfrost_mmu_as_put (pfdev, perfcnt->mapping->mmu);
include/drm/drm_gem.h +2 -2
@@ -518,8 +518,8 @@ struct page **drm_gem_get_pages(struct drm_gem_object *obj);void drm_gem_put_pages (struct drm_gem_object *obj,struct page **pages,bool dirty,bool accessed); -int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); -void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map); +void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map);int drm_gem_objects_lookup (struct drm_file *filp,void __user *bo_handles,int count,struct drm_gem_object ***objs_out);
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 2/18 drm/gem: Add _locked postfix to functions that have unlocked counterpart
In reply to Dmitry OsipenkoAdd _locked postfix to drm_gem functions that have unlocked counterpart functions to make GEM functions naming more consistent and intuitive in regards to the locking requirements.
--- drivers/gpu/drm/drm_gem.c | 6 +++--- include/drm/drm_gem.h | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-)
drivers/gpu/drm/drm_gem.c +3 -3
@@ -1488,10 +1488,10 @@ drm_gem_lru_scan(struct drm_gem_lru *lru,EXPORT_SYMBOL (drm_gem_lru_scan);/** -* drm_gem_evict - helper to evict backing pages for a GEM object +* drm_gem_evict_locked - helper to evict backing pages for a GEM object * @obj: obj in question */ -int drm_gem_evict (struct drm_gem_object *obj) +int drm_gem_evict_locked (struct drm_gem_object *obj) {dma_resv_assert_held (obj->resv);
@@ -1503,4 +1503,4 @@ int drm_gem_evict(struct drm_gem_object *obj)return 0 ; } -EXPORT_SYMBOL (drm_gem_evict); +EXPORT_SYMBOL (drm_gem_evict_locked);
include/drm/drm_gem.h +1 -1
@@ -542,7 +542,7 @@ unsigned long drm_gem_lru_scan(struct drm_gem_lru *lru,unsigned long *remaining, bool (*shrink)(struct drm_gem_object *obj)); -int drm_gem_evict (struct drm_gem_object *obj); +int drm_gem_evict_locked (struct drm_gem_object *obj);#ifdef CONFIG_LOCKDEP /**
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 3/18 drm/shmem-helper: Make all exported symbols GPL
In reply to Dmitry OsipenkoMake all drm-shmem exported symbols GPL to make them consistent with the rest of drm-shmem symbols.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +8 -8
@@ -226,7 +226,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_put_pages); +EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages);static int drm_gem_shmem_pin_locked (struct drm_gem_shmem_object *shmem) {
@@ -271,7 +271,7 @@ int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)return ret; } -EXPORT_SYMBOL (drm_gem_shmem_pin); +EXPORT_SYMBOL_GPL (drm_gem_shmem_pin);/** * drm_gem_shmem_unpin - Unpin backing pages for a shmem GEM object
@@ -290,7 +290,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)drm_gem_shmem_unpin_locked (shmem);dma_resv_unlock (shmem->base.resv); } -EXPORT_SYMBOL (drm_gem_shmem_unpin); +EXPORT_SYMBOL_GPL (drm_gem_shmem_unpin);/* * drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object
@@ -360,7 +360,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return ret; } -EXPORT_SYMBOL (drm_gem_shmem_vmap); +EXPORT_SYMBOL_GPL (drm_gem_shmem_vmap);/* * drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object
@@ -396,7 +396,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem, shmem->vaddr =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_vunmap); +EXPORT_SYMBOL_GPL (drm_gem_shmem_vunmap); static int drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
@@ -435,7 +435,7 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)return (madv >=0 ); } -EXPORT_SYMBOL (drm_gem_shmem_madvise); +EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise);void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) {
@@ -467,7 +467,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); } -EXPORT_SYMBOL (drm_gem_shmem_purge); +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
@@ -642,7 +642,7 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); } -EXPORT_SYMBOL (drm_gem_shmem_print_info); +EXPORT_SYMBOL_GPL (drm_gem_shmem_print_info);/** * drm_gem_shmem_get_sg_table - Provide a scatter/gather table of pinned
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 4/18 drm/shmem-helper: Refactor locked/unlocked functions
In reply to Dmitry OsipenkoAdd locked and remove unlocked postfixes from drm-shmem function names, making names consistent with the drm/gem core code.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 64 +++++++++---------- drivers/gpu/drm/lima/lima_gem.c | 8 +-- drivers/gpu/drm/panfrost/panfrost_drv.c | 2 +- drivers/gpu/drm/panfrost/panfrost_gem.c | 6 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 2 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- drivers/gpu/drm/v3d/v3d_bo.c | 4 +- drivers/gpu/drm/virtio/virtgpu_object.c | 4 +- include/drm/drm_gem_shmem_helper.h | 36 +++++------ 9 files changed, 64 insertions(+), 64 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +32 -32
@@ -43,8 +43,8 @@ static const struct drm_gem_object_funcs drm_gem_shmem_funcs = { .pin = drm_gem_shmem_object_pin, .unpin = drm_gem_shmem_object_unpin, .get_sg_table = drm_gem_shmem_object_get_sg_table, - .vmap = drm_gem_shmem_object_vmap, - .vunmap = drm_gem_shmem_object_vunmap, + .vmap = drm_gem_shmem_object_vmap_locked, + .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = drm_gem_shmem_object_mmap, .vm_ops = &drm_gem_shmem_vm_ops, };
@@ -153,7 +153,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); }if (shmem->pages) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem);drm_WARN_ON (obj->dev, shmem->pages_use_count);
@@ -165,7 +165,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages;
@@ -199,12 +199,12 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem) }/* -* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object +* drm_gem_shmem_put_pages_locked - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object * * This function decreases the use count and puts the backing pages when use drops to zero. */ -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;
@@ -226,7 +226,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } -EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages); +EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);static int drm_gem_shmem_pin_locked (struct drm_gem_shmem_object *shmem) {
@@ -234,7 +234,7 @@ static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages_locked (shmem);return ret; }
@@ -243,7 +243,7 @@ static void drm_gem_shmem_unpin_locked(struct drm_gem_shmem_object *shmem) {dma_resv_assert_held (shmem->base.resv); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem); }/**
@@ -293,7 +293,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)EXPORT_SYMBOL_GPL (drm_gem_shmem_unpin);/* -* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object +* drm_gem_shmem_vmap_locked - Create a virtual mapping for a shmem GEM object * @shmem: shmem GEM object * @map: Returns the kernel virtual address of the SHMEM GEM object's backing * store.
@@ -302,13 +302,13 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_unpin);* exists for the buffer backing the shmem GEM object. It hides the differences * between dma-buf imported and natively allocated objects. * -* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap(). +* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap_locked(). * * Returns: * 0 on success or a negative error code on failure. */ -int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +int drm_gem_shmem_vmap_locked (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;int ret =0 ;
@@ -331,7 +331,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages_locked (shmem);if (ret) goto err_zero_use;
@@ -354,28 +354,28 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem); err_zero_use: shmem->vmap_use_count =0 ;return ret; } -EXPORT_SYMBOL_GPL (drm_gem_shmem_vmap); +EXPORT_SYMBOL_GPL (drm_gem_shmem_vmap_locked);/* -* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object +* drm_gem_shmem_vunmap_locked - Unmap a virtual mapping for a shmem GEM object * @shmem: shmem GEM object * @map: Kernel virtual address where the SHMEM GEM object was mapped * * This function cleans up a kernel virtual address mapping acquired by -* drm_gem_shmem_vmap(). The mapping is only removed when the use count drops to -* zero. +* drm_gem_shmem_vmap_locked(). The mapping is only removed when the use count +* drops to zero. * * This function hides the differences between dma-buf imported and natively * allocated objects. */ -void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +void drm_gem_shmem_vunmap_locked (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;
@@ -391,12 +391,12 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem); } shmem->vaddr =NULL ; } -EXPORT_SYMBOL_GPL (drm_gem_shmem_vunmap); +EXPORT_SYMBOL_GPL (drm_gem_shmem_vunmap_locked); static int drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
@@ -424,7 +424,7 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,/* Update madvise status, returns true if not purged, else * false or -errno. */ -int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) +int drm_gem_shmem_madvise_locked (struct drm_gem_shmem_object *shmem,int madv) {dma_resv_assert_held (shmem->base.resv);
@@ -435,9 +435,9 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)return (madv >=0 ); } -EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise); +EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise_locked); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;
@@ -451,7 +451,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); shmem->sgt =NULL ; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem); shmem->madv =-1 ;
@@ -467,7 +467,7 @@ void drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); } -EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge_locked);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
@@ -564,7 +564,7 @@ static void drm_gem_shmem_vm_close(struct vm_area_struct *vma)struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj);dma_resv_lock (shmem->base.resv,NULL ); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_close (vma);
@@ -611,7 +611,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct }dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages_locked (shmem);dma_resv_unlock (shmem->base.resv);if (ret)
@@ -679,7 +679,7 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_drm_WARN_ON (obj->dev, obj->import_attach); - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages_locked (shmem);if (ret)return ERR_PTR (ret);
@@ -701,7 +701,7 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_sg_free_table (sgt);kfree (sgt); err_put_pages: -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_put_pages_locked (shmem);return ERR_PTR (ret); }
drivers/gpu/drm/lima/lima_gem.c +4 -4
@@ -180,7 +180,7 @@ static int lima_gem_pin(struct drm_gem_object *obj)if (bo->heap_size)return -EINVAL ; -return drm_gem_shmem_pin (&bo->base); +return drm_gem_shmem_object_pin (obj); }static int lima_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map)
@@ -190,7 +190,7 @@ static int lima_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map)if (bo->heap_size)return -EINVAL ; -return drm_gem_shmem_vmap (&bo->base, map); +return drm_gem_shmem_object_vmap_locked (obj, map); }static int lima_gem_mmap (struct drm_gem_object *obj,struct vm_area_struct *vma)
@@ -200,7 +200,7 @@ static int lima_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *vma)if (bo->heap_size)return -EINVAL ; -return drm_gem_shmem_mmap (&bo->base, vma); +return drm_gem_shmem_object_mmap (obj, vma); }static const struct drm_gem_object_funcs lima_gem_funcs = {
@@ -212,7 +212,7 @@ static const struct drm_gem_object_funcs lima_gem_funcs = { .unpin = drm_gem_shmem_object_unpin, .get_sg_table = drm_gem_shmem_object_get_sg_table, .vmap = lima_gem_vmap, - .vunmap = drm_gem_shmem_object_vunmap, + .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = lima_gem_mmap, .vm_ops = &drm_gem_shmem_vm_ops, };
drivers/gpu/drm/panfrost/panfrost_drv.c +1 -1
@@ -436,7 +436,7 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, } } - args->retained =drm_gem_shmem_madvise (&bo->base, args->madv); + args->retained =drm_gem_shmem_madvise_locked (&bo->base, args->madv);if (args->retained) {if (args->madv ==PANFROST_MADV_DONTNEED )
drivers/gpu/drm/panfrost/panfrost_gem.c +3 -3
@@ -192,7 +192,7 @@ static int panfrost_gem_pin(struct drm_gem_object *obj)if (bo->is_heap)return -EINVAL ; -return drm_gem_shmem_pin (&bo->base); +return drm_gem_shmem_object_pin (obj); }static const struct drm_gem_object_funcs panfrost_gem_funcs = {
@@ -203,8 +203,8 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .pin = panfrost_gem_pin, .unpin = drm_gem_shmem_object_unpin, .get_sg_table = drm_gem_shmem_object_get_sg_table, - .vmap = drm_gem_shmem_object_vmap, - .vunmap = drm_gem_shmem_object_vunmap, + .vmap = drm_gem_shmem_object_vmap_locked, + .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = drm_gem_shmem_object_mmap, .vm_ops = &drm_gem_shmem_vm_ops, };
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +1 -1
@@ -52,7 +52,7 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj) goto unlock_mappings;panfrost_gem_teardown_mappings_locked (bo); -drm_gem_shmem_purge (&bo->base); +drm_gem_shmem_purge_locked (&bo->base); ret = true;dma_resv_unlock (shmem->base.resv);
drivers/gpu/drm/panfrost/panfrost_mmu.c +1 -1
@@ -535,7 +535,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, err_map:sg_free_table (sgt); err_pages: -drm_gem_shmem_put_pages (&bo->base); +drm_gem_shmem_put_pages_locked (&bo->base); err_unlock:dma_resv_unlock (obj->resv); err_bo:
drivers/gpu/drm/v3d/v3d_bo.c +2 -2
@@ -56,8 +56,8 @@ static const struct drm_gem_object_funcs v3d_gem_funcs = { .pin = drm_gem_shmem_object_pin, .unpin = drm_gem_shmem_object_unpin, .get_sg_table = drm_gem_shmem_object_get_sg_table, - .vmap = drm_gem_shmem_object_vmap, - .vunmap = drm_gem_shmem_object_vunmap, + .vmap = drm_gem_shmem_object_vmap_locked, + .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = drm_gem_shmem_object_mmap, .vm_ops = &drm_gem_shmem_vm_ops, };
drivers/gpu/drm/virtio/virtgpu_object.c +2 -2
@@ -106,8 +106,8 @@ static const struct drm_gem_object_funcs virtio_gpu_shmem_funcs = { .pin = drm_gem_shmem_object_pin, .unpin = drm_gem_shmem_object_unpin, .get_sg_table = drm_gem_shmem_object_get_sg_table, - .vmap = drm_gem_shmem_object_vmap, - .vunmap = drm_gem_shmem_object_vunmap, + .vmap = drm_gem_shmem_object_vmap_locked, + .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = drm_gem_shmem_object_mmap, .vm_ops = &drm_gem_shmem_vm_ops, };
include/drm/drm_gem_shmem_helper.h +18 -18
@@ -99,16 +99,16 @@ struct drm_gem_shmem_object {struct drm_gem_shmem_object *drm_gem_shmem_create (struct drm_device *dev,size_t size);void drm_gem_shmem_free (struct drm_gem_shmem_object *shmem); -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem); +void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem);int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_unpin (struct drm_gem_shmem_object *shmem); -int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map); -void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map); +int drm_gem_shmem_vmap_locked (struct drm_gem_shmem_object *shmem, +struct iosys_map *map); +void drm_gem_shmem_vunmap_locked (struct drm_gem_shmem_object *shmem, +struct iosys_map *map);int drm_gem_shmem_mmap (struct drm_gem_shmem_object *shmem,struct vm_area_struct *vma); -int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_madvise_locked (struct drm_gem_shmem_object *shmem,int madv);static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) {
@@ -117,7 +117,7 @@ static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem !shmem->base.dma_buf && !shmem->base.import_attach; } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);
@@ -208,22 +208,22 @@ static inline struct sg_table *drm_gem_shmem_object_get_sg_table(struct drm_gem_ }/* -* drm_gem_shmem_object_vmap - GEM object function for drm_gem_shmem_vmap() +* drm_gem_shmem_object_vmap_locked - GEM object function for drm_gem_shmem_vmap_locked() * @obj: GEM object * @map: Returns the kernel virtual address of the SHMEM GEM object's backing store. * -* This function wraps drm_gem_shmem_vmap(). Drivers that employ the shmem helpers should -* use it as their &drm_gem_object_funcs.vmap handler. +* This function wraps drm_gem_shmem_vmap_locked(). Drivers that employ the shmem +* helpers should use it as their &drm_gem_object_funcs.vmap handler. * * Returns: * 0 on success or a negative error code on failure. */ -static inline int drm_gem_shmem_object_vmap (struct drm_gem_object *obj, -struct iosys_map *map) +static inline int drm_gem_shmem_object_vmap_locked (struct drm_gem_object *obj, +struct iosys_map *map) {struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -return drm_gem_shmem_vmap (shmem, map); +return drm_gem_shmem_vmap_locked (shmem, map); }/*
@@ -231,15 +231,15 @@ static inline int drm_gem_shmem_object_vmap(struct drm_gem_object *obj,* @obj: GEM object * @map: Kernel virtual address where the SHMEM GEM object was mapped * -* This function wraps drm_gem_shmem_vunmap(). Drivers that employ the shmem helpers should -* use it as their &drm_gem_object_funcs.vunmap handler. +* This function wraps drm_gem_shmem_vunmap_locked(). Drivers that employ the shmem +* helpers should use it as their &drm_gem_object_funcs.vunmap handler. */ -static inline void drm_gem_shmem_object_vunmap (struct drm_gem_object *obj, -struct iosys_map *map) +static inline void drm_gem_shmem_object_vunmap_locked (struct drm_gem_object *obj, +struct iosys_map *map) {struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -drm_gem_shmem_vunmap (shmem, map); +drm_gem_shmem_vunmap_locked (shmem, map); }/**
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 5/18 drm/shmem-helper: Remove obsoleted is_iomem test
In reply to Dmitry OsipenkoEverything that uses the mapped buffer should be agnostic to is_iomem. The only reason for the is_iomem test is that we're setting shmem->vaddr to the returned map->vaddr. Now that the shmem->vaddr code is gone, remove the obsoleted is_iomem test to clean up the code.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 6 ------ 1 file changed, 6 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +0 -6
@@ -315,12 +315,6 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,if (obj->import_attach) { ret =dma_buf_vmap (obj->import_attach->dmabuf, map); -if (!ret) { -if (drm_WARN_ON (obj->dev, map->is_iomem)) { -dma_buf_vunmap (obj->import_attach->dmabuf, map); -return -EIO ; - } - } }else {pgprot_t prot =PAGE_KERNEL ;
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 6/18 drm/shmem-helper: Add and use pages_pin_count
In reply to Dmitry OsipenkoAdd separate pages_pin_count for tracking of whether drm-shmem pages are moveable or not. With the addition of memory shrinker support to drm-shmem, the pages_use_count will no longer determine whether pages are hard-pinned in memory, but whether pages exist and are soft-pinned (and could be swapped out). The pages_pin_count > 1 will hard-pin pages in memory.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 24 ++++++++++++++++-------- include/drm/drm_gem_shmem_helper.h | 10 ++++++++++ 2 files changed, 26 insertions(+), 8 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +16 -8
@@ -234,18 +234,16 @@ static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); +if (refcount_inc_not_zero (&shmem->pages_pin_count)) +return 0 ; + ret =drm_gem_shmem_get_pages_locked (shmem); +if (!ret) +refcount_set (&shmem->pages_pin_count,1 );return ret; } -static void drm_gem_shmem_unpin_locked (struct drm_gem_shmem_object *shmem) -{ -dma_resv_assert_held (shmem->base.resv); - -drm_gem_shmem_put_pages_locked (shmem); -} -/** * drm_gem_shmem_pin - Pin backing pages for a shmem GEM object * @shmem: shmem GEM object
@@ -263,6 +261,9 @@ int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)drm_WARN_ON (obj->dev, obj->import_attach); +if (refcount_inc_not_zero (&shmem->pages_pin_count)) +return 0 ; + ret =dma_resv_lock_interruptible (shmem->base.resv,NULL );if (ret)return ret;
@@ -286,8 +287,14 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)drm_WARN_ON (obj->dev, obj->import_attach); +if (refcount_dec_not_one (&shmem->pages_pin_count)) +return ; +dma_resv_lock (shmem->base.resv,NULL ); -drm_gem_shmem_unpin_locked (shmem); + +if (refcount_dec_and_test (&shmem->pages_pin_count)) +drm_gem_shmem_put_pages_locked (shmem); +dma_resv_unlock (shmem->base.resv); }EXPORT_SYMBOL_GPL (drm_gem_shmem_unpin);
@@ -632,6 +639,7 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,if (shmem->base.import_attach)return ; +drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count));drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr);
include/drm/drm_gem_shmem_helper.h +10 -0
@@ -39,6 +39,16 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages allowed to be evicted and purged by memory +* shrinker only when the count is zero, otherwise pages +* are hard-pinned in memory. +*/ +refcount_t pages_pin_count; +/** * @madv: State for madvise *
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 7/18 drm/shmem-helper: Use refcount_t for pages_use_count
In reply to Dmitry OsipenkoUse atomic refcount_t helper for pages_use_count to optimize pin/unpin functions by skipping reservation locking while GEM's pin refcount > 1.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 33 +++++++++++-------------- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- include/drm/drm_gem_shmem_helper.h | 2 +- 4 files changed, 18 insertions(+), 21 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +15 -18
@@ -155,7 +155,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (shmem->pages)drm_gem_shmem_put_pages_locked (shmem); -drm_WARN_ON (obj->dev, shmem->pages_use_count); +drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count));dma_resv_unlock (shmem->base.resv); }
@@ -172,14 +172,13 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); -if (shmem->pages_use_count++ >0 ) +if (refcount_inc_not_zero (&shmem->pages_use_count))return 0 ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {drm_dbg_kms (obj->dev,"Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }
@@ -195,6 +194,8 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages = pages; +refcount_set (&shmem->pages_use_count,1 ); +return 0 ; }
@@ -210,21 +211,17 @@ void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); -if (drm_WARN_ON_ONCE (obj->dev, !shmem->pages_use_count)) -return ; - -if (--shmem->pages_use_count >0 ) -return ; - +if (refcount_dec_and_test (&shmem->pages_use_count)) {#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT );#endif -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; + } }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);
@@ -551,8 +548,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)* mmap'd, vm_open() just grabs an additional reference for the new * mm the vma is getting copied into (ie. on fork()). */ -if (!drm_WARN_ON_ONCE (obj->dev, !shmem->pages_use_count)) - shmem->pages_use_count++; +drm_WARN_ON_ONCE (obj->dev, + !refcount_inc_not_zero (&shmem->pages_use_count));dma_resv_unlock (shmem->base.resv);
@@ -640,7 +637,7 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,return ;drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count)); -drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count); +drm_printf_indent (p, indent,"pages_use_count=%u\n" ,refcount_read (&shmem->pages_use_count));drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }
drivers/gpu/drm/lima/lima_gem.c +1 -1
@@ -47,7 +47,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) } bo->base.pages = pages; - bo->base.pages_use_count =1 ; +refcount_set (&bo->base.pages_use_count,1 );mapping_set_unevictable (mapping); }
drivers/gpu/drm/panfrost/panfrost_mmu.c +1 -1
@@ -487,7 +487,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, goto err_unlock; } bo->base.pages = pages; - bo->base.pages_use_count =1 ; +refcount_set (&bo->base.pages_use_count,1 ); }else { pages = bo->base.pages;if (pages[page_offset]) {
include/drm/drm_gem_shmem_helper.h +1 -1
@@ -37,7 +37,7 @@ struct drm_gem_shmem_object {* Reference count on the pages table. * The pages are put when the count reaches zero. */ -unsigned int pages_use_count; +refcount_t pages_use_count;/** * @pages_pin_count:
Signature · 1 line
2.41.0
Re: [PATCH v17 07/18] drm/shmem-helper: Use refcount_t for pages_use_count
Quoted 5 lines · On Fri, 15 Sep 2023 02:27:10 +0300 Dmitry Osipenko…
Use atomic
refcount_thelper forpages_use_countto optimize pin/unpin functions by skipping reservation locking while GEM's pin refcount > 1.Suggested-by Boris Brezillon <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>
Quoted 134 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 33 +++++++++++-------------- drivers/gpu/drm/lima/lima_gem.c | 2 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- include/drm/drm_gem_shmem_helper.h | 2 +- 4 files changed, 18 insertions(+), 21 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +15
-18@@ -155,7 +155,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (shmem->pages)drm_gem_shmem_put_pages_locked (shmem); -drm_WARN_ON (obj->dev, shmem->pages_use_count); +drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count));dma_resv_unlock (shmem->base.resv); }@@ -172,14 +172,13 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); -if (shmem->pages_use_count++ >0 ) +if (refcount_inc_not_zero (&shmem->pages_use_count))return 0 ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {drm_dbg_kms (obj->dev,"Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -195,6 +194,8 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages = pages; +refcount_set (&shmem->pages_use_count,1 ); +return 0 ; }@@ -210,21 +211,17 @@ void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv); -if (drm_WARN_ON_ONCE (obj->dev, !shmem->pages_use_count)) -return ; - -if (--shmem->pages_use_count >0 ) -return ; - +if (refcount_dec_and_test (&shmem->pages_use_count)) {#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT );#endif -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; + } }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);@@ -551,8 +548,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)* mmap'd, vm_open() just grabs an additional reference for the new * mm the vma is getting copied into (ie. on fork()). */ -if (!drm_WARN_ON_ONCE (obj->dev, !shmem->pages_use_count)) - shmem->pages_use_count++; +drm_WARN_ON_ONCE (obj->dev, + !refcount_inc_not_zero (&shmem->pages_use_count));dma_resv_unlock (shmem->base.resv);@@ -640,7 +637,7 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,return ;drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count)); -drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count); +drm_printf_indent (p, indent,"pages_use_count=%u\n" ,refcount_read (&shmem->pages_use_count));drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }drivers/gpu/drm/lima/lima_gem.c +1
-1@@ -47,7 +47,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) } bo->base.pages = pages; - bo->base.pages_use_count =1 ; +refcount_set (&bo->base.pages_use_count,1 );mapping_set_unevictable (mapping); }drivers/gpu/drm/panfrost/panfrost_mmu.c +1
-1@@ -487,7 +487,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, goto err_unlock; } bo->base.pages = pages; - bo->base.pages_use_count =1 ; +refcount_set (&bo->base.pages_use_count,1 ); }else { pages = bo->base.pages;if (pages[page_offset]) {include/drm/drm_gem_shmem_helper.h +1
-1@@ -37,7 +37,7 @@ struct drm_gem_shmem_object {* Reference count on the pages table. * The pages are put when the count reaches zero. */ -unsigned int pages_use_count; +refcount_t pages_use_count;/** * @pages_pin_count:
Dmitry Osipenko
maintainerPATCH v17 8/18 drm/shmem-helper: Add and use lockless drm_gem_shmem_get_pages()
In reply to Dmitry OsipenkoAdd lockless drm_gem_shmem_get_pages() helper that skips taking reservation lock if pages_use_count is non-zero, leveraging from atomicity of the refcount_t. Make drm_gem_shmem_mmap() to utilize the new helper.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +15 -4
@@ -225,6 +225,20 @@ void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked); +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int ret; + +if (refcount_inc_not_zero (&shmem->pages_use_count)) +return 0 ; + +dma_resv_lock (shmem->base.resv,NULL ); + ret =drm_gem_shmem_get_pages_locked (shmem); +dma_resv_unlock (shmem->base.resv); + +return ret; +} +static int drm_gem_shmem_pin_locked (struct drm_gem_shmem_object *shmem) {int ret;
@@ -608,10 +622,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structreturn ret; } -dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages_locked (shmem); -dma_resv_unlock (shmem->base.resv); - + ret =drm_gem_shmem_get_pages (shmem);if (ret)return ret;
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 9/18 drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin
In reply to Dmitry OsipenkoThe vmapped pages shall be pinned in memory and previously get/put_pages() were implicitly hard-pinning/unpinning the pages. This will no longer be the case with addition of memory shrinker because pages_use_count > 0 won't determine anymore whether pages are hard-pinned (they will be soft-pinned), while the new pages_pin_count will do the hard-pinning. Switch the vmap/vunmap() to use pin/unpin() functions in a preparation of addition of the memory shrinker support to drm-shmem.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 19 ++++++++++++------- include/drm/drm_gem_shmem_helper.h | 2 +- 2 files changed, 13 insertions(+), 8 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +12 -7
@@ -255,6 +255,14 @@ static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)return ret; } +static void drm_gem_shmem_unpin_locked (struct drm_gem_shmem_object *shmem) +{ +dma_resv_assert_held (shmem->base.resv); + +if (refcount_dec_and_test (&shmem->pages_pin_count)) +drm_gem_shmem_put_pages_locked (shmem); +} +/** * drm_gem_shmem_pin - Pin backing pages for a shmem GEM object * @shmem: shmem GEM object
@@ -302,10 +310,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)return ;dma_resv_lock (shmem->base.resv,NULL ); - -if (refcount_dec_and_test (&shmem->pages_pin_count)) -drm_gem_shmem_put_pages_locked (shmem); - +drm_gem_shmem_unpin_locked (shmem);dma_resv_unlock (shmem->base.resv); }EXPORT_SYMBOL_GPL (drm_gem_shmem_unpin);
@@ -343,7 +348,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages_locked (shmem); + ret =drm_gem_shmem_pin_locked (shmem);if (ret) goto err_zero_use;
@@ -366,7 +371,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages_locked (shmem); +drm_gem_shmem_unpin_locked (shmem); err_zero_use: shmem->vmap_use_count =0 ;
@@ -403,7 +408,7 @@ void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages_locked (shmem); +drm_gem_shmem_unpin_locked (shmem); } shmem->vaddr =NULL ;
include/drm/drm_gem_shmem_helper.h +1 -1
@@ -123,7 +123,7 @@ int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv);static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) {return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && + !refcount_read (&shmem->pages_pin_count) && shmem->sgt && !shmem->base.dma_buf && !shmem->base.import_attach; }
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 10/18 drm/shmem-helper: Use refcount_t for vmap_use_count
In reply to Dmitry OsipenkoUse refcount_t helper for vmap_use_count to make refcounting consistent with pages_use_count and pages_pin_count that use refcount_t. This also makes vmapping to benefit from the refcount_t's overflow checks.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 28 +++++++++++--------------- include/drm/drm_gem_shmem_helper.h | 2 +- 2 files changed, 13 insertions(+), 17 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +12 -16
@@ -144,7 +144,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); -drm_WARN_ON (obj->dev, shmem->vmap_use_count); +drm_WARN_ON (obj->dev,refcount_read (&shmem->vmap_use_count));if (shmem->sgt) { dma_unmap_sgtable(obj->dev->dev, shmem->sgt,
@@ -343,23 +343,25 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,dma_resv_assert_held (shmem->base.resv); -if (shmem->vmap_use_count++ >0 ) { +if (refcount_inc_not_zero (&shmem->vmap_use_count)) {iosys_map_set_vaddr (map, shmem->vaddr);return 0 ; } ret =drm_gem_shmem_pin_locked (shmem);if (ret) - goto err_zero_use; +return ret;if (shmem->map_wc) prot =pgprot_writecombine (prot); shmem->vaddr =vmap (shmem->pages, obj->size >>PAGE_SHIFT ,VM_MAP , prot); -if (!shmem->vaddr) +if (!shmem->vaddr) { ret = -ENOMEM ; -else + }else {iosys_map_set_vaddr (map, shmem->vaddr); +refcount_set (&shmem->vmap_use_count,1 ); + } }if (ret) {
@@ -372,8 +374,6 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach)drm_gem_shmem_unpin_locked (shmem); -err_zero_use: - shmem->vmap_use_count =0 ;return ret; }
@@ -401,14 +401,10 @@ void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, }else {dma_resv_assert_held (shmem->base.resv); -if (drm_WARN_ON_ONCE (obj->dev, !shmem->vmap_use_count)) -return ; - -if (--shmem->vmap_use_count >0 ) -return ; - -vunmap (shmem->vaddr); -drm_gem_shmem_unpin_locked (shmem); +if (refcount_dec_and_test (&shmem->vmap_use_count)) { +vunmap (shmem->vaddr); +drm_gem_shmem_unpin_locked (shmem); + } } shmem->vaddr =NULL ;
@@ -654,7 +650,7 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count));drm_printf_indent (p, indent,"pages_use_count=%u\n" ,refcount_read (&shmem->pages_use_count)); -drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"vmap_use_count=%u\n" ,refcount_read (&shmem->vmap_use_count));drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }EXPORT_SYMBOL_GPL (drm_gem_shmem_print_info);
include/drm/drm_gem_shmem_helper.h +1 -1
@@ -81,7 +81,7 @@ struct drm_gem_shmem_object {* Reference count on the virtual address. * The address are un-mapped when the count reaches zero. */ -unsigned int vmap_use_count; +refcount_t vmap_use_count;/** * @pages_mark_dirty_on_put:
Signature · 1 line
2.41.0
kernel test robot
build botRe: [PATCH v17 10/18] drm/shmem-helper: Use refcount_t for vmap_use_count
kernel test robot noticed "WARNING:at_drivers/gpu/drm/drm_fbdev_generic.c:#drm_fbdev_generic_helper_fb_dirty" on:
Hello,
kernel test robot noticed "WARNING:at_drivers/gpu/drm/drm_fbdev_generic.c:#drm_fbdev_generic_helper_fb_dirty" on:
commit: e5f31d1a2da5d43187a0e1fc4d1882982ab0d9b8 ("[PATCH v17 10/18] drm/shmem-helper: Use refcount_t for vmap_use_count")
url: https://github.com/intel-lab-lkp/linux/commits/Dmitry-Osipenko/drm-gem-Change-locked-unlocked-postfix-of-drm_gem_v-unmap-function-names/20230915-073105
base: git://anongit.freedesktop.org/drm/drm drm-next
patch link: https://lore.kernel.org/all/[email protected]/
patch subject: [PATCH v17 10/18] drm/shmem-helper: Use refcount_t for vmap_use_count
in testcase: rcuscale
version:
with following parameters:
runtime: 300s scale_type: tasks
compiler: gcc-11
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 23.102568][ T8] ------------[ cut here ]------------ [ 23.102576][ T8] platform vkms: Damage blitter failed: ret=-12 [ 23.102605][ T8] WARNING: CPU: 0 PID: 8 at drivers/gpu/drm/drm_fbdev_generic.c:226 drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102614][ T8] Modules linked in: [ 23.102619][ T8] CPU: 0 PID: 8 Comm: kworker/0:0 Tainted: G N 6.6.0-rc1-00010-ge5f31d1a2da5 #1 debbcfa26ac7594ab4e1fde44696b4efa3ea485a [ 23.102624][ T8] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 [ 23.102627][ T8] Workqueue: events drm_fb_helper_damage_work [ 23.102631][ T8] EIP: drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102633][ T8] Code: 05 a8 49 81 c2 01 8b 40 08 8b 58 2c 85 db 75 02 8b 18 89 55 e0 e8 90 7a 04 00 8b 55 e0 52 53 50 68 d4 41 28 c2 e8 74 68 a5 ff <0f> 0b 8b 55 e0 83 c4 10 e9 7b ff ff ff 8d 76 00 8b 45 e4 8b 55 e8 All code ======== 0: 05 a8 49 81 c2 add $0xc28149a8,%eax 5: 01 8b 40 08 8b 58 add %ecx,0x588b0840(%rbx) b: 2c 85 sub $0x85,%al d: db 75 02 (bad) 0x2(%rbp) 10: 8b 18 mov (%rax),%ebx 12: 89 55 e0 mov %edx,-0x20(%rbp) 15: e8 90 7a 04 00 call 0x47aaa 1a: 8b 55 e0 mov -0x20(%rbp),%edx 1d: 52 push %rdx 1e: 53 push %rbx 1f: 50 push %rax 20: 68 d4 41 28 c2 push $0xffffffffc22841d4 25: e8 74 68 a5 ff call 0xffffffffffa5689e 2a:* 0f 0b ud2 <-- trapping instruction 2c: 8b 55 e0 mov -0x20(%rbp),%edx 2f: 83 c4 10 add $0x10,%esp 32: e9 7b ff ff ff jmp 0xffffffffffffffb2 37: 8d 76 00 lea 0x0(%rsi),%esi 3a: 8b 45 e4 mov -0x1c(%rbp),%eax 3d: 8b 55 e8 mov -0x18(%rbp),%edx
Code starting with the faulting instruction
===========================================
0: 0f 0b ud2 2: 8b 55 e0 mov -0x20(%rbp),%edx 5: 83 c4 10 add $0x10,%esp 8: e9 7b ff ff ff jmp 0xffffffffffffff88 d: 8d 76 00 lea 0x0(%rsi),%esi 10: 8b 45 e4 mov -0x1c(%rbp),%eax 13: 8b 55 e8 mov -0x18(%rbp),%edx [ 23.102636][ T8] EAX: 00000000 EBX: ee027ab0 ECX: 00000000 EDX: 00000000 [ 23.102638][ T8] ESI: c624b600 EDI: c624b738 EBP: c3585ef8 ESP: c3585ec4 [ 23.102640][ T8] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 EFLAGS: 00010246 [ 23.102647][ T8] CR0: 80050033 CR2: ffd3a000 CR3: 0295e000 CR4: 00040690 [ 23.102649][ T8] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 23.102651][ T8] DR6: fffe0ff0 DR7: 00000400 [ 23.102653][ T8] Call Trace: [ 23.102656][ T8] ? show_regs (arch/x86/kernel/dumpstack.c:479 arch/x86/kernel/dumpstack.c:465) [ 23.102661][ T8] ? drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102664][ T8] ? __warn (kernel/panic.c:673) [ 23.102668][ T8] ? drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102670][ T8] ? report_bug (lib/bug.c:201 lib/bug.c:219) [ 23.102677][ T8] ? exc_overflow (arch/x86/kernel/traps.c:250) [ 23.102681][ T8] ? handle_bug (arch/x86/kernel/traps.c:216) [ 23.102685][ T8] ? exc_invalid_op (arch/x86/kernel/traps.c:258 (discriminator 1)) [ 23.102689][ T8] ? handle_exception (arch/x86/entry/entry_32.S:1056) [ 23.102701][ T8] ? exc_overflow (arch/x86/kernel/traps.c:250) [ 23.102704][ T8] ? drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102708][ T8] ? exc_overflow (arch/x86/kernel/traps.c:250) [ 23.102710][ T8] ? drm_fbdev_generic_helper_fb_dirty (drivers/gpu/drm/drm_fbdev_generic.c:226) [ 23.102715][ T8] ? trace_hardirqs_on (kernel/trace/trace_preemptirq.c:63) [ 23.102720][ T8] drm_fb_helper_damage_work (drivers/gpu/drm/drm_fb_helper.c:391 drivers/gpu/drm/drm_fb_helper.c:413) [ 23.102723][ T8] process_one_work (kernel/workqueue.c:2635) [ 23.102731][ T8] worker_thread (kernel/workqueue.c:2697 kernel/workqueue.c:2784) [ 23.102736][ T8] kthread (kernel/kthread.c:388) [ 23.102740][ T8] ? process_one_work (kernel/workqueue.c:2730) [ 23.102743][ T8] ? kthread_complete_and_exit (kernel/kthread.c:341) [ 23.102747][ T8] ret_from_fork (arch/x86/kernel/process.c:153) [ 23.102750][ T8] ? kthread_complete_and_exit (kernel/kthread.c:341) [ 23.102754][ T8] ret_from_fork_asm (arch/x86/entry/entry_32.S:741) [ 23.102757][ T8] entry_INT80_32 (arch/x86/entry/entry_32.S:947) [ 23.102765][ T8] irq event stamp: 249311 [ 23.102767][ T8] hardirqs last enabled at (249317): vprintk_emit (arch/x86/include/asm/irqflags.h:42 arch/x86/include/asm/irqflags.h:77 arch/x86/include/asm/irqflags.h:135 kernel/printk/printk.c:1961 kernel/printk/printk.c:2306) [ 23.102772][ T8] hardirqs last disabled at (249322): vprintk_emit (kernel/printk/printk.c:1940 kernel/printk/printk.c:2306) [ 23.102775][ T8] softirqs last enabled at (235498): __do_softirq (arch/x86/include/asm/preempt.h:27 kernel/softirq.c:400 kernel/softirq.c:582) [ 23.102778][ T8] softirqs last disabled at (235491): do_softirq_own_stack (arch/x86/kernel/irq_32.c:57 arch/x86/kernel/irq_32.c:147) [ 23.102781][ T8] ---[ end trace 0000000000000000 ]--- [ 23.104064][ T1] Console: switching to colour frame buffer device 128x48 [ 23.189012][ T1] platform vkms: [drm] fb0: vkmsdrmfb frame buffer device
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20230926/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Dmitry Osipenko
maintainerPATCH v17 11/18 drm/shmem-helper: Improve drm_gem_shmem_vmap_locked() error handling
In reply to Dmitry OsipenkoRemove error unwinding from drm_gem_shmem_vmap_locked() making error paths consistent for both dmabuf and non-dmabuf cases and improving readability of the code. Often it's preferred to do the explicit error unwinding, but this multi-path function is an exception.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +3 -10
@@ -334,7 +334,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base; -int ret =0 ; +int ret;if (obj->import_attach) { ret =dma_buf_vmap (obj->import_attach->dmabuf, map);
@@ -357,6 +357,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, shmem->vaddr =vmap (shmem->pages, obj->size >>PAGE_SHIFT ,VM_MAP , prot);if (!shmem->vaddr) { +drm_gem_shmem_unpin_locked (shmem); ret = -ENOMEM ; }else {iosys_map_set_vaddr (map, shmem->vaddr);
@@ -364,16 +365,8 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, } } -if (ret) { +if (ret)drm_dbg_kms (obj->dev,"Failed to vmap pages, error %d\n" , ret); - goto err_put_pages; - } - -return 0 ; - -err_put_pages: -if (!obj->import_attach) -drm_gem_shmem_unpin_locked (shmem);return ret; }
Signature · 1 line
2.41.0
Boris Brezillon
maintainerRe: [PATCH v17 11/18] drm/shmem-helper: Improve drm_gem_shmem_vmap_locked() error handling
Quoted 6 lines · On Fri, 15 Sep 2023 02:27:14 +0300 Dmitry Osipenko…
Remove error unwinding from
drm_gem_shmem_vmap_locked()making error paths consistent for both dmabuf and non-dmabuf cases and improving readability of the code. Often it's preferred to do the explicit error unwinding, but this multi-path function is an exception.Suggested-by Boris Brezillon <[email protected]>
Actually, I suggested returning directly, but I hadn't noticed the debug message which you pointed in one of your replies. No longer sure it's a good idea to drop the error path, since it makes code addition more error-prone (pretty easy to forget the drm_gem_shmem_unpin_locked() in new error paths).
Quoted 44 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +3
-10@@ -334,7 +334,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base; -int ret =0 ; +int ret;if (obj->import_attach) { ret =dma_buf_vmap (obj->import_attach->dmabuf, map);@@ -357,6 +357,7 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, shmem->vaddr =vmap (shmem->pages, obj->size >>PAGE_SHIFT ,VM_MAP , prot);if (!shmem->vaddr) { +drm_gem_shmem_unpin_locked (shmem); ret = -ENOMEM ; }else {iosys_map_set_vaddr (map, shmem->vaddr);@@ -364,16 +365,8 @@ int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, } } -if (ret) { +if (ret)drm_dbg_kms (obj->dev,"Failed to vmap pages, error %d\n" , ret); - goto err_put_pages; - } - -return 0 ; - -err_put_pages: -if (!obj->import_attach) -drm_gem_shmem_unpin_locked (shmem);return ret; }
Dmitry Osipenko
maintainerPATCH v17 12/18 drm/shmem-helper: Prepare drm_gem_shmem_free() to shrinker addition
In reply to Dmitry OsipenkoPrepare drm_gem_shmem_free() to addition of memory shrinker support to drm-shmem by adding and using variant of put_pages() that doesn't touch reservation lock. Reservation shouldn't be touched because lockdep will trigger a bogus warning about locking contention with fs_reclaim code paths that can't happen during the time when GEM is freed and lockdep doesn't know about that.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 55 +++++++++++++++++--------- 1 file changed, 37 insertions(+), 18 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +37 -18
@@ -128,6 +128,41 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static void +__drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} + +static void +__drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) +{ +/* +* Destroying the object is a special case. Acquiring the obj +* lock in drm_gem_shmem_put_pages_locked() can cause a locking +* order inversion between reservation_ww_class_mutex and fs_reclaim +* when called from drm_gem_shmem_free(). +* +* This deadlock is not actually possible, because no one should +* be already holding the lock when drm_gem_shmem_free() is called. +* Unfortunately lockdep is not aware of this detail. So when the +* refcount drops to zero, make sure that the reservation lock +* isn't touched here. +*/ +if (refcount_dec_and_test (&shmem->pages_use_count)) +__drm_gem_shmem_release_pages (shmem); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free
@@ -142,8 +177,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { -dma_resv_lock (shmem->base.resv,NULL ); -drm_WARN_ON (obj->dev,refcount_read (&shmem->vmap_use_count));if (shmem->sgt) {
@@ -153,11 +186,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); }if (shmem->pages) -drm_gem_shmem_put_pages_locked (shmem); +__drm_gem_shmem_put_pages (shmem);drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count)); - -dma_resv_unlock (shmem->base.resv); }drm_gem_object_release (obj);
@@ -207,21 +238,9 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv); -if (refcount_dec_and_test (&shmem->pages_use_count)) { -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif - -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; - } +__drm_gem_shmem_put_pages (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);
Signature · 1 line
2.41.0
Boris Brezillon
maintainerRe: [PATCH v17 12/18] drm/shmem-helper: Prepare drm_gem_shmem_free() to shrinker addition
Quoted 17 lines · On Fri, 15 Sep 2023 02:27:15 +0300 Dmitry Osipenko…
Prepare
drm_gem_shmem_free()to addition of memory shrinker support to drm-shmem by adding and using variant ofput_pages()that doesn't touch reservation lock. Reservation shouldn't be touched because lockdep will trigger a bogus warning about locking contention withfs_reclaimcode paths that can't happen during the time when GEM is freed and lockdep doesn't know about that.Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 55 +++++++++++++++++--------- 1 file changed, 37 insertions(+), 18 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +0
-0
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 132
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-0@@ -128,6 +128,41 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static void +__drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem)
Could we find more descriptive names to replace those __ prefixes? drm_gem_shmem_free_pages(), drm_gem_shmem_drop_pages()?
Quoted 6 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +6
-0+{ +struct drm_gem_object *obj = &shmem->base; + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT );
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 148
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} + +static void +__drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem)
Maybe drm_gem_shmem_put_pages_no_lock_check()? But honestly, I'm not sure we want to make it a function since it's only going to be needed in drm_gem_shmem_free(). I think we can just inline
if (refcount_dec_and_test(&shmem->pages_use_count)) __drm_gem_shmem_release_pages(shmem);
there.
Quoted 65 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +19
-18+{ +/* +* Destroying the object is a special case. Acquiring the obj +* lock in drm_gem_shmem_put_pages_locked() can cause a locking +* order inversion between reservation_ww_class_mutex and fs_reclaim +* when called from drm_gem_shmem_free(). +* +* This deadlock is not actually possible, because no one should +* be already holding the lock when drm_gem_shmem_free() is called. +* Unfortunately lockdep is not aware of this detail. So when the +* refcount drops to zero, make sure that the reservation lock +* isn't touched here. +*/ +if (refcount_dec_and_test (&shmem->pages_use_count)) +__drm_gem_shmem_release_pages (shmem); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,8 +177,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { -dma_resv_lock (shmem->base.resv,NULL ); -drm_WARN_ON (obj->dev,refcount_read (&shmem->vmap_use_count));if (shmem->sgt) {@@ -153,11 +186,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); }if (shmem->pages) -drm_gem_shmem_put_pages_locked (shmem); +__drm_gem_shmem_put_pages (shmem);drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count)); - -dma_resv_unlock (shmem->base.resv); }drm_gem_object_release (obj);@@ -207,21 +238,9 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv); -if (refcount_dec_and_test (&shmem->pages_use_count)) { -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif - -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; - } +__drm_gem_shmem_put_pages (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);
Dmitry Osipenko
maintainerPATCH v17 13/18 drm/shmem-helper: Add memory shrinker
In reply to Dmitry OsipenkoIntroduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement evict() callback of GEM object where driver should check whether object is purgeable or evictable using drm-shmem helpers and perform the shrinking action
2. Initialize drm-shmem internals using drmm_gem_shmem_init(drm_device), which will register drm-shmem shrinker
3. Implement madvise IOCTL that will use drm_gem_shmem_madvise()
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 390 +++++++++++++++++- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- include/drm/drm_device.h | 10 +- include/drm/drm_gem_shmem_helper.h | 71 +++- 4 files changed, 459 insertions(+), 21 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +376 -14
@@ -20,6 +20,7 @@#include <drm/drm_device.h> #include <drm/drm_drv.h> #include <drm/drm_gem_shmem_helper.h> +#include <drm/drm_managed.h> #include <drm/drm_prime.h> #include <drm/drm_print.h>
@@ -88,8 +89,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)if (ret) goto err_release; -INIT_LIST_HEAD (&shmem->madv_list); -if (!private) {/* * Our buffers are kept pinned, so allocating them
@@ -128,11 +127,51 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_assert_held (shmem->base.resv); + +return (shmem->madv >=0 ) && shmem->base.funcs->evict && +refcount_read (&shmem->pages_use_count) && + !refcount_read (&shmem->pages_pin_count) && + !shmem->base.dma_buf && !shmem->base.import_attach && + shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem *shmem_mm = obj->dev->shmem_mm; +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!shmem_shrinker || obj->import_attach) +return ; + +if (shmem->madv <0 ) +drm_gem_lru_remove (&shmem->base); +else if (drm_gem_shmem_is_evictable (shmem) ||drm_gem_shmem_is_purgeable (shmem)) +drm_gem_lru_move_tail (&shmem_shrinker->lru_evictable, &shmem->base); +else if (shmem->evicted) +drm_gem_lru_move_tail (&shmem_shrinker->lru_evicted, &shmem->base); +else if (!shmem->pages) +drm_gem_lru_remove (&shmem->base); +else +drm_gem_lru_move_tail (&shmem_shrinker->lru_pinned, &shmem->base); +} +static void __drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; +if (!shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted && shmem->madv >=0 ); +return ; + } +#ifdef CONFIG_X86 if (shmem->map_wc)set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT );
@@ -185,7 +224,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (refcount_read (&shmem->pages_use_count))__drm_gem_shmem_put_pages (shmem);drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count));
@@ -196,15 +235,26 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem,bool init) {struct drm_gem_object *obj = &shmem->base;struct page **pages;dma_resv_assert_held (shmem->base.resv); -if (refcount_inc_not_zero (&shmem->pages_use_count)) +if (shmem->madv <0 ) { +drm_WARN_ON (obj->dev, shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted);return 0 ; + } + +if (drm_WARN_ON (obj->dev, !(init ^refcount_read (&shmem->pages_use_count)))) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {
@@ -225,8 +275,29 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages = pages; +return 0 ; +} + +static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (refcount_inc_not_zero (&shmem->pages_use_count)) +return 0 ; + + err =drm_gem_shmem_acquire_pages (shmem, true); +if (err) +return err; +refcount_set (&shmem->pages_use_count,1 ); +drm_gem_shmem_update_pages_state_locked (shmem); +return 0 ; }
@@ -241,6 +312,8 @@ void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv);__drm_gem_shmem_put_pages (shmem); + +drm_gem_shmem_update_pages_state_locked (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);
@@ -268,8 +341,15 @@ static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)return 0 ; ret =drm_gem_shmem_get_pages_locked (shmem); -if (!ret) +if (!ret) { + ret =drm_gem_shmem_swapin_locked (shmem); +if (ret) { +drm_gem_shmem_put_pages_locked (shmem); +return ret; + } +refcount_set (&shmem->pages_pin_count,1 ); + }return ret; }
@@ -458,29 +538,54 @@ int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state_locked (shmem); +return (madv >=0 ); }EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise_locked); -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) +{ +struct drm_gem_object *obj = &shmem->base; +int ret; + + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret) +return ret; + + ret =drm_gem_shmem_madvise_locked (shmem, madv); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise); + +static void drm_gem_shmem_unpin_pages_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +__drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); +sg_free_table (shmem->sgt);kfree (shmem->sgt); shmem->sgt =NULL ; +} -drm_gem_shmem_put_pages_locked (shmem); +void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; - shmem->madv =-1 ; +drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); +drm_gem_shmem_unpin_pages_locked (shmem);drm_gem_free_mmap_offset (obj);/* Our goal here is to return as much of the memory as
@@ -491,9 +596,59 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 );invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state_locked (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_purge_locked); +/** +* drm_gem_shmem_swapin_locked() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swapin_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct sg_table *sgt; +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem, false); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } + + shmem->sgt = sgt; + shmem->evicted = false; + +drm_gem_shmem_update_pages_state_locked (shmem); + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swapin_locked); +/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object * @file: DRM file structure to create the dumb buffer for
@@ -540,22 +695,38 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -drm_WARN_ON_ONCE (obj->dev, !shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check whether we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || + !refcount_read (&shmem->pages_use_count)); +drm_WARN_ON_ONCE (obj->dev, !shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swapin_locked (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + +/* +* shmem->pages is guaranteed to be valid while reservation +* lock is held and drm_gem_shmem_swapin_locked() succeeds. +*/ page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;
@@ -578,6 +749,7 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)drm_WARN_ON_ONCE (obj->dev, !refcount_inc_not_zero (&shmem->pages_use_count)); +drm_gem_shmem_update_pages_state_locked (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);
@@ -663,7 +835,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count));drm_printf_indent (p, indent,"pages_use_count=%u\n" ,refcount_read (&shmem->pages_use_count));drm_printf_indent (p, indent,"vmap_use_count=%u\n" ,refcount_read (&shmem->vmap_use_count)); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL_GPL (drm_gem_shmem_print_info);
@@ -718,6 +892,8 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_ shmem->sgt = sgt; +drm_gem_shmem_update_pages_state_locked (shmem); +return sgt; err_free_sgt:
@@ -794,6 +970,192 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_gem_shmem_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = +to_drm_gem_shmem_shrinker (shrinker); +unsigned long count = shmem_shrinker->lru_evictable.count; + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +void drm_gem_shmem_evict_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +drm_WARN_ON (obj->dev, !drm_gem_shmem_is_evictable (shmem)); +drm_WARN_ON (obj->dev, shmem->evicted); + +drm_gem_shmem_unpin_pages_locked (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state_locked (shmem); +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict_locked); + +static bool drm_gem_shmem_shrinker_evict_locked (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < obj->size >>PAGE_SHIFT ) +return false; + + err =drm_gem_evict_locked (obj); +if (err) +return false; + +return true; +} + +static bool drm_gem_shmem_shrinker_purge_locked (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_purgeable (shmem)) +return false; + + err =drm_gem_evict_locked (obj); +if (err) +return false; + +return true; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker; +unsigned long nr_to_scan = sc->nr_to_scan; +unsigned long remaining =0 ; +unsigned long freed =0 ; + + shmem_shrinker =to_drm_gem_shmem_shrinker (shrinker); + +/* purge as many objects as we can */ + freed +=drm_gem_lru_scan (&shmem_shrinker->lru_evictable, + nr_to_scan, &remaining, + drm_gem_shmem_shrinker_purge_locked); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_lru_scan (&shmem_shrinker->lru_evictable, + nr_to_scan - freed, &remaining, + drm_gem_shmem_shrinker_evict_locked); + +return (freed >0 && remaining >0 ) ? freed :SHRINK_STOP ; +} + +static int drm_gem_shmem_shrinker_init (struct drm_gem_shmem *shmem_mm, +const char *shrinker_name) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; +int err; + + shmem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + shmem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + shmem_shrinker->base.seeks =DEFAULT_SEEKS ; + +mutex_init (&shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_evictable, &shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_evicted, &shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_pinned, &shmem_shrinker->lock); + + err =register_shrinker (&shmem_shrinker->base, shrinker_name); +if (err) { +mutex_destroy (&shmem_shrinker->lock); +return err; + } + +return 0 ; +} + +static void drm_gem_shmem_shrinker_release (struct drm_device *dev, +struct drm_gem_shmem *shmem_mm) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; + +unregister_shrinker (&shmem_shrinker->base); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_evictable.list)); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_evicted.list)); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_pinned.list)); +mutex_destroy (&shmem_shrinker->lock); +} + +static int drm_gem_shmem_init (struct drm_device *dev) +{ +int err; + +if (drm_WARN_ON (dev, dev->shmem_mm)) +return -EBUSY ; + + dev->shmem_mm =kzalloc (sizeof (*dev->shmem_mm),GFP_KERNEL ); +if (!dev->shmem_mm) +return -ENOMEM ; + + err =drm_gem_shmem_shrinker_init (dev->shmem_mm, dev->unique); +if (err) + goto free_gem_shmem; + +return 0 ; + +free_gem_shmem: +kfree (dev->shmem_mm); + dev->shmem_mm =NULL ; + +return err; +} + +static void drm_gem_shmem_release (struct drm_device *dev,void *ptr) +{ +struct drm_gem_shmem *shmem_mm = dev->shmem_mm; + +drm_gem_shmem_shrinker_release (dev, shmem_mm); + dev->shmem_mm =NULL ; +kfree (shmem_mm); +} + +/** +* drmm_gem_shmem_init() - Initialize drm-shmem internals +* @dev: DRM device +* +* Cleanup is automatically managed as part of DRM device releasing. +* Calling this function multiple times will result in a error. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drmm_gem_shmem_init (struct drm_device *dev) +{ +int err; + + err =drm_gem_shmem_init (dev); +if (err) +return err; + + err =drmm_add_action_or_reset (dev, drm_gem_shmem_release,NULL ); +if (err) +return err; + +return 0 ; +} +EXPORT_SYMBOL_GPL (drmm_gem_shmem_init); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" ); +
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8 -1
@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !refcount_read (&shmem->pages_pin_count) && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {
@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }
include/drm/drm_device.h +8 -2
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;
@@ -290,8 +291,13 @@ struct drm_device {/** @vma_offset_manager: GEM information */ struct drm_vma_offset_manager *vma_offset_manager; -/** @vram_mm: VRAM MM memory manager */ -struct drm_vram_mm *vram_mm; +union { +/** @vram_mm: VRAM MM memory manager */ +struct drm_vram_mm *vram_mm; + +/** @shmem_mm: SHMEM GEM memory manager */ +struct drm_gem_shmem *shmem_mm; + };/** * @switch_power_state:
include/drm/drm_gem_shmem_helper.h +67 -4
113 lines not on this page · raw message
Signature · 1 line
2.41.0
Boris Brezillon
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 50 lines · On Fri, 15 Sep 2023 02:27:16 +0300 Dmitry Osipenko…
Introduce common drm-shmem shrinker for DRM drivers.
To start using drm-shmem shrinker drivers should do the following:
1. Implement
evict()callback of GEM object where driver should check whether object is purgeable or evictable using drm-shmem helpers and perform the shrinking action2. Initialize drm-shmem internals using
drmm_gem_shmem_init(drm_device), which will register drm-shmem shrinker3. Implement madvise IOCTL that will use
drm_gem_shmem_madvise()Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 390 +++++++++++++++++- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- include/drm/drm_device.h | 10 +- include/drm/drm_gem_shmem_helper.h | 71 +++- 4 files changed, 459 insertions(+), 21 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +4
-2@@ -20,6 +20,7 @@#include <drm/drm_device.h> #include <drm/drm_drv.h> #include <drm/drm_gem_shmem_helper.h> +#include <drm/drm_managed.h> #include <drm/drm_prime.h> #include <drm/drm_print.h> @@ -88,8 +89,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)if (ret) goto err_release; -INIT_LIST_HEAD (&shmem->madv_list); -if (!private) {/* * Our buffers are kept pinned, so allocating them @@ -128,11 +127,51 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_assert_held (shmem->base.resv);
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 142
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+ +return (shmem->madv >=0 ) && shmem->base.funcs->evict && +refcount_read (&shmem->pages_use_count) && + !refcount_read (&shmem->pages_pin_count) && + !shmem->base.dma_buf && !shmem->base.import_attach && + shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state_locked (struct drm_gem_shmem_object *shmem)
Nit: it's not exactly the pages state you update
here, more the reclaimable state,
so maybe drm_gem_shmem_update_reclaimable_state_locked() or simply drm_gem_shmem_update_lru_locked()?
Quoted 36 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +27
-0+{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem *shmem_mm = obj->dev->shmem_mm; +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!shmem_shrinker || obj->import_attach) +return ; + +if (shmem->madv <0 ) +drm_gem_lru_remove (&shmem->base); +else if (drm_gem_shmem_is_evictable (shmem) ||drm_gem_shmem_is_purgeable (shmem)) +drm_gem_lru_move_tail (&shmem_shrinker->lru_evictable, &shmem->base); +else if (shmem->evicted) +drm_gem_lru_move_tail (&shmem_shrinker->lru_evicted, &shmem->base); +else if (!shmem->pages) +drm_gem_lru_remove (&shmem->base); +else +drm_gem_lru_move_tail (&shmem_shrinker->lru_pinned, &shmem->base); +} +static void __drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; +if (!shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted && shmem->madv >=0 ); +return ; + } +#ifdef CONFIG_X86 if (shmem->map_wc)set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT );
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 228
Quoted 6 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-1@@ -185,7 +224,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (refcount_read (&shmem->pages_use_count))__drm_gem_shmem_put_pages (shmem);
If you drop the implicit pages_use_count ref every time the sgt is destroyed, you can move the __drm_gem_shmem_put_pages(shmem) call to the if(shmem->sgt != NULL) branch, which makes a lot more sense.
Quoted 18 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +4
-2drm_WARN_ON (obj->dev,refcount_read (&shmem->pages_use_count));@@ -196,15 +235,26 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem,bool init) {struct drm_gem_object *obj = &shmem->base;struct page **pages;dma_resv_assert_held (shmem->base.resv); -if (refcount_inc_not_zero (&shmem->pages_use_count)) +if (shmem->madv <0 ) { +drm_WARN_ON (obj->dev, shmem->pages);
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 257
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +9
-0+return -ENOMEM ; + } + +if (shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted);return 0 ; + } + +if (drm_WARN_ON (obj->dev, !(init ^refcount_read (&shmem->pages_use_count)))) +return -EINVAL ;
OOC, why do we care? Is there any difference between initial and re-pin that make the page allocation impossible? Feels like, if there's a check to do, it should be done in the caller instead, and you can drop the init param here.
Quoted 77 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +40
-2pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {@@ -225,8 +275,29 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages = pages; +return 0 ; +} + +static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (refcount_inc_not_zero (&shmem->pages_use_count)) +return 0 ; + + err =drm_gem_shmem_acquire_pages (shmem, true); +if (err) +return err; +refcount_set (&shmem->pages_use_count,1 ); +drm_gem_shmem_update_pages_state_locked (shmem); +return 0 ; }@@ -241,6 +312,8 @@ void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)dma_resv_assert_held (shmem->base.resv);__drm_gem_shmem_put_pages (shmem); + +drm_gem_shmem_update_pages_state_locked (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_put_pages_locked);@@ -268,8 +341,15 @@ static int drm_gem_shmem_pin_locked(struct drm_gem_shmem_object *shmem)return 0 ; ret =drm_gem_shmem_get_pages_locked (shmem); -if (!ret) +if (!ret) { + ret =drm_gem_shmem_swapin_locked (shmem); +if (ret) { +drm_gem_shmem_put_pages_locked (shmem); +return ret; + } +refcount_set (&shmem->pages_pin_count,1 ); + }return ret; }@@ -458,29 +538,54 @@ int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state_locked (shmem); +return (madv >=0 ); }EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise_locked); -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) +{ +struct drm_gem_object *obj = &shmem->base; +int ret; + + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret)
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 563
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+return ret; + + ret =drm_gem_shmem_madvise_locked (shmem, madv); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_madvise); + +static void drm_gem_shmem_unpin_pages_locked (struct drm_gem_shmem_object *shmem)
The naming becomes quite confusing, with drm_gem_shmem_unpin_locked() and drm_gem_shmem_unpin_pages_locked(). By the look of it, it seems to do exactly the opposite of drm_gem_shmem_swapin_locked(), except for the missing ->evicted = true, which we can move here anyway, given drm_gem_shmem_purge_locked() explicitly set it to false anyway. The other thing that's missing is the drm_gem_shmem_update_pages_state_locked(), but it can also be moved there I think, if the the ->madv update happens before the drm_gem_shmem_unpin_pages_locked() call in drm_gem_shmem_purge_locked().
So, how about renaming this function drm_gem_shmem_swapout_locked()?
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 573
Quoted 11 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1{struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );
Are we sure we'll always have sgt != NULL? IIRC, if the GEM is only mmap-ed in userspace, get_sgt() is not necessarily called by the driver (needed to map in GPU space), and we have a potential NULL deref here. Maybe that changed at some point in the series, and sgt is unconditionally populated when get_pages() is called now.
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 574
Quoted 1 line
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-0+__drm_gem_shmem_release_pages (shmem);
Make sure you drop the implicit pages_use_count ref the sgt had, this way you can still tie the necessity to drop the pages to sgt != NULL in drm_gem_shmem_free().
Quoted 41 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +22
-3+drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); +sg_free_table (shmem->sgt);kfree (shmem->sgt); shmem->sgt =NULL ; +} -drm_gem_shmem_put_pages_locked (shmem); +void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; - shmem->madv =-1 ; +drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); +drm_gem_shmem_unpin_pages_locked (shmem);drm_gem_free_mmap_offset (obj);/* Our goal here is to return as much of the memory as @@ -491,9 +596,59 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 );invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state_locked (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_purge_locked); +/** +* drm_gem_shmem_swapin_locked() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure.
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem) · line 625
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+*/ +int drm_gem_shmem_swapin_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct sg_table *sgt; +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->evicted) {
Nit:
if (!shmem->evicted) return 0;
Quoted 377 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +236
-3+ err =drm_gem_shmem_acquire_pages (shmem, false); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } + + shmem->sgt = sgt; + shmem->evicted = false; + +drm_gem_shmem_update_pages_state_locked (shmem); + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swapin_locked); +/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object * @file: DRM file structure to create the dumb buffer for @@ -540,22 +695,38 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -drm_WARN_ON_ONCE (obj->dev, !shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check whether we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || + !refcount_read (&shmem->pages_use_count)); +drm_WARN_ON_ONCE (obj->dev, !shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swapin_locked (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + +/* +* shmem->pages is guaranteed to be valid while reservation +* lock is held and drm_gem_shmem_swapin_locked() succeeds. +*/ page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -578,6 +749,7 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)drm_WARN_ON_ONCE (obj->dev, !refcount_inc_not_zero (&shmem->pages_use_count)); +drm_gem_shmem_update_pages_state_locked (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -663,7 +835,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"pages_pin_count=%u\n" ,refcount_read (&shmem->pages_pin_count));drm_printf_indent (p, indent,"pages_use_count=%u\n" ,refcount_read (&shmem->pages_use_count));drm_printf_indent (p, indent,"vmap_use_count=%u\n" ,refcount_read (&shmem->vmap_use_count)); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL_GPL (drm_gem_shmem_print_info);@@ -718,6 +892,8 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_ shmem->sgt = sgt; +drm_gem_shmem_update_pages_state_locked (shmem); +return sgt; err_free_sgt:@@ -794,6 +970,192 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_gem_shmem_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = +to_drm_gem_shmem_shrinker (shrinker); +unsigned long count = shmem_shrinker->lru_evictable.count; + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +void drm_gem_shmem_evict_locked (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +drm_WARN_ON (obj->dev, !drm_gem_shmem_is_evictable (shmem)); +drm_WARN_ON (obj->dev, shmem->evicted); + +drm_gem_shmem_unpin_pages_locked (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state_locked (shmem); +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict_locked); + +static bool drm_gem_shmem_shrinker_evict_locked (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < obj->size >>PAGE_SHIFT ) +return false; + + err =drm_gem_evict_locked (obj); +if (err) +return false; + +return true; +} + +static bool drm_gem_shmem_shrinker_purge_locked (struct drm_gem_object *obj) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +int err; + +if (!drm_gem_shmem_is_purgeable (shmem)) +return false; + + err =drm_gem_evict_locked (obj); +if (err) +return false; + +return true; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker; +unsigned long nr_to_scan = sc->nr_to_scan; +unsigned long remaining =0 ; +unsigned long freed =0 ; + + shmem_shrinker =to_drm_gem_shmem_shrinker (shrinker); + +/* purge as many objects as we can */ + freed +=drm_gem_lru_scan (&shmem_shrinker->lru_evictable, + nr_to_scan, &remaining, + drm_gem_shmem_shrinker_purge_locked); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_lru_scan (&shmem_shrinker->lru_evictable, + nr_to_scan - freed, &remaining, + drm_gem_shmem_shrinker_evict_locked); + +return (freed >0 && remaining >0 ) ? freed :SHRINK_STOP ; +} + +static int drm_gem_shmem_shrinker_init (struct drm_gem_shmem *shmem_mm, +const char *shrinker_name) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; +int err; + + shmem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + shmem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + shmem_shrinker->base.seeks =DEFAULT_SEEKS ; + +mutex_init (&shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_evictable, &shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_evicted, &shmem_shrinker->lock); +drm_gem_lru_init (&shmem_shrinker->lru_pinned, &shmem_shrinker->lock); + + err =register_shrinker (&shmem_shrinker->base, shrinker_name); +if (err) { +mutex_destroy (&shmem_shrinker->lock); +return err; + } + +return 0 ; +} + +static void drm_gem_shmem_shrinker_release (struct drm_device *dev, +struct drm_gem_shmem *shmem_mm) +{ +struct drm_gem_shmem_shrinker *shmem_shrinker = &shmem_mm->shrinker; + +unregister_shrinker (&shmem_shrinker->base); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_evictable.list)); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_evicted.list)); +drm_WARN_ON (dev, !list_empty (&shmem_shrinker->lru_pinned.list)); +mutex_destroy (&shmem_shrinker->lock); +} + +static int drm_gem_shmem_init (struct drm_device *dev) +{ +int err; + +if (drm_WARN_ON (dev, dev->shmem_mm)) +return -EBUSY ; + + dev->shmem_mm =kzalloc (sizeof (*dev->shmem_mm),GFP_KERNEL ); +if (!dev->shmem_mm) +return -ENOMEM ; + + err =drm_gem_shmem_shrinker_init (dev->shmem_mm, dev->unique); +if (err) + goto free_gem_shmem; + +return 0 ; + +free_gem_shmem: +kfree (dev->shmem_mm); + dev->shmem_mm =NULL ; + +return err; +} + +static void drm_gem_shmem_release (struct drm_device *dev,void *ptr) +{ +struct drm_gem_shmem *shmem_mm = dev->shmem_mm; + +drm_gem_shmem_shrinker_release (dev, shmem_mm); + dev->shmem_mm =NULL ; +kfree (shmem_mm); +} + +/** +* drmm_gem_shmem_init() - Initialize drm-shmem internals +* @dev: DRM device +* +* Cleanup is automatically managed as part of DRM device releasing. +* Calling this function multiple times will result in a error. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drmm_gem_shmem_init (struct drm_device *dev) +{ +int err; + + err =drm_gem_shmem_init (dev); +if (err) +return err; + + err =drmm_add_action_or_reset (dev, drm_gem_shmem_release,NULL ); +if (err) +return err; + +return 0 ; +} +EXPORT_SYMBOL_GPL (drmm_gem_shmem_init); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" ); +drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !refcount_read (&shmem->pages_pin_count) && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }include/drm/drm_device.h +8
-2@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -290,8 +291,13 @@ struct drm_device {/** @vma_offset_manager: GEM information */ struct drm_vma_offset_manager *vma_offset_manager; -/** @vram_mm: VRAM MM memory manager */ -struct drm_vram_mm *vram_mm; +union { +/** @vram_mm: VRAM MM memory manager */ +struct drm_vram_mm *vram_mm; + +/** @shmem_mm: SHMEM GEM memory manager */ +struct drm_gem_shmem *shmem_mm; + };/** * @switch_power_state: include/drm/drm_gem_shmem_helper.h +3
-1@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -13,6 +14,7 @@#include <drm/drm_prime.h> struct dma_buf_attachment ; +struct drm_device ;struct drm_mode_create_dumb ;struct drm_printer ;struct sg_table ;@@ -53,8 +55,8 @@ struct drm_gem_shmem_object {* @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;
include/drm/drm_gem_shmem_helper.h struct drm_gem_shmem_object { · line 109
Quoted 7 lines
include/drm/drm_gem_shmem_helper.h +4
-0@@ -101,6 +103,12 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc : 1; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker.
Can we add a few words explaining how purged (madv < 0) and evicted are different. AFAIU, purging is done after BOs have been explicitly flagged as purgeable by the driver, with no way to re-populate the BO after it's been purged, while eviction is about all automatic BO swap{out,in} on memory pressure.
Quoted 85 lines
include/drm/drm_gem_shmem_helper.h +60
-3+*/ + bool evicted : 1; };#define to_drm_gem_shmem_obj (obj) \@@ -119,14 +127,22 @@ void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,int drm_gem_shmem_mmap (struct drm_gem_shmem_object *shmem,struct vm_area_struct *vma);int drm_gem_shmem_madvise_locked (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv);static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !refcount_read (&shmem->pages_pin_count) && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +dma_resv_assert_held (shmem->base.resv); + +return (shmem->madv >0 ) && shmem->base.funcs->evict && +refcount_read (&shmem->pages_use_count) && + !refcount_read (&shmem->pages_pin_count) && + !shmem->base.dma_buf && !shmem->base.import_attach && + (shmem->sgt || shmem->evicted); } +int drm_gem_shmem_swapin_locked (struct drm_gem_shmem_object *shmem); + +void drm_gem_shmem_evict_locked (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);@@ -270,6 +286,53 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* drm_gem_shmem_object_madvise - unlocked GEM object function for drm_gem_shmem_madvise_locked() +* @obj: GEM object +* @madv: Madvise value +* +* This function wraps drm_gem_shmem_madvise_locked(), providing unlocked variant. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +static inline int drm_gem_shmem_object_madvise (struct drm_gem_object *obj,int madv) +{ +struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); + +return drm_gem_shmem_madvise (shmem, madv); +} + +/** +* struct drm_gem_shmem_shrinker - Memory shrinker of GEM shmem memory manager +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct drm_gem_lru lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct drm_gem_lru lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct drm_gem_lru lru_evicted; +}; + +/** +* struct drm_gem_shmem - GEM shmem memory manager +*/ +struct drm_gem_shmem { +/** @shrinker: GEM shmem shrinker */ +struct drm_gem_shmem_shrinker shrinker; +}; + +int drmm_gem_shmem_init (struct drm_device *dev); +/* * Driver ops */
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 11 lines · Boris Brezillon
The naming becomes quite confusing, with
drm_gem_shmem_unpin_locked()anddrm_gem_shmem_unpin_pages_locked(). By the look of it, it seems to do exactly the opposite ofdrm_gem_shmem_swapin_locked(), except for the missing ->evicted = true, which we can move here anyway, givendrm_gem_shmem_purge_locked()explicitly set it to false anyway. The other thing that's missing is thedrm_gem_shmem_update_pages_state_locked(), but it can also be moved there I think, if the the ->madv update happens before thedrm_gem_shmem_unpin_pages_locked()call indrm_gem_shmem_purge_locked().
Quoted 1 line
So, how about renaming this function
drm_gem_shmem_swapout_locked()?
The swapout name would be misleading to me because pages aren't moved to swap, but allowed to be moved. I'll rename it to drm_gem_shmem_shrinker_unpin_locked().
Quoted 10 lines
Quoted 10 lines
{
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;
Quoted 6 lines
Quoted 1 line
dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);Are we sure we'll always have sgt != NULL? IIRC, if the GEM is only mmap-ed in userspace,
get_sgt()is not necessarily called by the driver (needed to map in GPU space), and we have a potential NULL deref here. Maybe that changed at some point in the series, and sgt is unconditionally populated whenget_pages()is called now.
The sgt is always set in this function because it's part of shrinker and shrinker doesn't touch GEMs without sgt.
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 574
Quoted 1 line
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-0+__drm_gem_shmem_release_pages (shmem);Make sure you drop the implicit
pages_use_countref the sgt had, this way you can still tie the necessity to drop the pages to sgt != NULL indrm_gem_shmem_free().
This will require further refcnt re-initialization when pages are restored if it's dropped to zero. I don't see how this will improve anything.
Signature · 2 lines
Best regards, Dmitry
Boris Brezillon
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 12 lines · On Tue, 26 Sep 2023 03:30:35 +0300 Dmitry Osipenko…
Quoted 11 lines · Boris Brezillon
The naming becomes quite confusing, with
drm_gem_shmem_unpin_locked()anddrm_gem_shmem_unpin_pages_locked(). By the look of it, it seems to do exactly the opposite ofdrm_gem_shmem_swapin_locked(), except for the missing ->evicted = true, which we can move here anyway, givendrm_gem_shmem_purge_locked()explicitly set it to false anyway. The other thing that's missing is thedrm_gem_shmem_update_pages_state_locked(), but it can also be moved there I think, if the the ->madv update happens before thedrm_gem_shmem_unpin_pages_locked()call indrm_gem_shmem_purge_locked().
Quoted 5 lines
Quoted 1 line
So, how about renaming this function
drm_gem_shmem_swapout_locked()?The swapout name would be misleading to me because pages aren't moved to swap, but allowed to be moved. I'll rename it to
drm_gem_shmem_shrinker_unpin_locked().
If you go this way, I would argue that drm_gem_shmem_swapin_locked() is just as incorrect as drm_gem_shmem_swapout_locked(), in that drm_gem_get_pages() might just return pages that were flagged reclaimable but never reclaimed/swapped-out. I do think that having some symmetry in the naming makes more sense than being 100% accurate.
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
{
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;Quoted 6 lines
Quoted 1 line
dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);Are we sure we'll always have sgt != NULL? IIRC, if the GEM is only mmap-ed in userspace,
get_sgt()is not necessarily called by the driver (needed to map in GPU space), and we have a potential NULL deref here. Maybe that changed at some point in the series, and sgt is unconditionally populated whenget_pages()is called now.
Quoted 2 lines
The sgt is always set in this function because it's part of shrinker and shrinker doesn't touch GEMs without sgt.
Okay, that's questionable. Why would we not want to reclaim BOs that are only mapped in userspace (sgt == NULL && pages_use_count > 0 && pages_pin_count == 0)? I agree that creating such a BO would be pointless (why create a buffer through DRM if it's not passed to the GPU), but that's still something the API allows...
Quoted 9 lines
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 574
Quoted 2 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-0+__drm_gem_shmem_release_pages (shmem);Make sure you drop the implicit
pages_use_countref the sgt had, this way you can still tie the necessity to drop the pages to sgt != NULL indrm_gem_shmem_free().This will require further refcnt re-initialization when pages are restored if it's dropped to zero. I don't see how this will improve anything.
Sorry to disagree, but I do think it matters to have a clear ownership model, and if I look at the code (drm_gem_shmem_get_pages_sgt_locked()), the sgt clearly owns a reference to the pages it points to.
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 13 lines · Boris Brezillon
Quoted 13 lines · Boris Brezillon
Quoted 12 lines
The naming becomes quite confusing, with
drm_gem_shmem_unpin_locked()anddrm_gem_shmem_unpin_pages_locked(). By the look of it, it seems to do exactly the opposite ofdrm_gem_shmem_swapin_locked(), except for the missing ->evicted = true, which we can move here anyway, givendrm_gem_shmem_purge_locked()explicitly set it to false anyway. The other thing that's missing is thedrm_gem_shmem_update_pages_state_locked(), but it can also be moved there I think, if the the ->madv update happens before thedrm_gem_shmem_unpin_pages_locked()call indrm_gem_shmem_purge_locked().So, how about renaming this function
drm_gem_shmem_swapout_locked()?
Quoted 8 lines
Quoted 3 lines
The swapout name would be misleading to me because pages aren't moved to swap, but allowed to be moved. I'll rename it to
drm_gem_shmem_shrinker_unpin_locked().If you go this way, I would argue that
drm_gem_shmem_swapin_locked()is just as incorrect asdrm_gem_shmem_swapout_locked(), in thatdrm_gem_get_pages()might just return pages that were flagged reclaimable but never reclaimed/swapped-out. I do think that having some symmetry in the naming makes more sense than being 100% accurate.
That function is internal to drm-shmem and is used for both eviction and purging. Having "swap-out" invoked by the purging also doesn't sound good.
Given that the function in question mainly "unmaps" the pages, what about drm_gem_shmem_shkinker_unmap_pages_locked()?
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 573
Quoted 11 lines
{
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );Are we sure we'll always have sgt != NULL? IIRC, if the GEM is only mmap-ed in userspace,
get_sgt()is not necessarily called by the driver (needed to map in GPU space), and we have a potential NULL deref here. Maybe that changed at some point in the series, and sgt is unconditionally populated whenget_pages()is called now.
Quoted 7 lines
Quoted 2 lines
The sgt is always set in this function because it's part of shrinker and shrinker doesn't touch GEMs without sgt.
Okay, that's questionable. Why would we not want to reclaim BOs that are only mapped in userspace (sgt == NULL &&
pages_use_count> 0 &&pages_pin_count== 0)? I agree that creating such a BO would be pointless (why create a buffer through DRM if it's not passed to the GPU), but that's still something the API allows...
This is a pre-existing behaviour. There is no driver that uses pages without sgt, hence there is nobody to test such code paths.
Maybe will worth to explicitly prohibit usage of get_pages() without having sgt for clarity. But this should be separate to this patchset, IMO.
Signature · 2 lines
Best regards, Dmitry
Boris Brezillon
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 17 lines · On Mon, 2 Oct 2023 22:28:13 +0300 Dmitry Osipenko…
Quoted 16 lines · Boris Brezillon
Quoted 16 lines · Boris Brezillon
Quoted 11 lines
The naming becomes quite confusing, with
drm_gem_shmem_unpin_locked()anddrm_gem_shmem_unpin_pages_locked(). By the look of it, it seems to do exactly the opposite ofdrm_gem_shmem_swapin_locked(), except for the missing ->evicted = true, which we can move here anyway, givendrm_gem_shmem_purge_locked()explicitly set it to false anyway. The other thing that's missing is thedrm_gem_shmem_update_pages_state_locked(), but it can also be moved there I think, if the the ->madv update happens before thedrm_gem_shmem_unpin_pages_locked()call indrm_gem_shmem_purge_locked().Quoted 1 line
So, how about renaming this function
drm_gem_shmem_swapout_locked()?The swapout name would be misleading to me because pages aren't moved to swap, but allowed to be moved. I'll rename it to
drm_gem_shmem_shrinker_unpin_locked().
Quoted 8 lines
Quoted 5 lines
If you go this way, I would argue that
drm_gem_shmem_swapin_locked()is just as incorrect asdrm_gem_shmem_swapout_locked(), in thatdrm_gem_get_pages()might just return pages that were flagged reclaimable but never reclaimed/swapped-out. I do think that having some symmetry in the naming makes more sense than being 100% accurate.That function is internal to drm-shmem and is used for both eviction and purging. Having "swap-out" invoked by the purging also doesn't sound good.
The part that discards the GEM in the shmem file is outside this function (shmem_truncate_range()), so all this function does in practice is flag the pages as evictable (or rather, clear the unevictable flag), so they can be reclaimed. The swapout suggesting was mostly based on the fact it does exactly the opposite of swapin().
Quoted 3 lines
Given that the function in question mainly "unmaps" the pages, what about
drm_gem_shmem_shkinker_unmap_pages_locked()?
Unmap tends to refer to a VM related operation (removing a mapping in the CPU or GPU VM), so it's confusing too IMHO. What we do here is return pages to the shmem file logic, so they can be reclaimed.
Given the drm_gem function doing that is called drm_gem_put_pages(), maybe rename it drm_gem_shmem_shrinker_put_pages_locked(), and rename drm_gem_shmem_swapin_locked() into drm_gem_shmem_shrinker_get_pages_locked(), to be consistent.
Quoted 25 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 573
Quoted 12 lines
{
drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev;dma_resv_assert_held (shmem->base.resv); -drm_WARN_ON (obj->dev, !drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) +return ;dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );Are we sure we'll always have sgt != NULL? IIRC, if the GEM is only mmap-ed in userspace,
get_sgt()is not necessarily called by the driver (needed to map in GPU space), and we have a potential NULL deref here. Maybe that changed at some point in the series, and sgt is unconditionally populated whenget_pages()is called now.Quoted 7 lines
Quoted 2 lines
The sgt is always set in this function because it's part of shrinker and shrinker doesn't touch GEMs without sgt.
Okay, that's questionable. Why would we not want to reclaim BOs that are only mapped in userspace (sgt == NULL &&
pages_use_count> 0 &&pages_pin_count== 0)? I agree that creating such a BO would be pointless (why create a buffer through DRM if it's not passed to the GPU), but that's still something the API allows...
Quoted 5 lines
This is a pre-existing behaviour. There is no driver that uses pages without sgt, hence there is nobody to test such code paths.
Maybe will worth to explicitly prohibit usage of
get_pages()without having sgt for clarity.
Nope, I don't think we should. Panthor is dissociating the BO creation for the GPU VM map operation, meaning we only get to ask for an sgt when the BO is first mapped in GPU space. In the meantime, the shrinker logic might decide to evict an object that has been already CPU-mapped (using mmap()).
Quoted 1 line
But this should be separate to this patchset, IMO.
FYI, I'm not being picky just for fun, I do intend to use the shmem-shrinker in panthor at some point, and I think it's important to get it right from the beginning, even if all your existing users don't care. I mean, I would understand if what I was asking was requiring heavy changes to the existing logic, but, unless I'm wrong, I don't think it does.
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 8 lines · Boris Brezillon
Unmap tends to refer to a VM related operation (removing a mapping in the CPU or GPU VM), so it's confusing too IMHO. What we do here is return pages to the shmem file logic, so they can be reclaimed.
Given the
drm_gemfunction doing that is calleddrm_gem_put_pages(), maybe rename itdrm_gem_shmem_shrinker_put_pages_locked(), and renamedrm_gem_shmem_swapin_locked()intodrm_gem_shmem_shrinker_get_pages_locked(), to be consistent.
The swapin is a common term and function naming scheme among DRM drivers. That name shouldn't be changed.
The drm_gem_shmem_shrinker_put_pages_locked() sounds okay, let's use it.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
In reply to Boris BrezillonQuoted 10 lines · Boris Brezillon
Quoted 7 lines
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 574
Quoted 1 line
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-0+__drm_gem_shmem_release_pages (shmem);Make sure you drop the implicit
pages_use_countref the sgt had, this way you can still tie the necessity to drop the pages to sgt != NULL indrm_gem_shmem_free().This will require further refcnt re-initialization when pages are restored if it's dropped to zero. I don't see how this will improve anything.
Sorry to disagree, but I do think it matters to have a clear ownership model, and if I look at the code (
drm_gem_shmem_get_pages_sgt_locked()), the sgt clearly owns a reference to the pages it points to.
It creates too much unnecessary trouble because, again, pages_use_count can't drop to zero easily. Shrinker doesn't own the refcnt and not allowed to touch it. The pages_use_count is then used by things like mmap() and etc that use get_pages(), which can be invoked for evicted GEM.
I'd prefer to keep refcounting as is, don't see how to implement your suggestion. Will be happy to help with reviewing and testing patches made on top of this series ;)
Signature · 2 lines
Best regards, Dmitry
Boris Brezillon
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Hello Dmitry,
Quoted 8 lines · On Tue, 3 Oct 2023 03:31:32 +0300 Dmitry Osipenko…
Quoted 7 lines · Boris Brezillon
Quoted 7 lines
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_madvise_locked(struct drm_gem_shmem_object *shmem, int madv) · line 574
Quoted 1 line
drivers/gpu/drm/drm_gem_shmem_helper.c +1
-0+__drm_gem_shmem_release_pages (shmem);Make sure you drop the implicit
pages_use_countref the sgt had, this way you can still tie the necessity to drop the pages to sgt != NULL indrm_gem_shmem_free().This will require further refcnt re-initialization when pages are restored if it's dropped to zero. I don't see how this will improve anything.
Quoted 6 lines
Quoted 3 lines
Sorry to disagree, but I do think it matters to have a clear ownership model, and if I look at the code (
drm_gem_shmem_get_pages_sgt_locked()), the sgt clearly owns a reference to the pages it points to.It creates too much unnecessary trouble because, again,
pages_use_countcan't drop to zero easily.
Not saying pages_use_count should drop to zero, I'm just saying the reference that was owned by the sgt should be released when this sgt is freed, no matter if this sgt destruction is triggered by a GEM eviction, or because the GEM object is freed entirely.
Quoted 2 lines
Shrinker doesn't own the refcnt and not
allowed to touch it.
I'm not asking the shrinker to own a reference on the pages either. It's really the sgt that owns this reference.
Quoted 2 lines
The
pages_use_countis then used by things likemmap()and etc that useget_pages(), which can be invoked for evicted GEM.
Yes, and I still have a hard time seeing how this interferes with what I'm suggesting to be honest.
Quoted 3 lines
I'd prefer to keep refcounting as is, don't see how to implement your suggestion.
Can you be more specific? I don't really see what the problem is with decrementing pages_use_count when you free the sgt (eviction), and re-incrementing it when the sgt is restored (swapin).
Regards,
Boris
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 5 lines · Boris Brezillon
Quoted 2 lines
I'd prefer to keep refcounting as is, don't see how to implement your suggestion.
Can you be more specific? I don't really see what the problem is with decrementing
pages_use_countwhen you free the sgt (eviction), and re-incrementing it when the sgt is restored (swapin).
For the reference, we further discussed this question about refcounting with Boris offline and found how to implement the refcnt drop done by shrinker's evict/purge.
For evict/purge we can do:
if (!refcount_dec_not_one(&shmem->pages_use_count))
refcount_set(&shmem->pages_use_count, 0);and then for swapin:
if (!refcount_inc_not_zero(&shmem->pages_use_count))
refcount_set(&shmem->pages_use_count, 1);This resolves the issue with dropping refcnt to zero I was talking about, allowing to delegate sgt's refcnt ownership to shrinker.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
In reply to Boris BrezillonQuoted 18 lines · Boris Brezillon
Quoted 18 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +9
-2-static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem,bool init) {struct drm_gem_object *obj = &shmem->base;struct page **pages;dma_resv_assert_held (shmem->base.resv); -if (refcount_inc_not_zero (&shmem->pages_use_count)) +if (shmem->madv <0 ) { +drm_WARN_ON (obj->dev, shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted);return 0 ;
Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 257
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +4
-0+ } + +if (drm_WARN_ON (obj->dev, !(init ^refcount_read (&shmem->pages_use_count)))) +return -EINVAL ;OOC, why do we care? Is there any difference between initial and re-pin that make the page allocation impossible? Feels like, if there's a check to do, it should be done in the caller instead, and you can drop the init param here.
This is a sanity check that addresses additional refcnt tracking complexity imposed by shrinker.
This function is used by both init and re-pin that is invoked from several places in the code. It's not trivial to move that check to the callers.
Signature · 2 lines
Best regards, Dmitry
Boris Brezillon
maintainerRe: [PATCH v17 13/18] drm/shmem-helper: Add memory shrinker
Quoted 28 lines · On Tue, 26 Sep 2023 03:37:22 +0300 Dmitry Osipenko…
Quoted 18 lines · Boris Brezillon
Quoted 18 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +9
-2-static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem,bool init) {struct drm_gem_object *obj = &shmem->base;struct page **pages;dma_resv_assert_held (shmem->base.resv); -if (refcount_inc_not_zero (&shmem->pages_use_count)) +if (shmem->madv <0 ) { +drm_WARN_ON (obj->dev, shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +drm_WARN_ON (obj->dev, !shmem->evicted);return 0 ;Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) · line 257
Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +4
-0+ } + +if (drm_WARN_ON (obj->dev, !(init ^refcount_read (&shmem->pages_use_count)))) +return -EINVAL ;OOC, why do we care? Is there any difference between initial and re-pin that make the page allocation impossible? Feels like, if there's a check to do, it should be done in the caller instead, and you can drop the init param here.
Quoted 6 lines
This is a sanity check that addresses additional refcnt tracking complexity imposed by shrinker.
This function is used by both init and re-pin that is invoked from several places in the code. It's not trivial to move that check to the callers.
drm_gem_shmem_acquire_pages() is called twice, once with init=false, once with init=true. If you really care about this check, it can be moved to the callers so
1/ it's clearer (the XOR operation between init and refcount to check if refcount is zero on init and non-zero otherwise is convoluted) 2/ it doesn't leak to the function whose purpose it to [re-]acquire pages
Dmitry Osipenko
maintainerPATCH v17 14/18 drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
In reply to Dmitry OsipenkoExport drm_gem_shmem_get_pages_sgt_locked() that will be used by virtio-gpu shrinker during GEM swap-in operation done under the held reservation lock.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++- include/drm/drm_gem_shmem_helper.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +2 -1
@@ -865,7 +865,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_get_sg_table); -static struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem) +struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;int ret;
@@ -903,6 +903,7 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_drm_gem_shmem_put_pages_locked (shmem);return ERR_PTR (ret); } +EXPORT_SYMBOL_GPL (drm_gem_shmem_get_pages_sgt_locked);/** * drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a
include/drm/drm_gem_shmem_helper.h +1 -0
@@ -147,6 +147,7 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem); +struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent);
Signature · 1 line
2.41.0
Re: [PATCH v17 14/18] drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked()
Quoted 4 lines · On Fri, 15 Sep 2023 02:27:17 +0300 Dmitry Osipenko…
Export
drm_gem_shmem_get_pages_sgt_locked()that will be used by virtio-gpu shrinker during GEM swap-in operation done under the held reservation lock.Signed-off-by Dmitry Osipenko <[email protected]>
Quoted 38 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 3 ++- include/drm/drm_gem_shmem_helper.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)drivers/gpu/drm/drm_gem_shmem_helper.c +2
-1@@ -865,7 +865,7 @@ struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_get_sg_table); -static struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem) +struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;int ret;@@ -903,6 +903,7 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_drm_gem_shmem_put_pages_locked (shmem);return ERR_PTR (ret); } +EXPORT_SYMBOL_GPL (drm_gem_shmem_get_pages_sgt_locked);/** * drm_gem_shmem_get_pages_sgt - Pin pages, dma map them, and return a include/drm/drm_gem_shmem_helper.h +1
-0@@ -147,6 +147,7 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem); +struct sg_table *drm_gem_shmem_get_pages_sgt_locked (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent);
Dmitry Osipenko
maintainerPATCH v17 15/18 drm/virtio: Pin display framebuffer BO
In reply to Dmitry OsipenkoPrepare to addition of memory shrinker support by pinning display framebuffer BO pages in memory while they are in use by display on host. Shrinker is free to relocate framebuffer BO pages if it doesn't know that pages are in use, thus pin the pages to disallow shrinker to move them.
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 2 ++ drivers/gpu/drm/virtio/virtgpu_gem.c | 19 +++++++++++++++++++ drivers/gpu/drm/virtio/virtgpu_plane.c | 17 +++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +2 -0
@@ -313,6 +313,8 @@ void virtio_gpu_array_put_free(struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_delayed (struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_work (struct work_struct *work); +int virtio_gpu_gem_pin (struct virtio_gpu_object *bo); +void virtio_gpu_gem_unpin (struct virtio_gpu_object *bo);/* virtgpu_vq.c */ int virtio_gpu_alloc_vbufs (struct virtio_gpu_device *vgdev);
drivers/gpu/drm/virtio/virtgpu_gem.c +19 -0
@@ -294,3 +294,22 @@ void virtio_gpu_array_put_free_work(struct work_struct *work) }spin_unlock (&vgdev->obj_free_lock); } + +int virtio_gpu_gem_pin (struct virtio_gpu_object *bo) +{ +int err; + +if (virtio_gpu_is_shmem (bo)) { + err =drm_gem_shmem_pin (&bo->base); +if (err) +return err; + } + +return 0 ; +} + +void virtio_gpu_gem_unpin (struct virtio_gpu_object *bo) +{ +if (virtio_gpu_is_shmem (bo)) +drm_gem_shmem_unpin (&bo->base); +}
drivers/gpu/drm/virtio/virtgpu_plane.c +15 -2
@@ -238,20 +238,28 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,struct virtio_gpu_device *vgdev = dev->dev_private;struct virtio_gpu_framebuffer *vgfb;struct virtio_gpu_object *bo; +int err;if (!new_state->fb)return 0 ; vgfb =to_virtio_gpu_framebuffer (new_state->fb); bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); -if (!bo || (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)) + + err =virtio_gpu_gem_pin (bo); +if (err) +return err; + +if (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)return 0 ;if (bo->dumb && (plane->state->fb != new_state->fb)) { vgfb->fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 ); -if (!vgfb->fence) +if (!vgfb->fence) { +virtio_gpu_gem_unpin (bo);return -ENOMEM ; + } }return 0 ;
@@ -261,15 +269,20 @@ static void virtio_gpu_plane_cleanup_fb(struct drm_plane *plane, struct drm_plane_state *state) {struct virtio_gpu_framebuffer *vgfb; +struct virtio_gpu_object *bo;if (!state->fb)return ; vgfb =to_virtio_gpu_framebuffer (state->fb); + bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); +if (vgfb->fence) {dma_fence_put (&vgfb->fence->f); vgfb->fence =NULL ; } + +virtio_gpu_gem_unpin (bo); } static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 16/18 drm/virtio: Attach shmem BOs dynamically
In reply to Dmitry OsipenkoPrepare for addition of memory shrinker support by attaching shmem pages to host dynamically on first use. The attachment vq command wasn't fenced and there was no vq kick made in the BO creation code path, hence the the attachment already was happening dynamically, but implicitly. Making attachment explicitly dynamic will allow to simplify and reuse more code when shrinker will be added. The virtio_gpu_object_shmem_init() now works under held reservation lock, which will be important to have for shrinker.
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 7 +++ drivers/gpu/drm/virtio/virtgpu_gem.c | 26 ++++++++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 32 ++++++---- drivers/gpu/drm/virtio/virtgpu_object.c | 80 ++++++++++++++++++++----- drivers/gpu/drm/virtio/virtgpu_submit.c | 15 ++++- 5 files changed, 132 insertions(+), 28 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +7 -0
@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;
@@ -313,6 +314,8 @@ void virtio_gpu_array_put_free(struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_delayed (struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_work (struct work_struct *work); +int virtio_gpu_array_prepare (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object_array *objs);int virtio_gpu_gem_pin (struct virtio_gpu_object *bo);void virtio_gpu_gem_unpin (struct virtio_gpu_object *bo);
@@ -458,6 +461,10 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object_locked (struct virtio_gpu_object *bo); + +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */
drivers/gpu/drm/virtio/virtgpu_gem.c +26 -0
@@ -295,6 +295,26 @@ void virtio_gpu_array_put_free_work(struct work_struct *work)spin_unlock (&vgdev->obj_free_lock); } +int virtio_gpu_array_prepare (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object_array *objs) +{ +struct virtio_gpu_object *bo; +int ret =0 ; +u32 i; + +for (i =0 ; i < objs->nents; i++) { + bo =gem_to_virtio_gpu_obj (objs->objs[i]); + +if (virtio_gpu_is_shmem (bo) && bo->detached) { + ret =virtio_gpu_reattach_shmem_object_locked (bo); +if (ret) +break ; + } + } + +return ret; +} +int virtio_gpu_gem_pin (struct virtio_gpu_object *bo) {int err;
@@ -303,6 +323,12 @@ int virtio_gpu_gem_pin(struct virtio_gpu_object *bo) err =drm_gem_shmem_pin (&bo->base);if (err)return err; + + err =virtio_gpu_reattach_shmem_object (bo); +if (err) { +drm_gem_shmem_unpin (&bo->base); +return err; + } }return 0 ;
drivers/gpu/drm/virtio/virtgpu_ioctl.c +20 -12
@@ -246,6 +246,10 @@ static int virtio_gpu_transfer_from_host_ioctl(struct drm_device *dev,if (ret !=0 ) goto err_put_free; + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 );if (!fence) { ret = -ENOMEM ;
@@ -288,11 +292,25 @@ static int virtio_gpu_transfer_to_host_ioctl(struct drm_device *dev, void *data, goto err_put_free; } + ret =virtio_gpu_array_lock_resv (objs); +if (ret !=0 ) + goto err_put_free; + + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + + fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 ); +if (!fence) { + ret = -ENOMEM ; + goto err_unlock; + } +if (!vgdev->has_virgl_3d) {virtio_gpu_cmd_transfer_to_host_2d (vgdev, offset, args->box.w, args->box.h, args->box.x, args->box.y, - objs,NULL ); + objs, fence); }else {virtio_gpu_create_context (dev, file);
@@ -301,23 +319,13 @@ static int virtio_gpu_transfer_to_host_ioctl(struct drm_device *dev, void *data, goto err_put_free; } - ret =virtio_gpu_array_lock_resv (objs); -if (ret !=0 ) - goto err_put_free; - - ret = -ENOMEM ; - fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context, -0 ); -if (!fence) - goto err_unlock; -virtio_gpu_cmd_transfer_to_host_3d (vgdev, vfpriv ? vfpriv->ctx_id :0 , offset, args->level, args->stride, args->layer_stride, &args->box, objs, fence); -dma_fence_put (&fence->f); } +dma_fence_put (&fence->f);virtio_gpu_notify (vgdev);return 0 ;
drivers/gpu/drm/virtio/virtgpu_object.c +66 -14
127 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_submit.c +13 -2
@@ -465,8 +465,19 @@ static void virtio_gpu_install_out_fence_fd(struct virtio_gpu_submit *submit)static int virtio_gpu_lock_buflist (struct virtio_gpu_submit *submit) { -if (submit->buflist) -return virtio_gpu_array_lock_resv (submit->buflist); +int err; + +if (submit->buflist) { + err =virtio_gpu_array_lock_resv (submit->buflist); +if (err) +return err; + + err =virtio_gpu_array_prepare (submit->vgdev, submit->buflist); +if (err) { +virtio_gpu_array_unlock_resv (submit->buflist); +return err; + } + }return 0 ; }
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 17/18 drm/virtio: Support memory shrinking
In reply to Dmitry OsipenkoSupport generic drm-shmem memory shrinker and add new madvise IOCTL to the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as "don't need" using the new IOCTL to let shrinker purge the marked BOs on OOM, the shrinker will also evict unpurgeable shmem BOs from memory if guest supports SWAP file or partition.
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 13 +++++- drivers/gpu/drm/virtio/virtgpu_gem.c | 35 ++++++++++++++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 25 ++++++++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 8 ++++ drivers/gpu/drm/virtio/virtgpu_object.c | 61 +++++++++++++++++++++++++ drivers/gpu/drm/virtio/virtgpu_vq.c | 40 ++++++++++++++++ include/uapi/drm/virtgpu_drm.h | 14 ++++++ 7 files changed, 195 insertions(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +12 -1
@@ -278,7 +278,7 @@ struct virtio_gpu_fpriv { };/* virtgpu_ioctl.c */ -#define DRM_VIRTIO_NUM_IOCTLS 12 +#define DRM_VIRTIO_NUM_IOCTLS 13extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ];void virtio_gpu_create_context (struct drm_device *dev,struct drm_file *file);
@@ -316,6 +316,8 @@ void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,void virtio_gpu_array_put_free_work (struct work_struct *work);int virtio_gpu_array_prepare (struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs); +int virtio_gpu_gem_host_mem_release (struct virtio_gpu_object *bo); +int virtio_gpu_gem_madvise (struct virtio_gpu_object *obj,int madv);int virtio_gpu_gem_pin (struct virtio_gpu_object *bo);void virtio_gpu_gem_unpin (struct virtio_gpu_object *bo);
@@ -329,6 +331,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev, struct virtio_gpu_fence *fence);void virtio_gpu_cmd_unref_resource (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo); +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo); void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev, uint64_t offset, uint32_t width, uint32_t height,
@@ -349,6 +353,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,struct virtio_gpu_object *obj, struct virtio_gpu_mem_entry *ents, unsigned int nents); +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence);int virtio_gpu_attach_status_page (struct virtio_gpu_device *vgdev);int virtio_gpu_detach_status_page (struct virtio_gpu_device *vgdev); void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -499,4 +506,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,int virtio_gpu_execbuffer_ioctl (struct drm_device *dev,void *data,struct drm_file *file); +/* virtgpu_gem_shrinker.c */ +int virtio_gpu_gem_shrinker_init (struct virtio_gpu_device *vgdev); +void virtio_gpu_gem_shrinker_fini (struct virtio_gpu_device *vgdev); +#endif
drivers/gpu/drm/virtio/virtgpu_gem.c +35 -0
@@ -147,10 +147,20 @@ void virtio_gpu_gem_object_close(struct drm_gem_object *obj,struct virtio_gpu_device *vgdev = obj->dev->dev_private;struct virtio_gpu_fpriv *vfpriv = file->driver_priv;struct virtio_gpu_object_array *objs; +struct virtio_gpu_object *bo;if (!vgdev->has_virgl_3d)return ; + bo =gem_to_virtio_gpu_obj (obj); + +/* +* Purged BO was already detached and released, the resource ID +* is invalid by now. +*/ +if (!virtio_gpu_gem_madvise (bo,VIRTGPU_MADV_WILLNEED )) +return ; + objs =virtio_gpu_array_alloc (1 );if (!objs)return ;
@@ -315,6 +325,31 @@ int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,return ret; } +int virtio_gpu_gem_madvise (struct virtio_gpu_object *bo,int madv) +{ +if (virtio_gpu_is_shmem (bo)) +return drm_gem_shmem_object_madvise (&bo->base.base, madv); + +return 1 ; +} + +int virtio_gpu_gem_host_mem_release (struct virtio_gpu_object *bo) +{ +struct virtio_gpu_device *vgdev = bo->base.base.dev->dev_private; +int err; + +if (bo->created) { + err =virtio_gpu_cmd_release_resource (vgdev, bo); +if (err) +return err; + +virtio_gpu_notify (vgdev); + bo->created = false; + } + +return 0 ; +} +int virtio_gpu_gem_pin (struct virtio_gpu_object *bo) {int err;
drivers/gpu/drm/virtio/virtgpu_ioctl.c +25 -0
@@ -676,6 +676,28 @@ static int virtio_gpu_context_init_ioctl(struct drm_device *dev,return ret; } +static int virtio_gpu_madvise_ioctl (struct drm_device *dev, +void *data, +struct drm_file *file) +{ +struct drm_virtgpu_madvise *args = data; +struct virtio_gpu_object *bo; +struct drm_gem_object *obj; + +if (args->madv >VIRTGPU_MADV_DONTNEED ) +return -EOPNOTSUPP ; + + obj =drm_gem_object_lookup (file, args->bo_handle); +if (!obj) +return -ENOENT ; + + bo =gem_to_virtio_gpu_obj (obj); + args->retained =virtio_gpu_gem_madvise (bo, args->madv); +drm_gem_object_put (obj); + +return 0 ; +} +struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ] = {DRM_IOCTL_DEF_DRV (VIRTGPU_MAP , virtio_gpu_map_ioctl,DRM_RENDER_ALLOW ),
@@ -715,4 +737,7 @@ struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS] = {DRM_IOCTL_DEF_DRV (VIRTGPU_CONTEXT_INIT , virtio_gpu_context_init_ioctl,DRM_RENDER_ALLOW ), + +DRM_IOCTL_DEF_DRV (VIRTGPU_MADVISE , virtio_gpu_madvise_ioctl, + DRM_RENDER_ALLOW), };
drivers/gpu/drm/virtio/virtgpu_kms.c +8 -0
@@ -245,6 +245,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) goto err_scanouts; } + ret =drmm_gem_shmem_init (dev); +if (ret) { +DRM_ERROR ("shmem init failed\n" ); + goto err_modeset; + } +virtio_device_ready (vgdev->vdev);if (num_capsets)
@@ -259,6 +265,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) }return 0 ; +err_modeset: +virtio_gpu_modeset_fini (vgdev); err_scanouts:virtio_gpu_free_vbufs (vgdev); err_vbufs:
drivers/gpu/drm/virtio/virtgpu_object.c +61 -0
85 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_vq.c +40 -0
@@ -545,6 +545,21 @@ void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,virtio_gpu_cleanup_object (bo); } +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo) +{ +struct virtio_gpu_resource_unref *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_UNREF ); + cmd_p->resource_id =cpu_to_le32 (bo->hw_res_handle); + +return virtio_gpu_queue_ctrl_buffer (vgdev, vbuf); +} + void virtio_gpu_cmd_set_scanout(struct virtio_gpu_device *vgdev, uint32_t scanout_id, uint32_t resource_id, uint32_t width, uint32_t height,
@@ -645,6 +660,23 @@ virtio_gpu_cmd_resource_attach_backing(struct virtio_gpu_device *vgdev,virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); } +static void +virtio_gpu_cmd_resource_detach_backing (struct virtio_gpu_device *vgdev, +u32 resource_id, +struct virtio_gpu_fence *fence) +{ +struct virtio_gpu_resource_attach_backing *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING ); + cmd_p->resource_id =cpu_to_le32 (resource_id); + +virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); +} +static void virtio_gpu_cmd_get_display_info_cb (struct virtio_gpu_device *vgdev,struct virtio_gpu_vbuffer *vbuf) {
@@ -1107,6 +1139,14 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev, ents, nents, NULL); } +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence) +{ +virtio_gpu_cmd_resource_detach_backing (vgdev, obj->hw_res_handle, + fence); +} +void virtio_gpu_cursor_ping (struct virtio_gpu_device *vgdev,struct virtio_gpu_output *output) {
include/uapi/drm/virtgpu_drm.h +14 -0
@@ -48,6 +48,7 @@ extern "C" {#define DRM_VIRTGPU_GET_CAPS 0x09#define DRM_VIRTGPU_RESOURCE_CREATE_BLOB 0x0a#define DRM_VIRTGPU_CONTEXT_INIT 0x0b +#define DRM_VIRTGPU_MADVISE 0x0c#define VIRTGPU_EXECBUF_FENCE_FD_IN 0x01#define VIRTGPU_EXECBUF_FENCE_FD_OUT 0x02
@@ -211,6 +212,15 @@ struct drm_virtgpu_context_init {__u64 ctx_set_params; }; +#define VIRTGPU_MADV_WILLNEED 0 +#define VIRTGPU_MADV_DONTNEED 1 +struct drm_virtgpu_madvise { +__u32 bo_handle; +__u32 retained;/* out, non-zero if BO can be used */ +__u32 madv; +__u32 pad; +}; +/* * Event code that's given when VIRTGPU_CONTEXT_PARAM_POLL_RINGS_MASK is in * effect. The event size is sizeof(drm_event), since there is no additional
@@ -261,6 +271,10 @@ struct drm_virtgpu_context_init {DRM_IOWR (DRM_COMMAND_BASE +DRM_VIRTGPU_CONTEXT_INIT , \ struct drm_virtgpu_context_init) +#define DRM_IOCTL_VIRTGPU_MADVISE \ + DRM_IOWR(DRM_COMMAND_BASE + DRM_VIRTGPU_MADVISE, \ + struct drm_virtgpu_madvise) +#if defined(__cplusplus) }#endif
Signature · 1 line
2.41.0
Dmitry Osipenko
maintainerPATCH v17 18/18 drm/panfrost: Switch to generic memory shrinker
In reply to Dmitry OsipenkoReplace Panfrost's custom memory shrinker with a common drm-shmem memory shrinker.
--- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 27 ++-- drivers/gpu/drm/panfrost/panfrost_gem.c | 30 ++-- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 -- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 ------------------ drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++- include/drm/drm_gem_shmem_helper.h | 7 - 8 files changed, 47 insertions(+), 178 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
drivers/gpu/drm/panfrost/Makefile +0 -1
@@ -5,7 +5,6 @@ panfrost-y := \ panfrost_device.o \ panfrost_devfreq.o \ panfrost_gem.o \ - panfrost_gem_shrinker.o \ panfrost_gpu.o \ panfrost_job.o \ panfrost_mmu.o \
drivers/gpu/drm/panfrost/panfrost_device.h +0 -4
@@ -116,10 +116,6 @@ struct panfrost_device {atomic_t pending; } reset; -struct mutex shrinker_lock; -struct list_head shrinker_list; -struct shrinker shrinker; -struct panfrost_devfreq pfdevfreq; };
drivers/gpu/drm/panfrost/panfrost_drv.c +10 -17
@@ -170,7 +170,6 @@ panfrost_lookup_bos(struct drm_device *dev,break ; } -atomic_inc (&bo->gpu_usecount); job->mappings[i] = mapping; }
@@ -395,7 +394,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, {struct panfrost_file_priv *priv = file_priv->driver_priv;struct drm_panfrost_madvise *args = data; -struct panfrost_device *pfdev = dev->dev_private;struct drm_gem_object *gem_obj;struct panfrost_gem_object *bo;int ret =0 ;
@@ -408,11 +406,15 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, bo =to_panfrost_bo (gem_obj); +if (bo->is_heap) { + args->retained =1 ; + goto out_put_object; + } + ret =dma_resv_lock_interruptible (bo->base.base.resv,NULL );if (ret) goto out_put_object; -mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {struct panfrost_gem_mapping *first;
@@ -438,17 +440,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, args->retained =drm_gem_shmem_madvise_locked (&bo->base, args->madv); -if (args->retained) { -if (args->madv ==PANFROST_MADV_DONTNEED ) -list_move_tail (&bo->base.madv_list, - &pfdev->shrinker_list); -else if (args->madv ==PANFROST_MADV_WILLNEED ) -list_del_init (&bo->base.madv_list); - } - out_unlock_mappings:mutex_unlock (&bo->mappings.lock); -mutex_unlock (&pfdev->shrinker_lock);dma_resv_unlock (bo->base.base.resv); out_put_object:drm_gem_object_put (gem_obj);
@@ -577,9 +570,6 @@ static int panfrost_probe(struct platform_device *pdev) ddev->dev_private = pfdev; pfdev->ddev = ddev; -mutex_init (&pfdev->shrinker_lock); -INIT_LIST_HEAD (&pfdev->shrinker_list); - err =panfrost_device_init (pfdev);if (err) {if (err != -EPROBE_DEFER )
@@ -601,10 +591,14 @@ static int panfrost_probe(struct platform_device *pdev)if (err <0 ) goto err_out1; -panfrost_gem_shrinker_init (ddev); + err =drmm_gem_shmem_init (ddev); +if (err <0 ) + goto err_out2;return 0 ; +err_out2: +drm_dev_unregister (ddev); err_out1:pm_runtime_disable (pfdev->dev);panfrost_device_fini (pfdev);
@@ -620,7 +614,6 @@ static void panfrost_remove(struct platform_device *pdev)struct drm_device *ddev = pfdev->ddev;drm_dev_unregister (ddev); -panfrost_gem_shrinker_cleanup (ddev);pm_runtime_get_sync (pfdev->dev);pm_runtime_disable (pfdev->dev);
drivers/gpu/drm/panfrost/panfrost_gem.c +20 -10
@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)struct panfrost_gem_object *bo =to_panfrost_bo (obj);struct panfrost_device *pfdev = obj->dev->dev_private; -/* -* Make sure the BO is no longer inserted in the shrinker list before -* taking care of the destruction itself. If we don't do that we have a -* race condition between this function and what's done in -* panfrost_gem_shrinker_scan(). -*/ -mutex_lock (&pfdev->shrinker_lock); -list_del_init (&bo->base.madv_list); -mutex_unlock (&pfdev->shrinker_lock); -/* * If we still have mappings attached to the BO, there's a problem in * our refcounting.
@@ -195,6 +185,25 @@ static int panfrost_gem_pin(struct drm_gem_object *obj)return drm_gem_shmem_object_pin (obj); } +static int panfrost_shmem_evict (struct drm_gem_object *obj) +{ +struct panfrost_gem_object *bo =to_panfrost_bo (obj); + +if (!drm_gem_shmem_is_purgeable (&bo->base)) +return -EBUSY ; + +if (!mutex_trylock (&bo->mappings.lock)) +return -EBUSY ; + +panfrost_gem_teardown_mappings_locked (bo); + +drm_gem_shmem_purge_locked (&bo->base); + +mutex_unlock (&bo->mappings.lock); + +return 0 ; +} +static const struct drm_gem_object_funcs panfrost_gem_funcs = { .free = panfrost_gem_free_object, .open = panfrost_gem_open,
@@ -207,6 +216,7 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .vunmap = drm_gem_shmem_object_vunmap_locked, .mmap = drm_gem_shmem_object_mmap, .vm_ops = &drm_gem_shmem_vm_ops, + .evict = panfrost_shmem_evict, };/**
drivers/gpu/drm/panfrost/panfrost_gem.h +0 -9
@@ -30,12 +30,6 @@ struct panfrost_gem_object {struct mutex lock; } mappings; -/* -* Count the number of jobs referencing this BO so we don't let the -* shrinker reclaim this object prematurely. -*/ -atomic_t gpu_usecount; - bool noexec :1; bool is_heap :1; };
@@ -81,7 +75,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); -void panfrost_gem_shrinker_cleanup (struct drm_device *dev); -#endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0 -129
129 lines not on this page · raw message
drivers/gpu/drm/panfrost/panfrost_job.c +17 -1
@@ -272,6 +272,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,dma_resv_add_fence (bos[i]->resv, fence,DMA_RESV_USAGE_WRITE ); } +static int panfrost_objects_prepare (struct drm_gem_object **bos,int bo_count) +{ +struct panfrost_gem_object *bo; +int ret =0 ; + +while (!ret && bo_count--) { + bo =to_panfrost_bo (bos[bo_count]); + ret = bo->base.madv ? -ENOMEM :0 ; + } + +return ret; +} +int panfrost_job_push (struct panfrost_job *job) {struct panfrost_device *pfdev = job->pfdev;
@@ -283,6 +296,10 @@ int panfrost_job_push(struct panfrost_job *job)if (ret)return ret; + ret =panfrost_objects_prepare (job->bos, job->bo_count); +if (ret) + goto unlock; +mutex_lock (&pfdev->sched_lock);drm_sched_job_arm (&job->base);
@@ -324,7 +341,6 @@ static void panfrost_job_cleanup(struct kref *ref)if (!job->mappings[i])break ; -atomic_dec (&job->mappings[i]->obj->gpu_usecount);panfrost_gem_mapping_put (job->mappings[i]); }kvfree (job->mappings);
include/drm/drm_gem_shmem_helper.h +0 -7
@@ -60,13 +60,6 @@ struct drm_gem_shmem_object {*/ int madv; -/** -* @madv_list: List entry for madvise tracking -* -* Typically used by drivers to track purgeable objects -*/ -struct list_head madv_list; -/** * @sgt: Scatter/gather table for imported PRIME buffers */
Signature · 1 line
2.41.0