Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
Subsystems (14)
Dmitry Osipenko
maintainerv6 cover Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers
Hello,
This patchset introduces memory shrinker for the VirtIO-GPU DRM driver and adds memory purging and eviction support to VirtIO-GPU driver.
The new dma-buf locking convention is introduced here as well.
During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL, it will also evict idling BOs to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.
The Panfrost driver is switched to use generic memory shrinker.
This patchset includes improvements and fixes for various things that I found while was working on the shrinker.
The Mesa and IGT patches will be kept on hold until this kernel series will be approved and merged.
This patchset was tested using Qemu and crosvm, including both cases of IOMMU off/on.
Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise IGT: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvise
https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise
Changelog:
v6: - Added new VirtIO-related fix patch that previously was sent separately
and didn't get much attention:
drm/gem: Properly annotate WW context on drm_gem_lock_reservations() error
- Added new patch that fixes mapping of imported dma-bufs for Tegra DRM and other affected drivers. It's also handy to have it for switching to the new dma-buf locking convention scheme:
drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
- Added new patch that fixes shrinker list corruption for stable Panfrost driver:
drm/panfrost: Fix shrinker list corruption by madvise IOCTL
- Added new minor patch-fix for drm-shmem:
drm/shmem-helper: Add missing vunmap on error
- Added fixes tag to the "Put mapping ..." patch like was suggested by Steven Price.
- Added new VirtIO-GPU driver improvement patch:
drm/virtio: Return proper error codes instead of -1
- Reworked shrinker patches like was suggested by Daniel Vetter:
- Introduced the new locking convention for dma-bufs. Tested on VirtIO-GPU, Panfrost, Lima, Tegra and Intel selftests.
- Dropped separate purge() callback. Now single evict() does everything.
- Dropped swap_in() callback from drm-shmem objects. DRM drivers now could and should restore only the required mappings.
- Dropped dynamic counting of evictable pages. This simplifies code in exchange to *potentially* burning more CPU time on OOM.
v5: - Added new for-stable patch "drm/panfrost: Put mapping instead of
shmem obj on panfrost_mmu_map_fault_addr() error" that corrects GEM's refcounting in case of error.
- The drm_gem_shmem_v[un]map() now takes a separate vmap_lock for imported GEMs to avoid recursive locking of DMA reservations. This addresses v4 comment from Thomas Zimmermann about the potential deadlocking of vmapping.
- Added ack from Thomas Zimmermann to "drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table()" patch.
- Dropped explicit shmem states from the generic shrinker patch as was requested by Thomas Zimmermann.
- Improved variable names and comments of the generic shrinker code.
- Extended drm_gem_shmem_print_info() with the shrinker-state info in the "drm/virtio: Support memory shrinking" patch.
- Moved evict()/swap_in()/purge() callbacks from drm_gem_object_funcs to drm_gem_shmem_object in the generic shrinker patch, for more consistency.
- Corrected bisectability of the patches that was broken in v4 by accident.
- The virtio_gpu_plane_prepare_fb() now uses drm_gem_shmem_pin() instead of drm_gem_shmem_set_unpurgeable_and_unevictable() and does it only for shmem BOs in the "drm/virtio: Support memory shrinking" patch.
- Made more functions private to drm_gem_shmem_helper.c as was requested by Thomas Zimmermann. This minimizes number of the public shmem helpers.
v4: - Corrected minor W=1 warnings reported by kernel test robot for v3.
- Renamed DRM_GEM_SHMEM_PAGES_STATE_ACTIVE/INACTIVE to PINNED/UNPINNED, for more clarity.
v3: - Hardened shrinker's count() with usage of READ_ONCE() since we don't
use atomic type for counting and technically compiler is free to re-fetch counter's variable.
- "Correct drm_gem_shmem_get_sg_table() error handling" now uses PTR_ERR_OR_ZERO(), fixing typo that was made in v2.
- Removed obsoleted shrinker from the Panfrost driver, which I missed to do in v2 by accident and Alyssa Rosenzweig managed to notice it.
- CCed stable kernels in all patches that make fixes, even the minor ones, like was suggested by Emil Velikov and added his r-b to the patches.
- Added t-b from Steven Price to the Panfrost's shrinker patch.
- Corrected doc-comment of drm_gem_shmem_object.madv, like was suggested by Steven Price. Comment now says that madv=1 means "object is purged" instead of saying that value is unused.
- Added more doc-comments to the new shmem shrinker API.
- The "Improve DMA API usage for shmem BOs" patch got more improvements by removing the obsoleted drm_dev_set_unique() quirk and its comment.
- Added patch that makes Virtio-GPU driver to use common dev_is_pci() helper, which was suggested by Robin Murphy.
- Added new "drm/shmem-helper: Take GEM reservation lock instead of drm_gem_shmem locks" patch, which was suggested by Daniel Vetter.
- Added new "drm/virtio: Simplify error handling of virtio_gpu_object_create()" patch.
- Improved "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, like was suggested by Daniel Vetter, by saying that function returns ERR_PTR() and not errno.
- virtio_gpu_purge_object() is fenced properly now, turned out virtio_gpu_notify() doesn't do fencing as I was supposing before. Stress testing of memory eviction revealed that.
- Added new patch that corrects virtio_gpu_plane_cleanup_fb() to use appropriate atomic plane state.
- SHMEM shrinker got eviction support.
- VirtIO-GPU driver now supports memory eviction. It's enabled for a non-blob GEMs only, i.e. for VirGL. The blobs don't support dynamic attaching/detaching of guest's memory, so it's not trivial to enable them.
- Added patch that removes obsoleted drm_gem_shmem_purge()
- Added patch that makes drm_gem_shmem_get_pages() private.
- Added patch that fixes lockup on dma_resv_reserve_fences() error.
v2: - Improved shrinker by using a more fine-grained locking to reduce
contention during scan of objects and dropped locking from the 'counting' callback by tracking count of shrinkable pages. This was suggested by Rob Clark in the comment to v1.
- Factored out common shrinker code into drm_gem_shmem_helper.c and switched Panfrost driver to use the new common memory shrinker. This was proposed by Thomas Zimmermann in his prototype series that he shared with us in the comment to v1. Note that I only compile-tested the Panfrost driver.
- Shrinker now takes object_name_lock during scan to prevent racing with dma-buf exporting.
- Shrinker now takes vmap_lock during scan to prevent racing with shmem vmap/unmap code.
- Added "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, which I sent out previously as a standalone change, since the drm_gem_shmem_helper.c is now touched by this patchset anyways and it doesn't hurt to group all the patches together.
Dmitry Osipenko (22):
drm/gem: Properly annotate WW context on drm_gem_lock_reservations()
error
drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj() drm/panfrost: Put mapping instead of shmem obj on
panfrost_mmu_map_fault_addr() error
drm/panfrost: Fix shrinker list corruption by madvise IOCTL drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling drm/virtio: Check whether transferred 2D BO is shmem drm/virtio: Unlock reservations on virtio_gpu_object_shmem_init()
error
drm/virtio: Unlock reservations on dma_resv_reserve_fences() error drm/virtio: Use appropriate atomic state in
virtio_gpu_plane_cleanup_fb()
drm/shmem-helper: Add missing vunmap on error drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table() drm/virtio: Simplify error handling of virtio_gpu_object_create() drm/virtio: Improve DMA API usage for shmem BOs
dma-buf: Introduce new locking convention drm/shmem-helper: Don't use vmap_use_count for dma-bufs drm/shmem-helper: Use reservation lock
drm/shmem-helper: Add generic memory shrinker
drm/gem: Add drm_gem_pin_unlocked()
drm/virtio: Support memory shrinking
drm/virtio: Use dev_is_pci()
drm/virtio: Return proper error codes instead of -1 drm/panfrost: Switch to generic memory shrinker
drivers/dma-buf/dma-buf.c | 270 ++++--- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +- drivers/gpu/drm/drm_gem.c | 69 +- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 718 ++++++++++++++---- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 10 +- drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/lima/lima_sched.c | 4 +- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 26 +- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 - .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 122 --- drivers/gpu/drm/panfrost/panfrost_job.c | 18 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 21 +- drivers/gpu/drm/panfrost/panfrost_perfcnt.c | 6 +- drivers/gpu/drm/qxl/qxl_object.c | 17 +- drivers/gpu/drm/qxl/qxl_prime.c | 4 +- drivers/gpu/drm/tegra/gem.c | 4 + drivers/gpu/drm/virtio/virtgpu_drv.c | 53 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 23 +- drivers/gpu/drm/virtio/virtgpu_gem.c | 59 +- drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 + drivers/gpu/drm/virtio/virtgpu_kms.c | 16 +- drivers/gpu/drm/virtio/virtgpu_object.c | 203 +++-- drivers/gpu/drm/virtio/virtgpu_plane.c | 28 +- drivers/gpu/drm/virtio/virtgpu_vq.c | 61 +- .../common/videobuf2/videobuf2-dma-contig.c | 11 +- .../media/common/videobuf2/videobuf2-dma-sg.c | 11 +- .../common/videobuf2/videobuf2-vmalloc.c | 11 +- include/drm/drm_device.h | 4 + include/drm/drm_gem.h | 6 + include/drm/drm_gem_shmem_helper.h | 99 ++- include/linux/dma-buf.h | 14 +- include/uapi/drm/virtgpu_drm.h | 14 + 37 files changed, 1349 insertions(+), 661 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 1/22 drm/gem: Properly annotate WW context on drm_gem_lock_reservations() error
Use ww_acquire_fini() in the error code paths. Otherwise lockdep thinks that lock is held when lock's memory is freed after the drm_gem_lock_reservations() error. The WW needs to be annotated as "freed", which fixes the noisy "WARNING: held lock freed!" splat of VirtIO-GPU driver with CONFIG_DEBUG_MUTEXES=y and enabled lockdep.
--- drivers/gpu/drm/drm_gem.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem.c +2 -2
@@ -1226,7 +1226,7 @@ drm_gem_lock_reservations(struct drm_gem_object **objs, int count, ret =dma_resv_lock_slow_interruptible (obj->resv, acquire_ctx);if (ret) { -ww_acquire_done (acquire_ctx); +ww_acquire_fini (acquire_ctx);return ret; } }
@@ -1251,7 +1251,7 @@ drm_gem_lock_reservations(struct drm_gem_object **objs, int count, goto retry; } -ww_acquire_done (acquire_ctx); +ww_acquire_fini (acquire_ctx);return ret; } }
Signature · 1 line
2.35.3
Re: [PATCH v6 01/22] drm/gem: Properly annotate WW context on drm_gem_lock_reservations() error
Hi,
Quoted 4 lines · Dmitry Osipenko
Use
ww_acquire_fini()in the error code paths. Otherwise lockdep thinks that lock is held when lock's memory is freed after thedrm_gem_lock_reservations()error. The WW needs to be annotated as "freed"
s /WW/ww_acquire_context/ ?
s /"freed"/"released"/ ?
Quoted 4 lines
, which fixes the noisy "WARNING: held lock freed!" splat of VirtIO-GPU driver with
CONFIG_DEBUG_MUTEXES=yand enabled lockdep.
Can you dig up the commit in error and add a Fixes: Tag?
Using that and "dim fixes" will also make the Cc: stable tag a bit more verbose.
With that fixed,
Quoted 27 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/drm_gem.c +2
-2@@ -1226,7 +1226,7 @@ drm_gem_lock_reservations(struct drm_gem_object **objs, int count, ret =dma_resv_lock_slow_interruptible (obj->resv, acquire_ctx);if (ret) { -ww_acquire_done (acquire_ctx); +ww_acquire_fini (acquire_ctx);return ret; } }@@ -1251,7 +1251,7 @@ drm_gem_lock_reservations(struct drm_gem_object **objs, int count, goto retry; } -ww_acquire_done (acquire_ctx); +ww_acquire_fini (acquire_ctx);return ret; } }
Dmitry Osipenko
maintainerRe: [PATCH v6 01/22] drm/gem: Properly annotate WW context on drm_gem_lock_reservations() error
Quoted 17 lines · Thomas Hellström (Intel)
Hi,
Quoted 4 lines · Dmitry Osipenko
Use
ww_acquire_fini()in the error code paths. Otherwise lockdep thinks that lock is held when lock's memory is freed after thedrm_gem_lock_reservations()error. The WW needs to be annotated as "freed"s /WW/ww_acquire_context/ ?
s /"freed"/"released"/ ?Quoted 4 lines
, which fixes the noisy "WARNING: held lock freed!" splat of VirtIO-GPU driver with
CONFIG_DEBUG_MUTEXES=yand enabled lockdep.
Quoted 8 lines
Can you dig up the commit in error and add a Fixes: Tag?
Using that and "dim fixes" will also make the Cc: stable tag a bit more verbose.
With that fixed,
Reviewed-by Thomas Hellström <[email protected]>
I'll update this patch, thank you for taking a look.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerPATCH v6 2/22 drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
In reply to Dmitry OsipenkoDrivers that use drm_gem_mmap() and drm_gem_mmap_obj() helpers don't handle imported dma-bufs properly, which results in mapping of something else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard lockup when userspace writes to the memory mapping of a dma-buf that was imported into Tegra's DRM GEM.
To fix this bug, move mapping of imported dma-bufs to drm_gem_mmap_obj(). Now mmaping of imported dma-bufs works properly for all DRM drivers.
--- drivers/gpu/drm/drm_gem.c | 3 +++ drivers/gpu/drm/drm_gem_shmem_helper.c | 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++ 3 files changed, 7 insertions(+), 9 deletions(-)
drivers/gpu/drm/drm_gem.c +3 -0
@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size,if (obj_size < vma->vm_end - vma->vm_start)return -EINVAL ; +if (obj->import_attach) +return dma_buf_mmap (obj->dma_buf, vma,0 ); +/* Take a ref for this mapping of the object, so that the fault * handler can dereference the mmap offset's pointer to the object. * This reference is cleaned up by the corresponding vm_close
drivers/gpu/drm/drm_gem_shmem_helper.c +0 -9
@@ -609,17 +609,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_vm_ops);*/ int drm_gem_shmem_mmap (struct drm_gem_shmem_object *shmem,struct vm_area_struct *vma) { -struct drm_gem_object *obj = &shmem->base;int ret; -if (obj->import_attach) { -/* Drop the reference drm_gem_mmap_obj() acquired.*/ -drm_gem_object_put (obj); - vma->vm_private_data =NULL ; - -return dma_buf_mmap (obj->dma_buf, vma,0 ); - } - ret =drm_gem_shmem_get_pages (shmem);if (ret) {drm_gem_vm_close (vma);
drivers/gpu/drm/tegra/gem.c +4 -0
@@ -564,6 +564,10 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) {struct tegra_bo *bo =to_tegra_bo (gem); +/* imported dmu-buf is mapped by drm_gem_mmap_obj() */ +if (gem->import_attach) +return 0 ; +if (!bo->pages) {unsigned long vm_pgoff = vma->vm_pgoff;int err;
Signature · 1 line
2.35.3
Re: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Quoted 8 lines · Dmitry Osipenko
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard lockup when userspace writes to the memory mapping of a dma-buf that was imported into Tegra's DRM GEM.To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj(). Now mmaping of imported dma-bufs works properly for all DRM drivers.
Same comment about Fixes: as in patch 1,
Quoted 14 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem.c | 3 +++ drivers/gpu/drm/drm_gem_shmem_helper.c | 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++ 3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0
drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, · line 1042
Quoted 5 lines
drivers/gpu/drm/drm_gem.c +2
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size,if (obj_size < vma->vm_end - vma->vm_start)return -EINVAL ; +if (obj->import_attach) +return dma_buf_mmap (obj->dma_buf, vma,0 );
If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed DMA_BUF_IOCTL_SYNC which is needed before and after cpu access of mmap'ed dma-bufs?
I was under the impression (admittedly without looking) that the few drivers that actually called into dma_buf_mmap() had some private user-mode driver code in place that ensured this happened.
/Thomas
Quoted 41 lines
drivers/gpu/drm/drm_gem.c +1
-0+/* Take a ref for this mapping of the object, so that the fault * handler can dereference the mmap offset's pointer to the object. * This reference is cleaned up by the corresponding vm_close drivers/gpu/drm/drm_gem_shmem_helper.c +0
-9@@ -609,17 +609,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_vm_ops);*/ int drm_gem_shmem_mmap (struct drm_gem_shmem_object *shmem,struct vm_area_struct *vma) { -struct drm_gem_object *obj = &shmem->base;int ret; -if (obj->import_attach) { -/* Drop the reference drm_gem_mmap_obj() acquired.*/ -drm_gem_object_put (obj); - vma->vm_private_data =NULL ; - -return dma_buf_mmap (obj->dma_buf, vma,0 ); - } - ret =drm_gem_shmem_get_pages (shmem);if (ret) {drm_gem_vm_close (vma);drivers/gpu/drm/tegra/gem.c +4
-0@@ -564,6 +564,10 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) {struct tegra_bo *bo =to_tegra_bo (gem); +/* imported dmu-buf is mapped by drm_gem_mmap_obj() */ +if (gem->import_attach) +return 0 ; +if (!bo->pages) {unsigned long vm_pgoff = vma->vm_pgoff;int err;
Dmitry Osipenko
maintainerRe: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Quoted 37 lines · Thomas Hellström (Intel)
Quoted 9 lines · Dmitry Osipenko
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard
lockup when userspace writes to the memory mapping of a dma-buf that was imported into Tegra's DRM GEM.To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj(). Now mmaping of imported dma-bufs works properly for all DRM drivers.Same comment about Fixes: as in patch 1,
Quoted 14 lines
Signed-off-by Dmitry Osipenko <[email protected]>---
drivers/gpu/drm/drm_gem.c | 3 +++drivers/gpu/drm/drm_gem_shmem_helper.c| 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object *obj, · line 1042
Quoted 5 lines
drivers/gpu/drm/drm_gem.c +1
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, if (obj_size < vma->vm_end - vma->vm_start) return -EINVAL; + if (obj->import_attach) + return dma_buf_mmap(obj->dma_buf, vma, 0);If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed
DMA_BUF_IOCTL_SYNCwhich is needed before and after cpu access of mmap'ed dma-bufs?
Quoted 3 lines
I was under the impression (admittedly without looking) that the few drivers that actually called into
dma_buf_mmap()had some private user-mode driver code in place that ensured this happened.
Since it's a userspace who does the mapping, then it should be a responsibility of userspace to do all the necessary syncing. I'm not sure whether anyone in userspace really needs to map imported dma-bufs in practice. Nevertheless, this use-case is broken and should be fixed by either allowing to do the mapping or prohibiting it.
Signature · 2 lines
Best regards, Dmitry
Re: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Quoted 35 lines · Dmitry Osipenko
Quoted 34 lines · Thomas Hellström (Intel)
Quoted 9 lines · Dmitry Osipenko
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard
lockup when userspace writes to the memory mapping of a dma-buf that was imported into Tegra's DRM GEM.To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj(). Now mmaping of imported dma-bufs works properly for all DRM drivers.Same comment about Fixes: as in patch 1,
Quoted 13 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem.c | 3 +++ drivers/gpu/drm/drm_gem_shmem_helper.c | 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++ 3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object *obj, · line 1042
Quoted 5 lines
drivers/gpu/drm/drm_gem.c +1
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, if (obj_size < vma->vm_end - vma->vm_start) return -EINVAL; + if (obj->import_attach) + return dma_buf_mmap(obj->dma_buf, vma, 0);If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed
DMA_BUF_IOCTL_SYNCwhich is needed before and after cpu access of mmap'ed dma-bufs?
Quoted 5 lines
Quoted 3 lines
I was under the impression (admittedly without looking) that the few drivers that actually called into
dma_buf_mmap()had some private user-mode driver code in place that ensured this happened.Since it's a userspace who does the mapping, then it should be a responsibility of userspace to do all the necessary syncing.
Sure, but nothing prohibits user-space to ignore the syncing thinking "It works anyway", testing those drivers where the syncing is a NOP. And when a driver that finally needs syncing is tested it's too late to fix all broken user-space.
Quoted 5 lines
I'm not
sure whether anyone in userspace really needs to map imported dma-bufs in practice. Nevertheless, this use-case is broken and should be fixed by either allowing to do the mapping or prohibiting it.
Then I'd vote for prohibiting it, at least for now. And for the future moving forward we could perhaps revisit the dma-buf need for syncing, requiring those drivers that actually need it to implement emulated coherent memory which can be done not too inefficiently (vmwgfx being one example).
/Thomas
Dmitry Osipenko
maintainerRe: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Quoted 56 lines · Thomas Hellström (Intel)
Quoted 38 lines · Dmitry Osipenko
Quoted 37 lines · Thomas Hellström (Intel)
Quoted 9 lines · Dmitry Osipenko
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something
else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard
lockup when userspace writes to the memory mapping of a dma-buf that was
imported into Tegra's DRM GEM.Quoted 3 lines
To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj().
Now mmaping of imported dma-bufs works properly for all DRM drivers.Same comment about Fixes: as in patch 1,
Quoted 13 lines
Signed-off-by Dmitry Osipenko <[email protected]>---
drivers/gpu/drm/drm_gem.c | 3 +++drivers/gpu/drm/drm_gem_shmem_helper.c| 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object *obj, · line 1042
Quoted 5 lines
drivers/gpu/drm/drm_gem.c +1
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, if (obj_size < vma->vm_end - vma->vm_start) return -EINVAL; + if (obj->import_attach) + return dma_buf_mmap(obj->dma_buf, vma, 0);If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed
DMA_BUF_IOCTL_SYNCwhich is needed before and after cpu access of mmap'ed dma-bufs?Quoted 5 lines
Quoted 3 lines
I was under the impression (admittedly without looking) that the few drivers that actually called into
dma_buf_mmap()had some private user-mode driver code in place that ensured this happened.Since it's a userspace who does the mapping, then it should be a responsibility of userspace to do all the necessary syncing.
Sure, but nothing prohibits user-space to ignore the syncing thinking "It works anyway", testing those drivers where the syncing is a NOP. And when a driver that finally needs syncing is tested it's too late to fix all broken user-space.
Quoted 5 lines
I'm not
sure whether anyone in userspace really needs to map imported dma-bufs in practice. Nevertheless, this use-case is broken and should be fixed by either allowing to do the mapping or prohibiting it.
Quoted 5 lines
Then I'd vote for prohibiting it, at least for now. And for the future moving forward we could perhaps revisit the dma-buf need for syncing, requiring those drivers that actually need it to implement emulated coherent memory which can be done not too inefficiently (vmwgfx being one example).
Alright, I'll change it to prohibit the mapping. This indeed should be a better option.
Signature · 2 lines
Best regards, Dmitry
Christian König
maintainerRe: [Linaro-mm-sig] Re: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Am 30.06.22 um 01:06 schrieb Dmitry Osipenko:
Quoted 55 lines · Thomas Hellström (Intel)
Quoted 54 lines · Dmitry Osipenko
Quoted 38 lines · Thomas Hellström (Intel)
Quoted 37 lines · Dmitry Osipenko
Quoted 9 lines
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something
else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard
lockup when userspace writes to the memory mapping of a dma-buf that was
imported into Tegra's DRM GEM.Quoted 3 lines
To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj().
Now mmaping of imported dma-bufs works properly for all DRM drivers.Same comment about Fixes: as in patch 1,
Quoted 13 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem.c | 3 +++ drivers/gpu/drm/drm_gem_shmem_helper.c | 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++ 3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object *obj, · line 1042
Quoted 5 lines
drivers/gpu/drm/drm_gem.c +1
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, if (obj_size < vma->vm_end - vma->vm_start) return -EINVAL; + if (obj->import_attach) + return dma_buf_mmap(obj->dma_buf, vma, 0);If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed
DMA_BUF_IOCTL_SYNCwhich is needed before and after cpu access of mmap'ed dma-bufs?Quoted 5 lines
Quoted 3 lines
I was under the impression (admittedly without looking) that the few drivers that actually called into
dma_buf_mmap()had some private user-mode driver code in place that ensured this happened.Since it's a userspace who does the mapping, then it should be a responsibility of userspace to do all the necessary syncing.
Sure, but nothing prohibits user-space to ignore the syncing thinking "It works anyway", testing those drivers where the syncing is a NOP. And when a driver that finally needs syncing is tested it's too late to fix all broken user-space.
Quoted 5 lines
I'm not
sure whether anyone in userspace really needs to map imported dma-bufs in practice. Nevertheless, this use-case is broken and should be fixed by either allowing to do the mapping or prohibiting it.
Quoted 7 lines
Quoted 5 lines
Then I'd vote for prohibiting it, at least for now. And for the future moving forward we could perhaps revisit the dma-buf need for syncing, requiring those drivers that actually need it to implement emulated coherent memory which can be done not too inefficiently (vmwgfx being one example).
Alright, I'll change it to prohibit the mapping. This indeed should be a better option.
Oh, yes please. But I would expect that some people start screaming.
Over time I've got tons of TTM patches because people illegally tried to mmap() imported DMA-bufs in their driver.
Anyway this is probably the right thing to do and we can work on fixing the fallout later on.
Regards,
Christian.
Dmitry Osipenko
maintainerRe: [Linaro-mm-sig] Re: [PATCH v6 02/22] drm/gem: Move mapping of imported dma-bufs to drm_gem_mmap_obj()
Quoted 65 lines · Christian König
Am 30.06.22 um 01:06 schrieb Dmitry Osipenko:
Quoted 56 lines · Thomas Hellström (Intel)
Quoted 55 lines · Dmitry Osipenko
Quoted 39 lines · Thomas Hellström (Intel)
Quoted 38 lines · Dmitry Osipenko
Quoted 9 lines
Drivers that use
drm_gem_mmap()anddrm_gem_mmap_obj()helpers don't handle imported dma-bufs properly, which results in mapping of something
else than the imported dma-buf. For example, on NVIDIA Tegra we get a hard
lockup when userspace writes to the memory mapping of a dma-buf that was
imported into Tegra's DRM GEM.Quoted 3 lines
To fix this bug, move mapping of imported dma-bufs to
drm_gem_mmap_obj().
Now mmaping of imported dma-bufs works properly for all DRM drivers.Same comment about Fixes: as in patch 1,
Quoted 13 lines
Signed-off-by Dmitry Osipenko <[email protected]>---
drivers/gpu/drm/drm_gem.c | 3 +++drivers/gpu/drm/drm_gem_shmem_helper.c| 9 --------- drivers/gpu/drm/tegra/gem.c | 4 ++++3 files changed, 7 insertions(+), 9 deletions(-)drivers/gpu/drm/drm_gem.c +0
-0drivers/gpu/drm/drm_gem.c int drm_gem_mmap_obj(struct drm_gem_object · line 1043
Quoted 6 lines
drivers/gpu/drm/drm_gem.c +1
-0@@ -1038,6 +1038,9 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, if (obj_size < vma->vm_end - vma->vm_start) return -EINVAL; + if (obj->import_attach) + return dma_buf_mmap(obj->dma_buf, vma, 0);If we start enabling mmaping of imported dma-bufs on a majority of drivers in this way, how do we ensure that user-space is not blindly using the object mmap without calling the needed
DMA_BUF_IOCTL_SYNCwhich is needed before and after cpu access of mmap'ed dma-bufs?Quoted 5 lines
Quoted 3 lines
I was under the impression (admittedly without looking) that the few drivers that actually called into
dma_buf_mmap()had some private user-mode driver code in place that ensured this happened.Since it's a userspace who does the mapping, then it should be a responsibility of userspace to do all the necessary syncing.
Sure, but nothing prohibits user-space to ignore the syncing thinking "It works anyway", testing those drivers where the syncing is a NOP. And when a driver that finally needs syncing is tested it's too late to fix all broken user-space.
Quoted 5 lines
I'm not
sure whether anyone in userspace really needs to map imported dma-bufs in practice. Nevertheless, this use-case is broken and should be fixed by either allowing to do the mapping or prohibiting it.Quoted 7 lines
Quoted 5 lines
Then I'd vote for prohibiting it, at least for now. And for the future moving forward we could perhaps revisit the dma-buf need for syncing, requiring those drivers that actually need it to implement emulated coherent memory which can be done not too inefficiently (vmwgfx being one example).
Alright, I'll change it to prohibit the mapping. This indeed should be a better option.
Quoted 7 lines
Oh, yes please. But I would expect that some people start screaming.
Over time I've got tons of TTM patches because people illegally tried to
mmap()imported DMA-bufs in their driver.Anyway this is probably the right thing to do and we can work on fixing the fallout later on.
I already sent out the patch [1] that prohibits the mapping. Would be great if you all could take a look and give a r-b, thanks in advance.
[1] https://patchwork.freedesktop.org/patch/492148/
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerPATCH v6 3/22 drm/panfrost: Put mapping instead of shmem obj on panfrost_mmu_map_fault_addr() error
In reply to Dmitry OsipenkoWhen panfrost_mmu_map_fault_addr() fails, the BO's mapping should be unreferenced and not the shmem object which backs the mapping.
--- drivers/gpu/drm/panfrost/panfrost_mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/panfrost/panfrost_mmu.c +1 -1
@@ -518,7 +518,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, err_pages:drm_gem_shmem_put_pages (&bo->base); err_bo: -drm_gem_object_put (&bo->base.base); +panfrost_gem_mapping_put (bomapping);return ret; }
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 4/22 drm/panfrost: Fix shrinker list corruption by madvise IOCTL
In reply to Dmitry OsipenkoCalling madvise IOCTL twice on BO causes memory shrinker list corruption and crashes kernel because BO is already on the list and it's added to the list again, while BO should be removed from from the list before it's re-added. Fix it.
--- drivers/gpu/drm/panfrost/panfrost_drv.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/panfrost/panfrost_drv.c +2 -2
@@ -433,8 +433,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (args->retained) {if (args->madv ==PANFROST_MADV_DONTNEED ) -list_add_tail (&bo->base.madv_list, - &pfdev->shrinker_list); +list_move_tail (&bo->base.madv_list, + &pfdev->shrinker_list);else if (args->madv ==PANFROST_MADV_WILLNEED )list_del_init (&bo->base.madv_list); }
Signature · 1 line
2.35.3
Re: [PATCH v6 04/22] drm/panfrost: Fix shrinker list corruption by madvise IOCTL
Quoted 30 lines · Dmitry Osipenko
Calling madvise IOCTL twice on BO causes memory shrinker list corruption and crashes kernel because BO is already on the list and it's added to the list again, while BO should be removed from from the list before it's re-added. Fix it.
Fixes 013b65101315 ("drm/panfrost: Add madvise and shrinker support")Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/panfrost/panfrost_drv.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/panfrost/panfrost_drv.c +2
-2@@ -433,8 +433,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (args->retained) {if (args->madv ==PANFROST_MADV_DONTNEED ) -list_add_tail (&bo->base.madv_list, - &pfdev->shrinker_list); +list_move_tail (&bo->base.madv_list, + &pfdev->shrinker_list);else if (args->madv ==PANFROST_MADV_WILLNEED )list_del_init (&bo->base.madv_list); }Signature · 1 line
2.35.3
Re: [PATCH v6 04/22] drm/panfrost: Fix shrinker list corruption by madvise IOCTL
In reply to Dmitry OsipenkoQuoted 8 lines · Dmitry Osipenko
Calling madvise IOCTL twice on BO causes memory shrinker list corruption and crashes kernel because BO is already on the list and it's added to the list again, while BO should be removed from from the list before it's re-added. Fix it.
Fixes 013b65101315 ("drm/panfrost: Add madvise and shrinker support")Signed-off-by Dmitry Osipenko <[email protected]>
Quoted 19 lines
--- drivers/gpu/drm/panfrost/panfrost_drv.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)drivers/gpu/drm/panfrost/panfrost_drv.c +2
-2@@ -433,8 +433,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (args->retained) {if (args->madv ==PANFROST_MADV_DONTNEED ) -list_add_tail (&bo->base.madv_list, - &pfdev->shrinker_list); +list_move_tail (&bo->base.madv_list, + &pfdev->shrinker_list);else if (args->madv ==PANFROST_MADV_WILLNEED )list_del_init (&bo->base.madv_list); }
Dmitry Osipenko
maintainerPATCH v6 5/22 drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling
In reply to Dmitry Osipenkodrm_gem_shmem_get_sg_table() never ever returned NULL on error. Correct the error handling to avoid crash on OOM.
--- drivers/gpu/drm/virtio/virtgpu_object.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/virtio/virtgpu_object.c +4 -2
@@ -168,9 +168,11 @@ static int virtio_gpu_object_shmem_init(struct virtio_gpu_device *vgdev,* since virtio_gpu doesn't support dma-buf import from other devices. */ shmem->pages =drm_gem_shmem_get_sg_table (&bo->base); -if (!shmem->pages) { + ret =PTR_ERR_OR_ZERO (shmem->pages); +if (ret) {drm_gem_shmem_unpin (&bo->base); -return -EINVAL ; + shmem->pages =NULL ; +return ret; }if (use_dma_api) {
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 6/22 drm/virtio: Check whether transferred 2D BO is shmem
In reply to Dmitry OsipenkoTransferred 2D BO always must be a shmem BO. Add check for that to prevent NULL dereference if userspace passes a VRAM BO.
--- drivers/gpu/drm/virtio/virtgpu_vq.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_vq.c +1 -1
@@ -595,7 +595,7 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,bool use_dma_api = !virtio_has_dma_quirk (vgdev->vdev);struct virtio_gpu_object_shmem *shmem =to_virtio_gpu_shmem (bo); -if (use_dma_api) +if (virtio_gpu_is_shmem (bo) && use_dma_api)dma_sync_sgtable_for_device (vgdev->vdev->dev.parent, shmem->pages,DMA_TO_DEVICE );
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 7/22 drm/virtio: Unlock reservations on virtio_gpu_object_shmem_init() error
In reply to Dmitry OsipenkoUnlock reservations in the error code path of virtio_gpu_object_create() to silence debug warning splat produced by ww_mutex_destroy(&obj->lock) when GEM is released with the held lock.
--- drivers/gpu/drm/virtio/virtgpu_object.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/virtio/virtgpu_object.c +2 -0
@@ -250,6 +250,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev, ret =virtio_gpu_object_shmem_init (vgdev, bo, &ents, &nents);if (ret !=0 ) { +if (fence) +virtio_gpu_array_unlock_resv (objs);virtio_gpu_array_put_free (objs);virtio_gpu_free_object (&shmem_obj->base);return ret;
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 8/22 drm/virtio: Unlock reservations on dma_resv_reserve_fences() error
In reply to Dmitry OsipenkoUnlock reservations on dma_resv_reserve_fences() error to fix recursive locking of the reservations when this error happens.
--- drivers/gpu/drm/virtio/virtgpu_gem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_gem.c +3 -1
@@ -228,8 +228,10 @@ int virtio_gpu_array_lock_resv(struct virtio_gpu_object_array *objs)for (i =0 ; i < objs->nents; ++i) { ret =dma_resv_reserve_fences (objs->objs[i]->resv,1 ); -if (ret) +if (ret) { +virtio_gpu_array_unlock_resv (objs);return ret; + } }return ret; }
Signature · 1 line
2.35.3
Re: [PATCH v6 08/22] drm/virtio: Unlock reservations on dma_resv_reserve_fences() error
Quoted 2 lines · Dmitry Osipenko
Unlock reservations on
dma_resv_reserve_fences()error to fix recursive locking of the reservations when this error happens.
Fixes:
Quoted 1 line
With that fixed,
Quoted 21 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/virtio/virtgpu_gem.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/gpu/drm/virtio/virtgpu_gem.c +3
-1@@ -228,8 +228,10 @@ int virtio_gpu_array_lock_resv(struct virtio_gpu_object_array *objs)for (i =0 ; i < objs->nents; ++i) { ret =dma_resv_reserve_fences (objs->objs[i]->resv,1 ); -if (ret) +if (ret) { +virtio_gpu_array_unlock_resv (objs);return ret; + } }return ret; }
Dmitry Osipenko
maintainerPATCH v6 9/22 drm/virtio: Use appropriate atomic state in virtio_gpu_plane_cleanup_fb()
In reply to Dmitry OsipenkoMake virtio_gpu_plane_cleanup_fb() to clean the state which DRM core wants to clean up and not the current plane's state. Normally the older atomic state is cleaned up, but the newer state could also be cleaned up in case of aborted commits.
--- drivers/gpu/drm/virtio/virtgpu_plane.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/virtio/virtgpu_plane.c +3 -3
@@ -266,14 +266,14 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane, }static void virtio_gpu_plane_cleanup_fb (struct drm_plane *plane, -struct drm_plane_state *old_state) +struct drm_plane_state *state) {struct virtio_gpu_framebuffer *vgfb; -if (!plane->state->fb) +if (!state->fb)return ; - vgfb =to_virtio_gpu_framebuffer (plane->state->fb); + vgfb =to_virtio_gpu_framebuffer (state->fb);if (vgfb->fence) {dma_fence_put (&vgfb->fence->f); vgfb->fence =NULL ;
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 10/22 drm/shmem-helper: Add missing vunmap on error
In reply to Dmitry OsipenkoThe vmapping of dma-buf may succeed, but DRM SHMEM rejects the iomem mappings, and thus, drm_gem_shmem_vmap_locked() should unvmap the iomem before erroring out.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 1 + 1 file changed, 1 insertion(+)
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -0
@@ -302,6 +302,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, ret =dma_buf_vmap (obj->import_attach->dmabuf, map);if (!ret) {if (WARN_ON (map->is_iomem)) { +dma_buf_vunmap (obj->import_attach->dmabuf, map); ret = -EIO ; goto err_put_pages; }
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 11/22 drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table()
In reply to Dmitry Osipenkodrm_gem_shmem_get_sg_table() never returns NULL on error, but a ERR_PTR. Correct the doc comment which says that it returns NULL on error.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +4 -2
@@ -654,7 +654,8 @@ EXPORT_SYMBOL(drm_gem_shmem_print_info);* drm_gem_shmem_get_pages_sgt() instead. * * Returns: -* A pointer to the scatter/gather table of pinned pages or NULL on failure. +* A pointer to the scatter/gather table of pinned pages or an ERR_PTR()-encoded +* error code on failure. */ struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem) {
@@ -680,7 +681,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);* drm_gem_shmem_get_sg_table() should not be directly called by drivers. * * Returns: -* A pointer to the scatter/gather table of pinned pages or errno on failure. +* A pointer to the scatter/gather table of pinned pages or an ERR_PTR()-encoded +* error code on failure. */ struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem) {
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 12/22 drm/virtio: Simplify error handling of virtio_gpu_object_create()
In reply to Dmitry OsipenkoChange the order of SHMEM initialization and reservation locking to make code cleaner and to prepare for transitioning of the common GEM SHMEM code to use the GEM's reservation lock instead of the shmem.page_lock.
There is no need to lock reservation during allocation of the SHMEM pages because the lock is needed only to avoid racing with the async host-side allocation. Hence we can safely move the SHMEM initialization out of the reservation lock.
--- drivers/gpu/drm/virtio/virtgpu_object.c | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-)
drivers/gpu/drm/virtio/virtgpu_object.c +4 -9
@@ -236,6 +236,10 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev, bo->dumb = params->dumb; + ret =virtio_gpu_object_shmem_init (vgdev, bo, &ents, &nents); +if (ret !=0 ) + goto err_put_id; +if (fence) { ret = -ENOMEM ; objs =virtio_gpu_array_alloc (1 );
@@ -248,15 +252,6 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev, goto err_put_objs; } - ret =virtio_gpu_object_shmem_init (vgdev, bo, &ents, &nents); -if (ret !=0 ) { -if (fence) -virtio_gpu_array_unlock_resv (objs); -virtio_gpu_array_put_free (objs); -virtio_gpu_free_object (&shmem_obj->base); -return ret; - } -if (params->blob) {if (params->blob_mem ==VIRTGPU_BLOB_MEM_GUEST ) bo->guest_blob = true;
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 13/22 drm/virtio: Improve DMA API usage for shmem BOs
In reply to Dmitry OsipenkoDRM API requires the DRM's driver to be backed with the device that can be used for generic DMA operations. The VirtIO-GPU device can't perform DMA operations if it uses PCI transport because PCI device driver creates a virtual VirtIO-GPU device that isn't associated with the PCI. Use PCI's GPU device for the DRM's device instead of the VirtIO-GPU device and drop DMA-related hacks from the VirtIO-GPU driver.
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 51 ++++++---------------- drivers/gpu/drm/virtio/virtgpu_drv.h | 5 +-- drivers/gpu/drm/virtio/virtgpu_kms.c | 7 ++-- drivers/gpu/drm/virtio/virtgpu_object.c | 56 +++++-------------------- drivers/gpu/drm/virtio/virtgpu_vq.c | 13 +++--- 5 files changed, 32 insertions(+), 100 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +12 -39
@@ -46,12 +46,11 @@ static int virtio_gpu_modeset = -1;MODULE_PARM_DESC (modeset,"Disable/Enable modesetting" );module_param_named (modeset, virtio_gpu_modeset, int,0400 ); -static int virtio_gpu_pci_quirk (struct drm_device *dev,struct virtio_device *vdev) +static int virtio_gpu_pci_quirk (struct drm_device *dev) { -struct pci_dev *pdev =to_pci_dev (vdev->dev.parent); +struct pci_dev *pdev =to_pci_dev (dev->dev);const char *pname =dev_name (&pdev->dev);bool vga = (pdev->class >>8 ) ==PCI_CLASS_DISPLAY_VGA ; -char unique[20 ];int ret; DRM_INFO("pci: %s detected at %s\n",
@@ -63,39 +62,7 @@ static int virtio_gpu_pci_quirk(struct drm_device *dev, struct virtio_device *vdreturn ret; } -/* -* Normally the drm_dev_set_unique() call is done by core DRM. -* The following comment covers, why virtio cannot rely on it. -* -* Unlike the other virtual GPU drivers, virtio abstracts the -* underlying bus type by using struct virtio_device. -* -* Hence the dev_is_pci() check, used in core DRM, will fail -* and the unique returned will be the virtio_device "virtio0", -* while a "pci:..." one is required. -* -* A few other ideas were considered: -* - Extend the dev_is_pci() check [in drm_set_busid] to -* consider virtio. -* Seems like a bigger hack than what we have already. -* -* - Point drm_device::dev to the parent of the virtio_device -* Semantic changes: -* * Using the wrong device for i2c, framebuffer_alloc and -* prime import. -* Visual changes: -* * Helpers such as DRM_DEV_ERROR, dev_info, drm_printer, -* will print the wrong information. -* -* We could address the latter issues, by introducing -* drm_device::bus_dev, ... which would be used solely for this. -* -* So for the moment keep things as-is, with a bulky comment -* for the next person who feels like removing this -* drm_dev_set_unique() quirk. -*/ -snprintf (unique,sizeof (unique),"pci:%s" , pname); -return drm_dev_set_unique (dev, unique); +return 0 ; }static int virtio_gpu_probe (struct virtio_device *vdev)
@@ -109,18 +76,24 @@ static int virtio_gpu_probe(struct virtio_device *vdev)if (virtio_gpu_modeset ==0 )return -EINVAL ; - dev =drm_dev_alloc (&driver, &vdev->dev); +/* +* The virtio-gpu device is a virtual device that doesn't have DMA +* ops assigned to it, nor DMA mask set and etc. Its parent device +* is actual GPU device we want to use it for the DRM's device in +* order to benefit from using generic DRM APIs. +*/ + dev =drm_dev_alloc (&driver, vdev->dev.parent);if (IS_ERR (dev))return PTR_ERR (dev); vdev->priv = dev;if (!strcmp (vdev->dev.parent->bus->name,"pci" )) { - ret =virtio_gpu_pci_quirk (dev, vdev); + ret =virtio_gpu_pci_quirk (dev);if (ret) goto err_free; } - ret =virtio_gpu_init (dev); + ret =virtio_gpu_init (vdev, dev);if (ret) goto err_free;
drivers/gpu/drm/virtio/virtgpu_drv.h +1 -4
@@ -100,8 +100,6 @@ struct virtio_gpu_object {struct virtio_gpu_object_shmem {struct virtio_gpu_object base; -struct sg_table *pages; -uint32_t mapped; };struct virtio_gpu_object_vram {
@@ -214,7 +212,6 @@ struct virtio_gpu_drv_cap_cache { };struct virtio_gpu_device { -struct device *dev;struct drm_device *ddev;struct virtio_device *vdev;
@@ -282,7 +279,7 @@ extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];void virtio_gpu_create_context (struct drm_device *dev,struct drm_file *file);/* virtgpu_kms.c */ -int virtio_gpu_init (struct drm_device *dev); +int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev);void virtio_gpu_deinit (struct drm_device *dev);void virtio_gpu_release (struct drm_device *dev);int virtio_gpu_driver_open (struct drm_device *dev,struct drm_file *file);
drivers/gpu/drm/virtio/virtgpu_kms.c +3 -4
@@ -110,7 +110,7 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev, vgdev->num_capsets = num_capsets; } -int virtio_gpu_init (struct drm_device *dev) +int virtio_gpu_init (struct virtio_device *vdev,struct drm_device *dev) {static vq_callback_t *callbacks[] = { virtio_gpu_ctrl_ack, virtio_gpu_cursor_ack
@@ -123,7 +123,7 @@ int virtio_gpu_init(struct drm_device *dev)u32 num_scanouts, num_capsets;int ret =0 ; -if (!virtio_has_feature (dev_to_virtio (dev->dev),VIRTIO_F_VERSION_1 )) +if (!virtio_has_feature (vdev,VIRTIO_F_VERSION_1 ))return -ENODEV ; vgdev =kzalloc (sizeof (struct virtio_gpu_device ),GFP_KERNEL );
@@ -132,8 +132,7 @@ int virtio_gpu_init(struct drm_device *dev) vgdev->ddev = dev; dev->dev_private = vgdev; - vgdev->vdev =dev_to_virtio (dev->dev); - vgdev->dev = dev->dev; + vgdev->vdev = vdev;spin_lock_init (&vgdev->display_info_lock);spin_lock_init (&vgdev->resource_export_lock);
drivers/gpu/drm/virtio/virtgpu_object.c +11 -45
82 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_vq.c +5 -8
@@ -593,11 +593,10 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,struct virtio_gpu_transfer_to_host_2d *cmd_p;struct virtio_gpu_vbuffer *vbuf;bool use_dma_api = !virtio_has_dma_quirk (vgdev->vdev); -struct virtio_gpu_object_shmem *shmem =to_virtio_gpu_shmem (bo);if (virtio_gpu_is_shmem (bo) && use_dma_api) -dma_sync_sgtable_for_device (vgdev->vdev->dev.parent, - shmem->pages,DMA_TO_DEVICE ); +dma_sync_sgtable_for_device (&vgdev->vdev->dev, + bo->base.sgt,DMA_TO_DEVICE ); cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p));memset (cmd_p,0 ,sizeof (*cmd_p));
@@ -1017,11 +1016,9 @@ void virtio_gpu_cmd_transfer_to_host_3d(struct virtio_gpu_device *vgdev,struct virtio_gpu_vbuffer *vbuf;bool use_dma_api = !virtio_has_dma_quirk (vgdev->vdev); -if (virtio_gpu_is_shmem (bo) && use_dma_api) { -struct virtio_gpu_object_shmem *shmem =to_virtio_gpu_shmem (bo); -dma_sync_sgtable_for_device (vgdev->vdev->dev.parent, - shmem->pages,DMA_TO_DEVICE ); - } +if (virtio_gpu_is_shmem (bo) && use_dma_api) +dma_sync_sgtable_for_device (&vgdev->vdev->dev, + bo->base.sgt,DMA_TO_DEVICE ); cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p));memset (cmd_p,0 ,sizeof (*cmd_p));
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 14/22 dma-buf: Introduce new locking convention
In reply to Dmitry OsipenkoAll dma-bufs have dma-reservation lock that allows drivers to perform exclusive operations over shared dma-bufs. Today's dma-buf API has incomplete locking specification, which creates dead lock situation for dma-buf importers and exporters that don't coordinate theirs locks.
This patch introduces new locking convention for dma-buf users. From now on all dma-buf importers are responsible for holding dma-buf reservation lock around operations performed over dma-bufs.
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers that need to manage imported dma-bufs under the held lock.
3. Converting all drivers to the new locking scheme.
--- drivers/dma-buf/dma-buf.c | 270 +++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +- drivers/gpu/drm/drm_gem.c | 33 +++ drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 10 +- drivers/gpu/drm/qxl/qxl_object.c | 17 +- drivers/gpu/drm/qxl/qxl_prime.c | 4 +- .../common/videobuf2/videobuf2-dma-contig.c | 11 +- .../media/common/videobuf2/videobuf2-dma-sg.c | 11 +- .../common/videobuf2/videobuf2-vmalloc.c | 11 +- include/drm/drm_gem.h | 3 + include/linux/dma-buf.h | 14 +- 13 files changed, 241 insertions(+), 159 deletions(-)
drivers/dma-buf/dma-buf.c +172 -98
@@ -552,7 +552,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) file->f_mode |=FMODE_LSEEK ; dmabuf->file = file; -mutex_init (&dmabuf->lock);INIT_LIST_HEAD (&dmabuf->attachments);mutex_lock (&db_list.lock);
@@ -737,14 +736,14 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, attach->importer_ops = importer_ops; attach->importer_priv = importer_priv; +dma_resv_lock (dmabuf->resv,NULL ); +if (dmabuf->ops->attach) { ret = dmabuf->ops->attach (dmabuf, attach);if (ret) goto err_attach; } -dma_resv_lock (dmabuf->resv,NULL );list_add (&attach->node, &dmabuf->attachments); -dma_resv_unlock (dmabuf->resv);/* When either the importer or the exporter can't handle dynamic * mappings we cache the mapping here to avoid issues with the
@@ -755,7 +754,6 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,struct sg_table *sgt;if (dma_buf_is_dynamic (attach->dmabuf)) { -dma_resv_lock (attach->dmabuf->resv,NULL ); ret = dmabuf->ops->pin (attach);if (ret) goto err_unlock;
@@ -768,15 +766,16 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, ret =PTR_ERR (sgt); goto err_unpin; } -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_unlock (attach->dmabuf->resv); attach->sgt = sgt; attach->dir =DMA_BIDIRECTIONAL ; } +dma_resv_unlock (dmabuf->resv); +return attach; err_attach: +dma_resv_unlock (attach->dmabuf->resv);kfree (attach);return ERR_PTR (ret);
@@ -785,10 +784,10 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, dmabuf->ops->unpin (attach); err_unlock: -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_unlock (attach->dmabuf->resv); +dma_resv_unlock (dmabuf->resv);dma_buf_detach (dmabuf, attach); +return ERR_PTR (ret); }EXPORT_SYMBOL_NS_GPL (dma_buf_dynamic_attach,DMA_BUF );
@@ -832,24 +831,23 @@ void dma_buf_detach(struct dma_buf *dmabuf, struct dma_buf_attachment *attach)if (WARN_ON (!dmabuf || !attach))return ; -if (attach->sgt) { -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_lock (attach->dmabuf->resv,NULL ); +if (WARN_ON (dmabuf != attach->dmabuf)) +return ; +dma_resv_lock (dmabuf->resv,NULL ); + +if (attach->sgt) {__unmap_dma_buf (attach, attach->sgt, attach->dir); -if (dma_buf_is_dynamic (attach->dmabuf)) { +if (dma_buf_is_dynamic (attach->dmabuf)) dmabuf->ops->unpin (attach); -dma_resv_unlock (attach->dmabuf->resv); - } } -dma_resv_lock (dmabuf->resv,NULL );list_del (&attach->node); -dma_resv_unlock (dmabuf->resv);if (dmabuf->ops->detach) dmabuf->ops->detach (dmabuf, attach); +dma_resv_unlock (dmabuf->resv);kfree (attach); }EXPORT_SYMBOL_NS_GPL (dma_buf_detach,DMA_BUF );
@@ -906,28 +904,18 @@ void dma_buf_unpin(struct dma_buf_attachment *attach)EXPORT_SYMBOL_NS_GPL (dma_buf_unpin,DMA_BUF );/** -* dma_buf_map_attachment - Returns the scatterlist table of the attachment; +* dma_buf_map_attachment_locked - Returns the scatterlist table of the attachment; * mapped into _device_ address space. Is a wrapper for map_dma_buf() of the * dma_buf_ops. * @attach: [in] attachment whose scatterlist is to be returned * @direction: [in] direction of DMA transfer * -* Returns sg_table containing the scatterlist to be returned; returns ERR_PTR -* on error. May return -EINTR if it is interrupted by a signal. -* -* On success, the DMA addresses and lengths in the returned scatterlist are -* PAGE_SIZE aligned. -* -* A mapping must be unmapped by using dma_buf_unmap_attachment(). Note that -* the underlying backing storage is pinned for as long as a mapping exists, -* therefore users/importers should not hold onto a mapping for undue amounts of -* time. +* Locked variant of dma_buf_map_attachment(). * -* Important: Dynamic importers must wait for the exclusive fence of the struct -* dma_resv attached to the DMA-BUF first. +* Caller is responsible for holding dmabuf's reservation lock. */ -struct sg_table *dma_buf_map_attachment (struct dma_buf_attachment *attach, -enum dma_data_direction direction) +struct sg_table *dma_buf_map_attachment_locked (struct dma_buf_attachment *attach, +enum dma_data_direction direction) {struct sg_table *sg_table;int r;
@@ -937,8 +925,7 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach,if (WARN_ON (!attach || !attach->dmabuf))return ERR_PTR (-EINVAL ); -if (dma_buf_attachment_is_dynamic (attach)) -dma_resv_assert_held (attach->dmabuf->resv); +dma_resv_assert_held (attach->dmabuf->resv);if (attach->sgt) {/*
@@ -953,7 +940,6 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach, }if (dma_buf_is_dynamic (attach->dmabuf)) { -dma_resv_assert_held (attach->dmabuf->resv);if (!IS_ENABLED (CONFIG_DMABUF_MOVE_NOTIFY )) { r = attach->dmabuf->ops->pin (attach);if (r)
@@ -993,42 +979,101 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach,#endif /* CONFIG_DMA_API_DEBUG */ return sg_table; } -EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment,DMA_BUF ); +EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment_locked,DMA_BUF );/** -* dma_buf_unmap_attachment - unmaps and decreases usecount of the buffer;might -* deallocate the scatterlist associated. Is a wrapper for unmap_dma_buf() of +* dma_buf_map_attachment - Returns the scatterlist table of the attachment; +* mapped into _device_ address space. Is a wrapper for map_dma_buf() of the * dma_buf_ops. -* @attach: [in] attachment to unmap buffer from -* @sg_table: [in] scatterlist info of the buffer to unmap -* @direction: [in] direction of DMA transfer +* @attach: [in] attachment whose scatterlist is to be returned +* @direction: [in] direction of DMA transfer * -* This unmaps a DMA mapping for @attached obtained by dma_buf_map_attachment(). +* Returns sg_table containing the scatterlist to be returned; returns ERR_PTR +* on error. May return -EINTR if it is interrupted by a signal. +* +* On success, the DMA addresses and lengths in the returned scatterlist are +* PAGE_SIZE aligned. +* +* A mapping must be unmapped by using dma_buf_unmap_attachment(). Note that +* the underlying backing storage is pinned for as long as a mapping exists, +* therefore users/importers should not hold onto a mapping for undue amounts of +* time. +* +* Important: Dynamic importers must wait for the exclusive fence of the struct +* dma_resv attached to the DMA-BUF first. */ -void dma_buf_unmap_attachment (struct dma_buf_attachment *attach, -struct sg_table *sg_table, +struct sg_table * +dma_buf_map_attachment (struct dma_buf_attachment *attach,enum dma_data_direction direction) { +struct sg_table *sg_table; +might_sleep (); -if (WARN_ON (!attach || !attach->dmabuf || !sg_table)) -return ; +if (WARN_ON (!attach || !attach->dmabuf)) +return ERR_PTR (-EINVAL ); + +dma_resv_lock (attach->dmabuf->resv,NULL ); + sg_table =dma_buf_map_attachment_locked (attach, direction); +dma_resv_unlock (attach->dmabuf->resv); -if (dma_buf_attachment_is_dynamic (attach)) -dma_resv_assert_held (attach->dmabuf->resv); +return sg_table; +} +EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment,DMA_BUF ); + +/** +* dma_buf_unmap_attachment_locked - Returns the scatterlist table of the attachment; +* mapped into _device_ address space. Is a wrapper for map_dma_buf() of the +* dma_buf_ops. +* @attach: [in] attachment whose scatterlist is to be returned +* @direction: [in] direction of DMA transfer +* +* Locked variant of dma_buf_unmap_attachment(). +* +* Caller is responsible for holding dmabuf's reservation lock. +*/ +void dma_buf_unmap_attachment_locked (struct dma_buf_attachment *attach, +struct sg_table *sg_table, +enum dma_data_direction direction) +{ +might_sleep (); + +dma_resv_assert_held (attach->dmabuf->resv);if (attach->sgt == sg_table)return ; -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_assert_held (attach->dmabuf->resv); -__unmap_dma_buf (attach, sg_table, direction);if (dma_buf_is_dynamic (attach->dmabuf) && !IS_ENABLED (CONFIG_DMABUF_MOVE_NOTIFY ))dma_buf_unpin (attach); } +EXPORT_SYMBOL_NS_GPL (dma_buf_unmap_attachment_locked,DMA_BUF ); + +/** +* dma_buf_unmap_attachment - unmaps and decreases usecount of the buffer;might +* deallocate the scatterlist associated. Is a wrapper for unmap_dma_buf() of +* dma_buf_ops. +* @attach: [in] attachment to unmap buffer from +* @sg_table: [in] scatterlist info of the buffer to unmap +* @direction: [in] direction of DMA transfer +* +* This unmaps a DMA mapping for @attached obtained by dma_buf_map_attachment(). +*/ +void dma_buf_unmap_attachment (struct dma_buf_attachment *attach, +struct sg_table *sg_table, +enum dma_data_direction direction) +{ +might_sleep (); + +if (WARN_ON (!attach || !attach->dmabuf || !sg_table)) +return ; + +dma_resv_lock (attach->dmabuf->resv,NULL ); +dma_buf_unmap_attachment_locked (attach, sg_table, direction); +dma_resv_unlock (attach->dmabuf->resv); +}EXPORT_SYMBOL_NS_GPL (dma_buf_unmap_attachment,DMA_BUF );/**
@@ -1224,6 +1269,31 @@ int dma_buf_end_cpu_access(struct dma_buf *dmabuf, }EXPORT_SYMBOL_NS_GPL (dma_buf_end_cpu_access,DMA_BUF ); +static int dma_buf_mmap_locked (struct dma_buf *dmabuf, +struct vm_area_struct *vma, +unsigned long pgoff) +{ +dma_resv_assert_held (dmabuf->resv); + +/* check if buffer supports mmap */ +if (!dmabuf->ops->mmap) +return -EINVAL ; + +/* check for offset overflow */ +if (pgoff +vma_pages (vma) < pgoff) +return -EOVERFLOW ; + +/* check for overflowing the buffer's size */ +if (pgoff +vma_pages (vma) > + dmabuf->size >>PAGE_SHIFT ) +return -EINVAL ; + +/* readjust the vma */ +vma_set_file (vma, dmabuf->file); + vma->vm_pgoff = pgoff; + +return dmabuf->ops->mmap (dmabuf, vma); +}/** * dma_buf_mmap - Setup up a userspace mmap with the given vma
@@ -1242,29 +1312,46 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);int dma_buf_mmap (struct dma_buf *dmabuf,struct vm_area_struct *vma,unsigned long pgoff) { +int ret; +if (WARN_ON (!dmabuf || !vma))return -EINVAL ; -/* check if buffer supports mmap */ -if (!dmabuf->ops->mmap) -return -EINVAL ; +dma_resv_lock (dmabuf->resv,NULL ); + ret =dma_buf_mmap_locked (dmabuf, vma, pgoff); +dma_resv_unlock (dmabuf->resv); -/* check for offset overflow */ -if (pgoff +vma_pages (vma) < pgoff) -return -EOVERFLOW ; +return ret; +} +EXPORT_SYMBOL_NS_GPL (dma_buf_mmap,DMA_BUF ); -/* check for overflowing the buffer's size */ -if (pgoff +vma_pages (vma) > - dmabuf->size >>PAGE_SHIFT ) -return -EINVAL ; +static int dma_buf_vmap_locked (struct dma_buf *dmabuf,struct iosys_map *map) +{ +struct iosys_map ptr; +int ret; -/* readjust the vma */ -vma_set_file (vma, dmabuf->file); - vma->vm_pgoff = pgoff; +dma_resv_assert_held (dmabuf->resv); -return dmabuf->ops->mmap (dmabuf, vma); +if (dmabuf->vmapping_counter) { + dmabuf->vmapping_counter++; +BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr)); + *map = dmabuf->vmap_ptr; +return ret; + } + +BUG_ON (iosys_map_is_set (&dmabuf->vmap_ptr)); + + ret = dmabuf->ops->vmap (dmabuf, &ptr); +if (WARN_ON_ONCE (ret)) +return ret; + + dmabuf->vmap_ptr = ptr; + dmabuf->vmapping_counter =1 ; + + *map = dmabuf->vmap_ptr; + +return 0 ; } -EXPORT_SYMBOL_NS_GPL (dma_buf_mmap,DMA_BUF );/** * dma_buf_vmap - Create virtual mapping for the buffer object into kernel
@@ -1284,8 +1371,7 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);*/ int dma_buf_vmap (struct dma_buf *dmabuf,struct iosys_map *map) { -struct iosys_map ptr; -int ret =0 ; +int ret;iosys_map_clear (map);
@@ -1295,52 +1381,40 @@ int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map)if (!dmabuf->ops->vmap)return -EINVAL ; -mutex_lock (&dmabuf->lock); -if (dmabuf->vmapping_counter) { - dmabuf->vmapping_counter++; -BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr)); - *map = dmabuf->vmap_ptr; - goto out_unlock; - } - -BUG_ON (iosys_map_is_set (&dmabuf->vmap_ptr)); - - ret = dmabuf->ops->vmap (dmabuf, &ptr); -if (WARN_ON_ONCE (ret)) - goto out_unlock; - - dmabuf->vmap_ptr = ptr; - dmabuf->vmapping_counter =1 ; - - *map = dmabuf->vmap_ptr; +dma_resv_lock (dmabuf->resv,NULL ); + ret =dma_buf_vmap_locked (dmabuf, map); +dma_resv_unlock (dmabuf->resv); -out_unlock: -mutex_unlock (&dmabuf->lock);return ret; }EXPORT_SYMBOL_NS_GPL (dma_buf_vmap,DMA_BUF ); -/** -* dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap. -* @dmabuf: [in] buffer to vunmap -* @map: [in] vmap pointer to vunmap -*/ -void dma_buf_vunmap (struct dma_buf *dmabuf,struct iosys_map *map) +static void dma_buf_vunmap_locked (struct dma_buf *dmabuf,struct iosys_map *map) { -if (WARN_ON (!dmabuf)) -return ; -BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr));BUG_ON (dmabuf->vmapping_counter ==0 );BUG_ON (!iosys_map_is_equal (&dmabuf->vmap_ptr, map)); -mutex_lock (&dmabuf->lock);if (--dmabuf->vmapping_counter ==0 ) {if (dmabuf->ops->vunmap) dmabuf->ops->vunmap (dmabuf, map);iosys_map_clear (&dmabuf->vmap_ptr); } -mutex_unlock (&dmabuf->lock); +} + +/** +* dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap. +* @dmabuf: [in] buffer to vunmap +* @map: [in] vmap pointer to vunmap +*/ +void dma_buf_vunmap (struct dma_buf *dmabuf,struct iosys_map *map) +{ +if (WARN_ON (!dmabuf)) +return ; + +dma_resv_lock (dmabuf->resv,NULL ); +dma_buf_vunmap_locked (dmabuf, map); +dma_resv_unlock (dmabuf->resv); }EXPORT_SYMBOL_NS_GPL (dma_buf_vunmap,DMA_BUF );
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4 -2
@@ -882,7 +882,8 @@ static int amdgpu_ttm_backend_bind(struct ttm_device *bdev,struct sg_table *sgt; attach = gtt->gobj->import_attach; - sgt =dma_buf_map_attachment (attach,DMA_BIDIRECTIONAL ); + sgt =dma_buf_map_attachment_locked (attach, +DMA_BIDIRECTIONAL );if (IS_ERR (sgt))return PTR_ERR (sgt);
@@ -1007,7 +1008,8 @@ static void amdgpu_ttm_backend_unbind(struct ttm_device *bdev,struct dma_buf_attachment *attach; attach = gtt->gobj->import_attach; -dma_buf_unmap_attachment (attach, ttm->sg,DMA_BIDIRECTIONAL ); +dma_buf_unmap_attachment_locked (attach, ttm->sg, +DMA_BIDIRECTIONAL ); ttm->sg =NULL ; }
drivers/gpu/drm/drm_client.c +2 -2
@@ -323,7 +323,7 @@ drm_client_buffer_vmap(struct drm_client_buffer *buffer,* fd_install step out of the driver backend hooks, to make that * final step optional for internal users. */ - ret =drm_gem_vmap (buffer->gem, map); + ret =drm_gem_vmap_unlocked (buffer->gem, map);if (ret)return ret;
@@ -345,7 +345,7 @@ void drm_client_buffer_vunmap(struct drm_client_buffer *buffer) {struct iosys_map *map = &buffer->map; -drm_gem_vunmap (buffer->gem, map); +drm_gem_vunmap_unlocked (buffer->gem, map); }EXPORT_SYMBOL (drm_client_buffer_vunmap);
drivers/gpu/drm/drm_gem.c +33 -0
@@ -1053,7 +1053,12 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, vma->vm_ops = obj->funcs->vm_ops;if (obj->funcs->mmap) { + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret) + goto err_drm_gem_object_put; + ret = obj->funcs->mmap (obj, vma); +dma_resv_unlock (obj->resv);if (ret) goto err_drm_gem_object_put;WARN_ON (!(vma->vm_flags &VM_DONTEXPAND ));
@@ -1158,6 +1163,8 @@ void drm_gem_print_info(struct drm_printer *p, unsigned int indent,int drm_gem_pin (struct drm_gem_object *obj) { +dma_resv_assert_held (obj->resv); +if (obj->funcs->pin)return obj->funcs->pin (obj);else
@@ -1166,6 +1173,8 @@ int drm_gem_pin(struct drm_gem_object *obj)void drm_gem_unpin (struct drm_gem_object *obj) { +dma_resv_assert_held (obj->resv); +if (obj->funcs->unpin) obj->funcs->unpin (obj); }
@@ -1174,6 +1183,8 @@ int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map) {int ret; +dma_resv_assert_held (obj->resv); +if (!obj->funcs->vmap)return -EOPNOTSUPP ;
@@ -1189,6 +1200,8 @@ EXPORT_SYMBOL(drm_gem_vmap);void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map) { +dma_resv_assert_held (obj->resv); +if (iosys_map_is_null (map))return ;
@@ -1200,6 +1213,26 @@ void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map) }EXPORT_SYMBOL (drm_gem_vunmap); +int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +{ +int ret; + +dma_resv_lock (obj->resv,NULL ); + ret =drm_gem_vmap (obj, map); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL (drm_gem_vmap_unlocked); + +void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +{ +dma_resv_lock (obj->resv,NULL ); +drm_gem_vunmap (obj, map); +dma_resv_unlock (obj->resv); +} +EXPORT_SYMBOL (drm_gem_vunmap_unlocked); +/** * drm_gem_lock_reservations - Sets up the ww context and acquires * the lock on an array of GEM objects.
drivers/gpu/drm/drm_gem_framebuffer_helper.c +3 -3
@@ -348,7 +348,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb,iosys_map_clear (&map[i]);continue ; } - ret =drm_gem_vmap (obj, &map[i]); + ret =drm_gem_vmap_unlocked (obj, &map[i]);if (ret) goto err_drm_gem_vunmap; }
@@ -370,7 +370,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, obj =drm_gem_fb_get_obj (fb, i);if (!obj)continue ; -drm_gem_vunmap (obj, &map[i]); +drm_gem_vunmap_unlocked (obj, &map[i]); }return ret; }
@@ -398,7 +398,7 @@ void drm_gem_fb_vunmap(struct drm_framebuffer *fb,continue ;if (iosys_map_is_null (&map[i]))continue ; -drm_gem_vunmap (obj, &map[i]); +drm_gem_vunmap_unlocked (obj, &map[i]); } }EXPORT_SYMBOL (drm_gem_fb_vunmap);
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +5 -5
@@ -72,7 +72,7 @@ static int i915_gem_dmabuf_vmap(struct dma_buf *dma_buf,struct drm_i915_gem_object *obj =dma_buf_to_obj (dma_buf);void *vaddr; - vaddr =i915_gem_object_pin_map_unlocked (obj,I915_MAP_WB ); + vaddr =i915_gem_object_pin_map (obj,I915_MAP_WB );if (IS_ERR (vaddr))return PTR_ERR (vaddr);
@@ -241,8 +241,8 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)assert_object_held (obj); - pages =dma_buf_map_attachment (obj->base.import_attach, -DMA_BIDIRECTIONAL ); + pages =dma_buf_map_attachment_locked (obj->base.import_attach, +DMA_BIDIRECTIONAL );if (IS_ERR (pages))return PTR_ERR (pages);
@@ -270,8 +270,8 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)static void i915_gem_object_put_pages_dmabuf (struct drm_i915_gem_object *obj,struct sg_table *pages) { -dma_buf_unmap_attachment (obj->base.import_attach, pages, -DMA_BIDIRECTIONAL ); +dma_buf_unmap_attachment_locked (obj->base.import_attach, pages, +DMA_BIDIRECTIONAL ); }static const struct drm_i915_gem_object_ops i915_gem_object_dmabuf_ops = {
drivers/gpu/drm/qxl/qxl_object.c +9 -8
@@ -168,9 +168,16 @@ int qxl_bo_vmap_locked(struct qxl_bo *bo, struct iosys_map *map) bo->map_count++; goto out; } - r =ttm_bo_vmap (&bo->tbo, &bo->map); + + r =__qxl_bo_pin (bo);if (r)return r; + + r =ttm_bo_vmap (&bo->tbo, &bo->map); +if (r) { +__qxl_bo_unpin (bo); +return r; + } bo->map_count =1 ;/* TODO: Remove kptr in favor of map everywhere. */
@@ -192,12 +199,6 @@ int qxl_bo_vmap(struct qxl_bo *bo, struct iosys_map *map)if (r)return r; - r =__qxl_bo_pin (bo); -if (r) { -qxl_bo_unreserve (bo); -return r; - } - r =qxl_bo_vmap_locked (bo, map);qxl_bo_unreserve (bo);return r;
@@ -247,6 +248,7 @@ void qxl_bo_vunmap_locked(struct qxl_bo *bo)return ; bo->kptr =NULL ;ttm_bo_vunmap (&bo->tbo, &bo->map); +__qxl_bo_unpin (bo); }int qxl_bo_vunmap (struct qxl_bo *bo)
@@ -258,7 +260,6 @@ int qxl_bo_vunmap(struct qxl_bo *bo)return r;qxl_bo_vunmap_locked (bo); -__qxl_bo_unpin (bo);qxl_bo_unreserve (bo);return 0 ; }
drivers/gpu/drm/qxl/qxl_prime.c +2 -2
@@ -59,7 +59,7 @@ int qxl_gem_prime_vmap(struct drm_gem_object *obj, struct iosys_map *map)struct qxl_bo *bo =gem_to_qxl_bo (obj);int ret; - ret =qxl_bo_vmap (bo, map); + ret =qxl_bo_vmap_locked (bo, map);if (ret <0 )return ret;
@@ -71,5 +71,5 @@ void qxl_gem_prime_vunmap(struct drm_gem_object *obj, {struct qxl_bo *bo =gem_to_qxl_bo (obj); -qxl_bo_vunmap (bo); +qxl_bo_vunmap_locked (bo); }
drivers/media/common/videobuf2/videobuf2-dma-contig.c +1 -10
@@ -382,18 +382,12 @@ static struct sg_table *vb2_dc_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_dc_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {
@@ -409,14 +403,11 @@ static struct sg_table *vb2_dc_dmabuf_ops_map(if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,DMA_ATTR_SKIP_CPU_SYNC )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }
drivers/media/common/videobuf2/videobuf2-dma-sg.c +1 -10
@@ -424,18 +424,12 @@ static struct sg_table *vb2_dma_sg_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_dma_sg_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {
@@ -446,14 +440,11 @@ static struct sg_table *vb2_dma_sg_dmabuf_ops_map(/* mapping to the client with new direction */ if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,0 )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }
drivers/media/common/videobuf2/videobuf2-vmalloc.c +1 -10
@@ -267,18 +267,12 @@ static struct sg_table *vb2_vmalloc_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_vmalloc_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {
@@ -289,14 +283,11 @@ static struct sg_table *vb2_vmalloc_dmabuf_ops_map(/* mapping to the client with new direction */ if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,0 )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }
include/drm/drm_gem.h +3 -0
@@ -410,4 +410,7 @@ void drm_gem_unlock_reservations(struct drm_gem_object **objs, int count,int drm_gem_dumb_map_offset (struct drm_file *file,struct drm_device *dev,u32 handle,u64 *offset); +int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +#endif /* __DRM_GEM_H__ */
include/linux/dma-buf.h +5 -9
@@ -326,15 +326,6 @@ struct dma_buf {/** @ops: dma_buf_ops associated with this buffer object. */ const struct dma_buf_ops *ops; -/** -* @lock: -* -* Used internally to serialize list manipulation, attach/detach and -* vmap/unmap. Note that in many cases this is superseeded by -* dma_resv_lock() on @resv. -*/ -struct mutex lock; -/** * @vmapping_counter: *
@@ -618,6 +609,11 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags);struct dma_buf *dma_buf_get (int fd);void dma_buf_put (struct dma_buf *dmabuf); +struct sg_table *dma_buf_map_attachment_locked (struct dma_buf_attachment *, +enum dma_data_direction ); +void dma_buf_unmap_attachment_locked (struct dma_buf_attachment *, +struct sg_table *, +enum dma_data_direction );struct sg_table *dma_buf_map_attachment (struct dma_buf_attachment *,enum dma_data_direction ); void dma_buf_unmap_attachment(struct dma_buf_attachment *, struct sg_table *,
Signature · 1 line
2.35.3
kernel test robot
build botRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
I love your patch! Perhaps something to improve:
Hi Dmitry,
I love your patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[cannot apply to drm/drm-next media-tree/master drm-intel/for-linux-next v5.18] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Dmitry-Osipenko/Add-generic-memory-shrinker-to-VirtIO-GPU-and-Panfrost-DRM-drivers/20220527-075717 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cdeffe87f790dfd1baa193020411ce9a538446d7 config: hexagon-randconfig-r045-20220524 (https://download.01.org/0day-ci/archive/20220527/[email protected]/config) compiler: clang version 15.0.0 (https://github.com/llvm/llvm-project 6f4644d194da594562027a5d458d9fb7a20ebc39)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/intel-lab-lkp/linux/commit/97f090c47ec995a8cf3bced98526ee3eaa25f10f
git remote add linux-review https://github.com/intel-lab-lkp/linux
git fetch --no-tags linux-review Dmitry-Osipenko/Add-generic-memory-shrinker-to-VirtIO-GPU-and-Panfrost-DRM-drivers/20220527-075717 git checkout 97f090c47ec995a8cf3bced98526ee3eaa25f10f
# save the config file
mkdir build_dir && cp config build_dir/.config
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 O=build_dir ARCH=hexagon SHELL=/bin/bash drivers/dma-buf/
If you fix the issue, kindly add following tag where applicable
All warnings (new ones prefixed by >>):
>> drivers/dma-buf/dma-buf.c:1339:10: warning: variable 'ret' is uninitialized when used here [-Wuninitialized]
return ret;
^~~
drivers/dma-buf/dma-buf.c:1331:9: note: initialize the variable 'ret' to silence this warning
int ret;
^
= 01 warning generated.
vim +/ret +1339 drivers/dma-buf/dma-buf.c
1327
1328 static int dma_buf_vmap_locked(struct dma_buf *dmabuf, struct iosys_map *map)
1329 {
1330 struct iosys_map ptr;
1331 int ret;
1332
1333 dma_resv_assert_held(dmabuf->resv);
1334
1335 if (dmabuf->vmapping_counter) {
1336 dmabuf->vmapping_counter++;
1337 BUG_ON(iosys_map_is_null(&dmabuf->vmap_ptr));
1338 *map = dmabuf->vmap_ptr;
> 1339 return ret;
1340 }
1341
1342 BUG_ON(iosys_map_is_set(&dmabuf->vmap_ptr));
1343
1344 ret = dmabuf->ops->vmap(dmabuf, &ptr);
1345 if (WARN_ON_ONCE(ret))
1346 return ret;
1347
1348 dmabuf->vmap_ptr = ptr;
1349 dmabuf->vmapping_counter = 1;
1350
1351 *map = dmabuf->vmap_ptr;
1352
1353 return 0;
1354 }
1355
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Quoted 8 lines · kernel test robot
Quoted 1 line
Quoted 1 line
drivers/dma-buf/dma-buf.c:1339:10: warning: variable 'ret' is uninitialized when used here [-Wuninitialized]return ret; ^~~ drivers/dma-buf/dma-buf.c:1331:9: note: initialize the variable 'ret' to silence this warning int ret; ^ = 01 warning generated.
Interestingly, GCC doesn't detect this problem and it's a valid warning. I'll correct it in v7.
Signature · 2 lines
Best regards, Dmitry
Christian König
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
In reply to Dmitry OsipenkoHi Dmitry,
First of all please separate out this patch from the rest of the series, since this is a complex separate structural change.
Am 27.05.22 um 01:50 schrieb Dmitry Osipenko:
Quoted 4 lines
All dma-bufs have dma-reservation lock that allows drivers to perform exclusive operations over shared dma-bufs. Today's dma-buf API has incomplete locking specification, which creates dead lock situation for dma-buf importers and exporters that don't coordinate theirs locks.
Well please drop that sentence. The locking specifications are actually very well defined, it's just that some drivers are a bit broken regarding them.
What you do here is rather moving all the non-dynamic drivers over to the dynamic locking specification (which is really nice to have).
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Quoted 10 lines
This patch introduces new locking convention for dma-buf users. From now on all dma-buf importers are responsible for holding dma-buf reservation lock around operations performed over dma-bufs.
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers that need to manage imported dma-bufs under the held lock.
Instead of adding new locked variants please mark all variants which expect to be called without a lock with an _unlocked postfix.
This should make it easier to remove those in a follow up patch set and then fully move the locking into the importer.
Quoted 1 line
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Quoted 23 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/dma-buf/dma-buf.c | 270 +++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +- drivers/gpu/drm/drm_gem.c | 33 +++ drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 10 +- drivers/gpu/drm/qxl/qxl_object.c | 17 +- drivers/gpu/drm/qxl/qxl_prime.c | 4 +- .../common/videobuf2/videobuf2-dma-contig.c | 11 +- .../media/common/videobuf2/videobuf2-dma-sg.c | 11 +- .../common/videobuf2/videobuf2-vmalloc.c | 11 +- include/drm/drm_gem.h | 3 + include/linux/dma-buf.h | 14 +- 13 files changed, 241 insertions(+), 159 deletions(-)drivers/dma-buf/dma-buf.c +0
-0
drivers/dma-buf/dma-buf.c struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) · line 555
Quoted 4 lines
drivers/dma-buf/dma-buf.c +0
-1@@ -552,7 +552,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) file->f_mode |=FMODE_LSEEK ; dmabuf->file = file; -mutex_init (&dmabuf->lock);
Please make removing dmabuf->lock a separate change.
Regards,
Christian.
Quoted 869 lines
drivers/dma-buf/dma-buf.c +0
-0INIT_LIST_HEAD (&dmabuf->attachments);mutex_lock (&db_list.lock);@@ -737,14 +736,14 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, attach->importer_ops = importer_ops; attach->importer_priv = importer_priv; 3. Converting all drivers to the new locking scheme.Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/dma-buf/dma-buf.c | 270 +++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +-drivers/dma-buf/dma-buf.c +172
-97+dma_resv_lock (dmabuf->resv,NULL ); +if (dmabuf->ops->attach) { ret = dmabuf->ops->attach (dmabuf, attach);if (ret) goto err_attach; } -dma_resv_lock (dmabuf->resv,NULL );list_add (&attach->node, &dmabuf->attachments); -dma_resv_unlock (dmabuf->resv);/* When either the importer or the exporter can't handle dynamic * mappings we cache the mapping here to avoid issues with the @@ -755,7 +754,6 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev,struct sg_table *sgt;if (dma_buf_is_dynamic (attach->dmabuf)) { -dma_resv_lock (attach->dmabuf->resv,NULL ); ret = dmabuf->ops->pin (attach);if (ret) goto err_unlock;@@ -768,15 +766,16 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, ret =PTR_ERR (sgt); goto err_unpin; } -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_unlock (attach->dmabuf->resv); attach->sgt = sgt; attach->dir =DMA_BIDIRECTIONAL ; } +dma_resv_unlock (dmabuf->resv); +return attach; err_attach: +dma_resv_unlock (attach->dmabuf->resv);kfree (attach);return ERR_PTR (ret);@@ -785,10 +784,10 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, dmabuf->ops->unpin (attach); err_unlock: -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_unlock (attach->dmabuf->resv); +dma_resv_unlock (dmabuf->resv);dma_buf_detach (dmabuf, attach); +return ERR_PTR (ret); }EXPORT_SYMBOL_NS_GPL (dma_buf_dynamic_attach,DMA_BUF );@@ -832,24 +831,23 @@ void dma_buf_detach(struct dma_buf *dmabuf, struct dma_buf_attachment *attach)if (WARN_ON (!dmabuf || !attach))return ; -if (attach->sgt) { -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_lock (attach->dmabuf->resv,NULL ); +if (WARN_ON (dmabuf != attach->dmabuf)) +return ; +dma_resv_lock (dmabuf->resv,NULL ); + +if (attach->sgt) {__unmap_dma_buf (attach, attach->sgt, attach->dir); -if (dma_buf_is_dynamic (attach->dmabuf)) { +if (dma_buf_is_dynamic (attach->dmabuf)) dmabuf->ops->unpin (attach); -dma_resv_unlock (attach->dmabuf->resv); - } } -dma_resv_lock (dmabuf->resv,NULL );list_del (&attach->node); -dma_resv_unlock (dmabuf->resv);if (dmabuf->ops->detach) dmabuf->ops->detach (dmabuf, attach); +dma_resv_unlock (dmabuf->resv);kfree (attach); }EXPORT_SYMBOL_NS_GPL (dma_buf_detach,DMA_BUF );@@ -906,28 +904,18 @@ void dma_buf_unpin(struct dma_buf_attachment *attach)EXPORT_SYMBOL_NS_GPL (dma_buf_unpin,DMA_BUF );/** -* dma_buf_map_attachment - Returns the scatterlist table of the attachment; +* dma_buf_map_attachment_locked - Returns the scatterlist table of the attachment; * mapped into _device_ address space. Is a wrapper for map_dma_buf() of the * dma_buf_ops. * @attach: [in] attachment whose scatterlist is to be returned * @direction: [in] direction of DMA transfer * -* Returns sg_table containing the scatterlist to be returned; returns ERR_PTR -* on error. May return -EINTR if it is interrupted by a signal. -* -* On success, the DMA addresses and lengths in the returned scatterlist are -* PAGE_SIZE aligned. -* -* A mapping must be unmapped by using dma_buf_unmap_attachment(). Note that -* the underlying backing storage is pinned for as long as a mapping exists, -* therefore users/importers should not hold onto a mapping for undue amounts of -* time. +* Locked variant of dma_buf_map_attachment(). * -* Important: Dynamic importers must wait for the exclusive fence of the struct -* dma_resv attached to the DMA-BUF first. +* Caller is responsible for holding dmabuf's reservation lock. */ -struct sg_table *dma_buf_map_attachment (struct dma_buf_attachment *attach, -enum dma_data_direction direction) +struct sg_table *dma_buf_map_attachment_locked (struct dma_buf_attachment *attach, +enum dma_data_direction direction) {struct sg_table *sg_table;int r;@@ -937,8 +925,7 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach,if (WARN_ON (!attach || !attach->dmabuf))return ERR_PTR (-EINVAL ); -if (dma_buf_attachment_is_dynamic (attach)) -dma_resv_assert_held (attach->dmabuf->resv); +dma_resv_assert_held (attach->dmabuf->resv);if (attach->sgt) {/* @@ -953,7 +940,6 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach, }if (dma_buf_is_dynamic (attach->dmabuf)) { -dma_resv_assert_held (attach->dmabuf->resv);if (!IS_ENABLED (CONFIG_DMABUF_MOVE_NOTIFY )) { r = attach->dmabuf->ops->pin (attach);if (r)@@ -993,42 +979,101 @@ struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach,#endif /* CONFIG_DMA_API_DEBUG */ return sg_table; } -EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment,DMA_BUF ); +EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment_locked,DMA_BUF );/** -* dma_buf_unmap_attachment - unmaps and decreases usecount of the buffer;might -* deallocate the scatterlist associated. Is a wrapper for unmap_dma_buf() of +* dma_buf_map_attachment - Returns the scatterlist table of the attachment; +* mapped into _device_ address space. Is a wrapper for map_dma_buf() of the * dma_buf_ops. -* @attach: [in] attachment to unmap buffer from -* @sg_table: [in] scatterlist info of the buffer to unmap -* @direction: [in] direction of DMA transfer +* @attach: [in] attachment whose scatterlist is to be returned +* @direction: [in] direction of DMA transfer * -* This unmaps a DMA mapping for @attached obtained by dma_buf_map_attachment(). +* Returns sg_table containing the scatterlist to be returned; returns ERR_PTR +* on error. May return -EINTR if it is interrupted by a signal. +* +* On success, the DMA addresses and lengths in the returned scatterlist are +* PAGE_SIZE aligned. +* +* A mapping must be unmapped by using dma_buf_unmap_attachment(). Note that +* the underlying backing storage is pinned for as long as a mapping exists, +* therefore users/importers should not hold onto a mapping for undue amounts of +* time. +* +* Important: Dynamic importers must wait for the exclusive fence of the struct +* dma_resv attached to the DMA-BUF first. */ -void dma_buf_unmap_attachment (struct dma_buf_attachment *attach, -struct sg_table *sg_table, +struct sg_table * +dma_buf_map_attachment (struct dma_buf_attachment *attach,enum dma_data_direction direction) { +struct sg_table *sg_table; +might_sleep (); -if (WARN_ON (!attach || !attach->dmabuf || !sg_table)) -return ; +if (WARN_ON (!attach || !attach->dmabuf)) +return ERR_PTR (-EINVAL ); + +dma_resv_lock (attach->dmabuf->resv,NULL ); + sg_table =dma_buf_map_attachment_locked (attach, direction); +dma_resv_unlock (attach->dmabuf->resv); -if (dma_buf_attachment_is_dynamic (attach)) -dma_resv_assert_held (attach->dmabuf->resv); +return sg_table; +} +EXPORT_SYMBOL_NS_GPL (dma_buf_map_attachment,DMA_BUF ); + +/** +* dma_buf_unmap_attachment_locked - Returns the scatterlist table of the attachment; +* mapped into _device_ address space. Is a wrapper for map_dma_buf() of the +* dma_buf_ops. +* @attach: [in] attachment whose scatterlist is to be returned +* @direction: [in] direction of DMA transfer +* +* Locked variant of dma_buf_unmap_attachment(). +* +* Caller is responsible for holding dmabuf's reservation lock. +*/ +void dma_buf_unmap_attachment_locked (struct dma_buf_attachment *attach, +struct sg_table *sg_table, +enum dma_data_direction direction) +{ +might_sleep (); + +dma_resv_assert_held (attach->dmabuf->resv);if (attach->sgt == sg_table)return ; -if (dma_buf_is_dynamic (attach->dmabuf)) -dma_resv_assert_held (attach->dmabuf->resv); -__unmap_dma_buf (attach, sg_table, direction);if (dma_buf_is_dynamic (attach->dmabuf) && !IS_ENABLED (CONFIG_DMABUF_MOVE_NOTIFY ))dma_buf_unpin (attach); } +EXPORT_SYMBOL_NS_GPL (dma_buf_unmap_attachment_locked,DMA_BUF ); + +/** +* dma_buf_unmap_attachment - unmaps and decreases usecount of the buffer;might +* deallocate the scatterlist associated. Is a wrapper for unmap_dma_buf() of +* dma_buf_ops. +* @attach: [in] attachment to unmap buffer from +* @sg_table: [in] scatterlist info of the buffer to unmap +* @direction: [in] direction of DMA transfer +* +* This unmaps a DMA mapping for @attached obtained by dma_buf_map_attachment(). +*/ +void dma_buf_unmap_attachment (struct dma_buf_attachment *attach, +struct sg_table *sg_table, +enum dma_data_direction direction) +{ +might_sleep (); + +if (WARN_ON (!attach || !attach->dmabuf || !sg_table)) +return ; + +dma_resv_lock (attach->dmabuf->resv,NULL ); +dma_buf_unmap_attachment_locked (attach, sg_table, direction); +dma_resv_unlock (attach->dmabuf->resv); +}EXPORT_SYMBOL_NS_GPL (dma_buf_unmap_attachment,DMA_BUF );/** @@ -1224,6 +1269,31 @@ int dma_buf_end_cpu_access(struct dma_buf *dmabuf, }EXPORT_SYMBOL_NS_GPL (dma_buf_end_cpu_access,DMA_BUF ); +static int dma_buf_mmap_locked (struct dma_buf *dmabuf, +struct vm_area_struct *vma, +unsigned long pgoff) +{ +dma_resv_assert_held (dmabuf->resv); + +/* check if buffer supports mmap */ +if (!dmabuf->ops->mmap) +return -EINVAL ; + +/* check for offset overflow */ +if (pgoff +vma_pages (vma) < pgoff) +return -EOVERFLOW ; + +/* check for overflowing the buffer's size */ +if (pgoff +vma_pages (vma) > + dmabuf->size >>PAGE_SHIFT ) +return -EINVAL ; + +/* readjust the vma */ +vma_set_file (vma, dmabuf->file); + vma->vm_pgoff = pgoff; + +return dmabuf->ops->mmap (dmabuf, vma); +}/** * dma_buf_mmap - Setup up a userspace mmap with the given vma @@ -1242,29 +1312,46 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_end_cpu_access, DMA_BUF);int dma_buf_mmap (struct dma_buf *dmabuf,struct vm_area_struct *vma,unsigned long pgoff) { +int ret; +if (WARN_ON (!dmabuf || !vma))return -EINVAL ; -/* check if buffer supports mmap */ -if (!dmabuf->ops->mmap) -return -EINVAL ; +dma_resv_lock (dmabuf->resv,NULL ); + ret =dma_buf_mmap_locked (dmabuf, vma, pgoff); +dma_resv_unlock (dmabuf->resv); -/* check for offset overflow */ -if (pgoff +vma_pages (vma) < pgoff) -return -EOVERFLOW ; +return ret; +} +EXPORT_SYMBOL_NS_GPL (dma_buf_mmap,DMA_BUF ); -/* check for overflowing the buffer's size */ -if (pgoff +vma_pages (vma) > - dmabuf->size >>PAGE_SHIFT ) -return -EINVAL ; +static int dma_buf_vmap_locked (struct dma_buf *dmabuf,struct iosys_map *map) +{ +struct iosys_map ptr; +int ret; -/* readjust the vma */ -vma_set_file (vma, dmabuf->file); - vma->vm_pgoff = pgoff; +dma_resv_assert_held (dmabuf->resv); -return dmabuf->ops->mmap (dmabuf, vma); +if (dmabuf->vmapping_counter) { + dmabuf->vmapping_counter++; +BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr)); + *map = dmabuf->vmap_ptr; +return ret; + } + +BUG_ON (iosys_map_is_set (&dmabuf->vmap_ptr)); + + ret = dmabuf->ops->vmap (dmabuf, &ptr); +if (WARN_ON_ONCE (ret)) +return ret; + + dmabuf->vmap_ptr = ptr; + dmabuf->vmapping_counter =1 ; + + *map = dmabuf->vmap_ptr; + +return 0 ; } -EXPORT_SYMBOL_NS_GPL (dma_buf_mmap,DMA_BUF );/** * dma_buf_vmap - Create virtual mapping for the buffer object into kernel @@ -1284,8 +1371,7 @@ EXPORT_SYMBOL_NS_GPL(dma_buf_mmap, DMA_BUF);*/ int dma_buf_vmap (struct dma_buf *dmabuf,struct iosys_map *map) { -struct iosys_map ptr; -int ret =0 ; +int ret;iosys_map_clear (map);@@ -1295,52 +1381,40 @@ int dma_buf_vmap(struct dma_buf *dmabuf, struct iosys_map *map)if (!dmabuf->ops->vmap)return -EINVAL ; -mutex_lock (&dmabuf->lock); -if (dmabuf->vmapping_counter) { - dmabuf->vmapping_counter++; -BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr)); - *map = dmabuf->vmap_ptr; - goto out_unlock; - } - -BUG_ON (iosys_map_is_set (&dmabuf->vmap_ptr)); - - ret = dmabuf->ops->vmap (dmabuf, &ptr); -if (WARN_ON_ONCE (ret)) - goto out_unlock; - - dmabuf->vmap_ptr = ptr; - dmabuf->vmapping_counter =1 ; - - *map = dmabuf->vmap_ptr; +dma_resv_lock (dmabuf->resv,NULL ); + ret =dma_buf_vmap_locked (dmabuf, map); +dma_resv_unlock (dmabuf->resv); -out_unlock: -mutex_unlock (&dmabuf->lock);return ret; }EXPORT_SYMBOL_NS_GPL (dma_buf_vmap,DMA_BUF ); -/** -* dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap. -* @dmabuf: [in] buffer to vunmap -* @map: [in] vmap pointer to vunmap -*/ -void dma_buf_vunmap (struct dma_buf *dmabuf,struct iosys_map *map) +static void dma_buf_vunmap_locked (struct dma_buf *dmabuf,struct iosys_map *map) { -if (WARN_ON (!dmabuf)) -return ; -BUG_ON (iosys_map_is_null (&dmabuf->vmap_ptr));BUG_ON (dmabuf->vmapping_counter ==0 );BUG_ON (!iosys_map_is_equal (&dmabuf->vmap_ptr, map)); -mutex_lock (&dmabuf->lock);if (--dmabuf->vmapping_counter ==0 ) {if (dmabuf->ops->vunmap) dmabuf->ops->vunmap (dmabuf, map);iosys_map_clear (&dmabuf->vmap_ptr); } -mutex_unlock (&dmabuf->lock); +} + +/** +* dma_buf_vunmap - Unmap a vmap obtained by dma_buf_vmap. +* @dmabuf: [in] buffer to vunmap +* @map: [in] vmap pointer to vunmap +*/ +void dma_buf_vunmap (struct dma_buf *dmabuf,struct iosys_map *map) +{ +if (WARN_ON (!dmabuf)) +return ; + +dma_resv_lock (dmabuf->resv,NULL ); +dma_buf_vunmap_locked (dmabuf, map); +dma_resv_unlock (dmabuf->resv); }EXPORT_SYMBOL_NS_GPL (dma_buf_vunmap,DMA_BUF );drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +4
-2@@ -882,7 +882,8 @@ static int amdgpu_ttm_backend_bind(struct ttm_device *bdev,struct sg_table *sgt; attach = gtt->gobj->import_attach; - sgt =dma_buf_map_attachment (attach,DMA_BIDIRECTIONAL ); + sgt =dma_buf_map_attachment_locked (attach, +DMA_BIDIRECTIONAL );if (IS_ERR (sgt))return PTR_ERR (sgt);@@ -1007,7 +1008,8 @@ static void amdgpu_ttm_backend_unbind(struct ttm_device *bdev,struct dma_buf_attachment *attach; attach = gtt->gobj->import_attach; -dma_buf_unmap_attachment (attach, ttm->sg,DMA_BIDIRECTIONAL ); +dma_buf_unmap_attachment_locked (attach, ttm->sg, +DMA_BIDIRECTIONAL ); ttm->sg =NULL ; }drivers/gpu/drm/drm_client.c +2
-2@@ -323,7 +323,7 @@ drm_client_buffer_vmap(struct drm_client_buffer *buffer,* fd_install step out of the driver backend hooks, to make that * final step optional for internal users. */ - ret =drm_gem_vmap (buffer->gem, map); + ret =drm_gem_vmap_unlocked (buffer->gem, map);if (ret)return ret;@@ -345,7 +345,7 @@ void drm_client_buffer_vunmap(struct drm_client_buffer *buffer) {struct iosys_map *map = &buffer->map; -drm_gem_vunmap (buffer->gem, map); +drm_gem_vunmap_unlocked (buffer->gem, map); }EXPORT_SYMBOL (drm_client_buffer_vunmap);drivers/gpu/drm/drm_gem.c +33
-0@@ -1053,7 +1053,12 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, vma->vm_ops = obj->funcs->vm_ops;if (obj->funcs->mmap) { + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret) + goto err_drm_gem_object_put; + ret = obj->funcs->mmap (obj, vma); +dma_resv_unlock (obj->resv);if (ret) goto err_drm_gem_object_put;WARN_ON (!(vma->vm_flags &VM_DONTEXPAND ));@@ -1158,6 +1163,8 @@ void drm_gem_print_info(struct drm_printer *p, unsigned int indent,int drm_gem_pin (struct drm_gem_object *obj) { +dma_resv_assert_held (obj->resv); +if (obj->funcs->pin)return obj->funcs->pin (obj);else @@ -1166,6 +1173,8 @@ int drm_gem_pin(struct drm_gem_object *obj)void drm_gem_unpin (struct drm_gem_object *obj) { +dma_resv_assert_held (obj->resv); +if (obj->funcs->unpin) obj->funcs->unpin (obj); }@@ -1174,6 +1183,8 @@ int drm_gem_vmap(struct drm_gem_object *obj, struct iosys_map *map) {int ret; +dma_resv_assert_held (obj->resv); +if (!obj->funcs->vmap)return -EOPNOTSUPP ;@@ -1189,6 +1200,8 @@ EXPORT_SYMBOL(drm_gem_vmap);void drm_gem_vunmap (struct drm_gem_object *obj,struct iosys_map *map) { +dma_resv_assert_held (obj->resv); +if (iosys_map_is_null (map))return ;@@ -1200,6 +1213,26 @@ void drm_gem_vunmap(struct drm_gem_object *obj, struct iosys_map *map) }EXPORT_SYMBOL (drm_gem_vunmap); +int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +{ +int ret; + +dma_resv_lock (obj->resv,NULL ); + ret =drm_gem_vmap (obj, map); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL (drm_gem_vmap_unlocked); + +void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map) +{ +dma_resv_lock (obj->resv,NULL ); +drm_gem_vunmap (obj, map); +dma_resv_unlock (obj->resv); +} +EXPORT_SYMBOL (drm_gem_vunmap_unlocked); +/** * drm_gem_lock_reservations - Sets up the ww context and acquires * the lock on an array of GEM objects. drivers/gpu/drm/drm_gem_framebuffer_helper.c +3
-3@@ -348,7 +348,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb,iosys_map_clear (&map[i]);continue ; } - ret =drm_gem_vmap (obj, &map[i]); + ret =drm_gem_vmap_unlocked (obj, &map[i]);if (ret) goto err_drm_gem_vunmap; }@@ -370,7 +370,7 @@ int drm_gem_fb_vmap(struct drm_framebuffer *fb, obj =drm_gem_fb_get_obj (fb, i);if (!obj)continue ; -drm_gem_vunmap (obj, &map[i]); +drm_gem_vunmap_unlocked (obj, &map[i]); }return ret; }@@ -398,7 +398,7 @@ void drm_gem_fb_vunmap(struct drm_framebuffer *fb,continue ;if (iosys_map_is_null (&map[i]))continue ; -drm_gem_vunmap (obj, &map[i]); +drm_gem_vunmap_unlocked (obj, &map[i]); } }EXPORT_SYMBOL (drm_gem_fb_vunmap);drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c +5
-5@@ -72,7 +72,7 @@ static int i915_gem_dmabuf_vmap(struct dma_buf *dma_buf,struct drm_i915_gem_object *obj =dma_buf_to_obj (dma_buf);void *vaddr; - vaddr =i915_gem_object_pin_map_unlocked (obj,I915_MAP_WB ); + vaddr =i915_gem_object_pin_map (obj,I915_MAP_WB );if (IS_ERR (vaddr))return PTR_ERR (vaddr);@@ -241,8 +241,8 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)assert_object_held (obj); - pages =dma_buf_map_attachment (obj->base.import_attach, -DMA_BIDIRECTIONAL ); + pages =dma_buf_map_attachment_locked (obj->base.import_attach, +DMA_BIDIRECTIONAL );if (IS_ERR (pages))return PTR_ERR (pages);@@ -270,8 +270,8 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)static void i915_gem_object_put_pages_dmabuf (struct drm_i915_gem_object *obj,struct sg_table *pages) { -dma_buf_unmap_attachment (obj->base.import_attach, pages, -DMA_BIDIRECTIONAL ); +dma_buf_unmap_attachment_locked (obj->base.import_attach, pages, +DMA_BIDIRECTIONAL ); }static const struct drm_i915_gem_object_ops i915_gem_object_dmabuf_ops = {drivers/gpu/drm/qxl/qxl_object.c +9
-8@@ -168,9 +168,16 @@ int qxl_bo_vmap_locked(struct qxl_bo *bo, struct iosys_map *map) bo->map_count++; goto out; } - r =ttm_bo_vmap (&bo->tbo, &bo->map); + + r =__qxl_bo_pin (bo);if (r)return r; + + r =ttm_bo_vmap (&bo->tbo, &bo->map); +if (r) { +__qxl_bo_unpin (bo); +return r; + } bo->map_count =1 ;/* TODO: Remove kptr in favor of map everywhere. */ @@ -192,12 +199,6 @@ int qxl_bo_vmap(struct qxl_bo *bo, struct iosys_map *map)if (r)return r; - r =__qxl_bo_pin (bo); -if (r) { -qxl_bo_unreserve (bo); -return r; - } - r =qxl_bo_vmap_locked (bo, map);qxl_bo_unreserve (bo);return r;@@ -247,6 +248,7 @@ void qxl_bo_vunmap_locked(struct qxl_bo *bo)return ; bo->kptr =NULL ;ttm_bo_vunmap (&bo->tbo, &bo->map); +__qxl_bo_unpin (bo); }int qxl_bo_vunmap (struct qxl_bo *bo)@@ -258,7 +260,6 @@ int qxl_bo_vunmap(struct qxl_bo *bo)return r;qxl_bo_vunmap_locked (bo); -__qxl_bo_unpin (bo);qxl_bo_unreserve (bo);return 0 ; }drivers/gpu/drm/qxl/qxl_prime.c +2
-2@@ -59,7 +59,7 @@ int qxl_gem_prime_vmap(struct drm_gem_object *obj, struct iosys_map *map)struct qxl_bo *bo =gem_to_qxl_bo (obj);int ret; - ret =qxl_bo_vmap (bo, map); + ret =qxl_bo_vmap_locked (bo, map);if (ret <0 )return ret;@@ -71,5 +71,5 @@ void qxl_gem_prime_vunmap(struct drm_gem_object *obj, {struct qxl_bo *bo =gem_to_qxl_bo (obj); -qxl_bo_vunmap (bo); +qxl_bo_vunmap_locked (bo); }drivers/media/common/videobuf2/videobuf2-dma-contig.c +1
-10@@ -382,18 +382,12 @@ static struct sg_table *vb2_dc_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_dc_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {@@ -409,14 +403,11 @@ static struct sg_table *vb2_dc_dmabuf_ops_map(if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,DMA_ATTR_SKIP_CPU_SYNC )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }drivers/media/common/videobuf2/videobuf2-dma-sg.c +1
-10@@ -424,18 +424,12 @@ static struct sg_table *vb2_dma_sg_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_dma_sg_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {@@ -446,14 +440,11 @@ static struct sg_table *vb2_dma_sg_dmabuf_ops_map(/* mapping to the client with new direction */ if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,0 )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }drivers/media/common/videobuf2/videobuf2-vmalloc.c +1
-10@@ -267,18 +267,12 @@ static struct sg_table *vb2_vmalloc_dmabuf_ops_map( struct dma_buf_attachment *db_attach, enum dma_data_direction dma_dir) { struct vb2_vmalloc_attachment *attach = db_attach->priv; -/* stealing dmabuf mutex to serialize map/unmap operations */ -struct mutex *lock = &db_attach->dmabuf->lock;struct sg_table *sgt; -mutex_lock (lock); - sgt = &attach->sgt;/* return previously mapped sg table */ -if (attach->dma_dir == dma_dir) { -mutex_unlock (lock); +if (attach->dma_dir == dma_dir)return sgt; - }/* release any previous cache */ if (attach->dma_dir !=DMA_NONE ) {@@ -289,14 +283,11 @@ static struct sg_table *vb2_vmalloc_dmabuf_ops_map(/* mapping to the client with new direction */ if (dma_map_sgtable (db_attach->dev, sgt, dma_dir,0 )) {pr_err ("failed to map scatterlist\n" ); -mutex_unlock (lock);return ERR_PTR (-EIO ); } attach->dma_dir = dma_dir; -mutex_unlock (lock); -return sgt; }include/drm/drm_gem.h +3
-0@@ -410,4 +410,7 @@ void drm_gem_unlock_reservations(struct drm_gem_object **objs, int count,int drm_gem_dumb_map_offset (struct drm_file *file,struct drm_device *dev,u32 handle,u64 *offset); +int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +#endif /* __DRM_GEM_H__ */ include/linux/dma-buf.h +5
-9@@ -326,15 +326,6 @@ struct dma_buf {/** @ops: dma_buf_ops associated with this buffer object. */ const struct dma_buf_ops *ops; -/** -* @lock: -* -* Used internally to serialize list manipulation, attach/detach and -* vmap/unmap. Note that in many cases this is superseeded by -* dma_resv_lock() on @resv. -*/ -struct mutex lock; -/** * @vmapping_counter: * @@ -618,6 +609,11 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags);struct dma_buf *dma_buf_get (int fd);void dma_buf_put (struct dma_buf *dmabuf); +struct sg_table *dma_buf_map_attachment_locked (struct dma_buf_attachment *, +enum dma_data_direction ); +void dma_buf_unmap_attachment_locked (struct dma_buf_attachment *, +struct sg_table *, +enum dma_data_direction );struct sg_table *dma_buf_map_attachment (struct dma_buf_attachment *,enum dma_data_direction ); void dma_buf_unmap_attachment(struct dma_buf_attachment *, struct sg_table *,
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Hello Christian,
Quoted 4 lines · Christian König
Hi Dmitry,
First of all please separate out this patch from the rest of the series, since this is a complex separate structural change.
I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
Quoted 6 lines
Am 27.05.22 um 01:50 schrieb Dmitry Osipenko:
Quoted 4 lines
All dma-bufs have dma-reservation lock that allows drivers to perform exclusive operations over shared dma-bufs. Today's dma-buf API has incomplete locking specification, which creates dead lock situation for dma-buf importers and exporters that don't coordinate theirs locks.
Quoted 6 lines
Well please drop that sentence. The locking specifications are actually very well defined, it's just that some drivers are a bit broken regarding them.
What you do here is rather moving all the non-dynamic drivers over to the dynamic locking specification (which is really nice to have).
Indeed, this will be a better description, thank you! I'll update it.
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
Quoted 8 lines
Quoted 8 lines
This patch introduces new locking convention for dma-buf users. From now on all dma-buf importers are responsible for holding dma-buf reservation lock around operations performed over dma-bufs.
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
Quoted 8 lines
Quoted 2 lines
2. Adding new locked variants of the dma-buf API functions for drivers that need to manage imported dma-bufs under the held lock.
Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.This should make it easier to remove those in a follow up patch set and then fully move the locking into the importer.
Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Quoted 5 lines
Quoted 1 line
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Radeon and Nouveau use gem_prime_import_sg_table() and they take resv lock already, seems they should be okay (?)
I assume all the basics should covered in this v6. At minimum Intel, Tegra, Panfrost, Lima and Rockchip drivers should be good. If I missed something, then please let me know and I'll correct it.
Quoted 17 lines
Quoted 17 lines
Signed-off-by Dmitry Osipenko <[email protected]>---
drivers/dma-buf/dma-buf.c | 270 +++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +- drivers/gpu/drm/drm_gem.c | 33 +++ drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 10 +- drivers/gpu/drm/qxl/qxl_object.c | 17 +- drivers/gpu/drm/qxl/qxl_prime.c | 4 +- .../common/videobuf2/videobuf2-dma-contig.c | 11 +-.../media/common/videobuf2/videobuf2-dma-sg.c| 11 +- .../common/videobuf2/videobuf2-vmalloc.c | 11 +- include/drm/drm_gem.h | 3 + include/linux/dma-buf.h | 14 +-13 files changed, 241 insertions(+), 159 deletions(-)
Quoted 11 lines
drivers/dma-buf/dma-buf.c struct dma_buf *dma_buf_export(const struct · line 555
Quoted 9 lines
drivers/dma-buf/dma-buf.c +0
-0@@ -552,7 +552,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) file->f_mode |= FMODE_LSEEK; dmabuf->file = file; - mutex_init(&dmabuf->lock);Please make removing
dmabuf->locka separate change.
Alright
Signature · 2 lines
Best regards, Dmitry
Christian König
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Hi Dmitry,
Am 30.05.22 um 15:26 schrieb Dmitry Osipenko:
Quoted 11 lines
Hello Christian,
Quoted 4 lines · Christian König
Hi Dmitry,
First of all please separate out this patch from the rest of the series, since this is a complex separate structural change.
I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
That sounds like you are underestimating a bit how much trouble this will be.
Quoted 7 lines
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
No, I never even finished creating it after realizing how much work it would be.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
This patch introduces new locking convention for dma-buf users. From now on all dma-buf importers are responsible for holding dma-buf reservation lock around operations performed over dma-bufs.
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers that need to manage imported dma-bufs under the held lock.
Quoted 8 lines
Quoted 5 lines
Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.This should make it easier to remove those in a follow up patch set and then fully move the locking into the importer.
Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Yes, I clearly think so.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Radeon and Nouveau use
gem_prime_import_sg_table()and they take resv lock already, seems they should be okay (?)
You are looking at the wrong side. You need to fix the export code path, not the import ones.
See for example attach on radeon works like this drm_gem_map_attach->drm_gem_pin->radeon_gem_prime_pin->radeon_bo_reserve->ttm_bo_reserve->dma_resv_lock.
Same for nouveau and probably a few other exporters as well. That will certainly cause a deadlock if you don't fix it.
I strongly suggest to do this step by step, first attach/detach and then the rest.
Regards,
Christian.
Quoted 32 lines
I assume all the basics should covered in this v6. At minimum Intel, Tegra, Panfrost, Lima and Rockchip drivers should be good. If I missed something, then please let me know and I'll correct it.
Quoted 17 lines
Quoted 17 lines
Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/dma-buf/dma-buf.c | 270 +++++++++++------- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 6 +- drivers/gpu/drm/drm_client.c | 4 +- drivers/gpu/drm/drm_gem.c | 33 +++ drivers/gpu/drm/drm_gem_framebuffer_helper.c | 6 +- drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 10 +- drivers/gpu/drm/qxl/qxl_object.c | 17 +- drivers/gpu/drm/qxl/qxl_prime.c | 4 +- .../common/videobuf2/videobuf2-dma-contig.c | 11 +- .../media/common/videobuf2/videobuf2-dma-sg.c | 11 +- .../common/videobuf2/videobuf2-vmalloc.c | 11 +- include/drm/drm_gem.h | 3 + include/linux/dma-buf.h | 14 +- 13 files changed, 241 insertions(+), 159 deletions(-)Quoted 10 lines
drivers/dma-buf/dma-buf.c struct dma_buf *dma_buf_export(const struct · line 555
Quoted 9 lines
drivers/dma-buf/dma-buf.c +0
-0@@ -552,7 +552,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info) file->f_mode |= FMODE_LSEEK; dmabuf->file = file; - mutex_init(&dmabuf->lock);Please make removing
dmabuf->locka separate change.
Quoted 2 lines
Alright
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Quoted 58 lines · Christian König
Hi Dmitry,
Am 30.05.22 um 15:26 schrieb Dmitry Osipenko:
Quoted 11 lines
Hello Christian,
Quoted 4 lines · Christian König
Hi Dmitry,
First of all please separate out this patch from the rest of the series, since this is a complex separate structural change.
I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
That sounds like you are underestimating a bit how much trouble this will be.
Quoted 7 lines
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
No, I never even finished creating it after realizing how much work it would be.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
This patch introduces new locking convention for dma-buf users. From now
on all dma-buf importers are responsible for holding dma-buf reservation
lock around operations performed over dma-bufs.This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers
that need to manage imported dma-bufs under the held lock.Quoted 8 lines
Quoted 5 lines
Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.This should make it easier to remove those in a follow up patch set and then fully move the locking into the importer.
Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Yes, I clearly think so.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Quoted 8 lines
Quoted 2 lines
Radeon and Nouveau use
gem_prime_import_sg_table()and they take resv lock already, seems they should be okay (?)You are looking at the wrong side. You need to fix the export code path, not the import ones.
See for example attach on radeon works like this
drm_gem_map_attach->drm_gem_pin->radeon_gem_prime_pin->radeon_bo_reserve->ttm_bo_reserve->dma_resv_lock.
Yeah, I was looking at the both sides, but missed this one.
Quoted 5 lines
Same for nouveau and probably a few other exporters as well. That will certainly cause a deadlock if you don't fix it.
I strongly suggest to do this step by step, first attach/detach and then the rest.
Thank you very much for the suggestions. I'll implement them in the next version.
Signature · 2 lines
Best regards, Dmitry
Re: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Quoted 55 lines · Dmitry Osipenko
Quoted 54 lines · Christian König
Hi Dmitry,
Am 30.05.22 um 15:26 schrieb Dmitry Osipenko:
Quoted 11 lines
Hello Christian,
Quoted 4 lines · Christian König
Hi Dmitry,
First of all please separate out this patch from the rest of the series, since this is a complex separate structural change.
I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
That sounds like you are underestimating a bit how much trouble this will be.
Quoted 7 lines
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
No, I never even finished creating it after realizing how much work it would be.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
This patch introduces new locking convention for dma-buf users. From now
on all dma-buf importers are responsible for holding dma-buf reservation
lock around operations performed over dma-bufs.This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for
drivers
that need to manage imported dma-bufs under the held lock.Quoted 8 lines
Quoted 5 lines
Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.This should make it easier to remove those in a follow up patch set and then fully move the locking into the importer.
Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Yes, I clearly think so.
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
Radeon and Nouveau use
gem_prime_import_sg_table()and they take resv lock already, seems they should be okay (?)You are looking at the wrong side. You need to fix the export code path, not the import ones.
See for example attach on radeon works like this
drm_gem_map_attach->drm_gem_pin->radeon_gem_prime_pin->radeon_bo_reserve->ttm_bo_reserve->dma_resv_lock.Yeah, I was looking at the both sides, but missed this one.
Also i915 will run into trouble with attach. In particular since i915 starts a full ww transaction in its attach callback to be able to lock other objects if migration is needed. I think i915 CI would catch this in a selftest.
Perhaps it's worthwile to take a step back and figure out, if the importer is required to lock, which callbacks might need a ww acquire context?
(And off-topic, Since we do a lot of fancy stuff under dma-resv locks including waiting for fences and other locks, IMO taking these locks uninterruptible should ring a warning bell)
/Thomas
Quoted 9 lines
Quoted 6 lines
Same for nouveau and probably a few other exporters as well. That will certainly cause a deadlock if you don't fix it.
I strongly suggest to do this step by step, first attach/detach and then the rest.
Thank you very much for the suggestions. I'll implement them in the next version.
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Quoted 67 lines · Thomas Hellström (Intel)
Quoted 65 lines · Dmitry Osipenko
Quoted 64 lines · Christian König
Hi Dmitry,
Am 30.05.22 um 15:26 schrieb Dmitry Osipenko:
Quoted 5 lines
Hello Christian,
Quoted 2 lines · Christian König
Hi Dmitry,
Quoted 7 lines
Quoted 3 lines
First of all please separate out this patch from the rest of the series,
since this is a complex separate structural change.I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
That sounds like you are underestimating a bit how much trouble this will be.
Quoted 7 lines
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
No, I never even finished creating it after realizing how much work it would be.
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
This patch introduces new locking convention for dma-buf users. From now
on all dma-buf importers are responsible for holding dma-buf reservation
lock around operations performed over dma-bufs.Quoted 7 lines
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers
that need to manage imported dma-bufs under the held lock.Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.Quoted 6 lines
Quoted 3 lines
This should make it easier to remove those in a follow up patch set and
then fully move the locking into the importer.Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Yes, I clearly think so.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Radeon and Nouveau use
gem_prime_import_sg_table()and they take resv lock already, seems they should be okay (?)You are looking at the wrong side. You need to fix the export code path, not the import ones.
See for example attach on radeon works like this
drm_gem_map_attach->drm_gem_pin->radeon_gem_prime_pin->radeon_bo_reserve->ttm_bo_reserve->dma_resv_lock.
Quoted 6 lines
Quoted 1 line
Yeah, I was looking at the both sides, but missed this one.
Also i915 will run into trouble with attach. In particular since i915 starts a full ww transaction in its attach callback to be able to lock other objects if migration is needed. I think i915 CI would catch this in a selftest.
Seems it indeed it should deadlock. But i915 selftests apparently should've caught it and they didn't, I'll re-check what happened.
Quoted 3 lines
Perhaps it's worthwile to take a step back and figure out, if the importer is required to lock, which callbacks might need a ww acquire context?
I'll take this into account, thanks.
Quoted 3 lines
(And off-topic, Since we do a lot of fancy stuff under dma-resv locks including waiting for fences and other locks, IMO taking these locks uninterruptible should ring a warning bell)
I had the same thought and had a version that used the interruptible locking variant, but then decided to fall back to the uninterruptible, don't remember why. I'll revisit this.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
Quoted 67 lines · Dmitry Osipenko
Quoted 66 lines · Thomas Hellström (Intel)
Quoted 65 lines · Dmitry Osipenko
Quoted 64 lines · Christian König
Hi Dmitry,
Am 30.05.22 um 15:26 schrieb Dmitry Osipenko:
Quoted 5 lines
Hello Christian,
Quoted 2 lines · Christian König
Hi Dmitry,
Quoted 7 lines
Quoted 3 lines
First of all please separate out this patch from the rest of the series,
since this is a complex separate structural change.I assume all the patches will go via the DRM tree in the end since the rest of the DRM patches in this series depend on this dma-buf change. But I see that separation may ease reviewing of the dma-buf changes, so let's try it.
That sounds like you are underestimating a bit how much trouble this will be.
Quoted 7 lines
Quoted 3 lines
I have tried this before and failed because catching all the locks in the right code paths are very tricky. So expect some fallout from this and make sure the kernel test robot and CI systems are clean.
Sure, I'll fix up all the reported things in the next iteration.
BTW, have you ever posted yours version of the patch? Will be great if we could compare the changed code paths.
No, I never even finished creating it after realizing how much work it would be.
Quoted 16 lines
Quoted 16 lines
Quoted 6 lines
This patch introduces new locking convention for dma-buf users. From now
on all dma-buf importers are responsible for holding dma-buf reservation
lock around operations performed over dma-bufs.Quoted 7 lines
This patch implements the new dma-buf locking convention by:
1. Making dma-buf API functions to take the reservation lock.
2. Adding new locked variants of the dma-buf API functions for drivers
that need to manage imported dma-bufs under the held lock.Instead of adding new locked variants please mark all variants which expect to be called without a lock with an
_unlockedpostfix.Quoted 6 lines
Quoted 3 lines
This should make it easier to remove those in a follow up patch set and
then fully move the locking into the importer.Do we really want to move all the locks to the importers? Seems the majority of drivers should be happy with the dma-buf helpers handling the locking for them.
Yes, I clearly think so.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
3. Converting all drivers to the new locking scheme.
I have strong doubts that you got all of them. At least radeon and nouveau should grab the reservation lock in their ->attach callbacks somehow.
Radeon and Nouveau use
gem_prime_import_sg_table()and they take resv lock already, seems they should be okay (?)You are looking at the wrong side. You need to fix the export code path, not the import ones.
See for example attach on radeon works like this
drm_gem_map_attach->drm_gem_pin->radeon_gem_prime_pin->radeon_bo_reserve->ttm_bo_reserve->dma_resv_lock.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Yeah, I was looking at the both sides, but missed this one.
Also i915 will run into trouble with attach. In particular since i915 starts a full ww transaction in its attach callback to be able to lock other objects if migration is needed. I think i915 CI would catch this in a selftest.
Seems it indeed it should deadlock. But i915 selftests apparently should've caught it and they didn't, I'll re-check what happened.
The i915 selftests use a separate mock_dmabuf_ops. That's why it works for the selftests, i.e. there is no deadlock.
Thomas, would i915 CI run a different set of tests or will it be the default i915 selftests ran by IGT?
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerRe: [PATCH v6 14/22] dma-buf: Introduce new locking convention
On 7/5/22 01:38, Dmitry Osipenko wrote:
...
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Also i915 will run into trouble with attach. In particular since i915 starts a full ww transaction in its attach callback to be able to lock other objects if migration is needed. I think i915 CI would catch this in a selftest.
Seems it indeed it should deadlock. But i915 selftests apparently should've caught it and they didn't, I'll re-check what happened.
Quoted 6 lines
The i915 selftests use a separate
mock_dmabuf_ops. That's why it works for the selftests, i.e. there is no deadlock.Thomas, would i915 CI run a different set of tests or will it be the default i915 selftests ran by IGT?
Nevermind, I had a local kernel change that was forgotten about.. it prevented the i915 live tests from running.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerPATCH v6 15/22 drm/shmem-helper: Don't use vmap_use_count for dma-bufs
In reply to Dmitry OsipenkoThere is no need to refcount vmappings of dma-bufs because dma-buf core has its own refcounting. Drop the refcounting of dma-bufs. This will ease replacing of all drm-shmem locks with a single dma-buf reservation lock, preparing drm-shmem code for addition of the generic drm-shmem shrinker.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 35 +++++++++++++++----------- 1 file changed, 20 insertions(+), 15 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +20 -15
@@ -293,24 +293,22 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,struct drm_gem_object *obj = &shmem->base;int ret =0 ; -if (shmem->vmap_use_count++ >0 ) { -iosys_map_set_vaddr (map, shmem->vaddr); -return 0 ; - } -if (obj->import_attach) { ret =dma_buf_vmap (obj->import_attach->dmabuf, map);if (!ret) {if (WARN_ON (map->is_iomem)) {dma_buf_vunmap (obj->import_attach->dmabuf, map); - ret = -EIO ; - goto err_put_pages; +return -EIO ; } - shmem->vaddr = map->vaddr; } }else {pgprot_t prot =PAGE_KERNEL ; +if (shmem->vmap_use_count++ >0 ) { +iosys_map_set_vaddr (map, shmem->vaddr); +return 0 ; + } + ret =drm_gem_shmem_get_pages (shmem);if (ret) goto err_zero_use;
@@ -376,15 +374,15 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, {struct drm_gem_object *obj = &shmem->base; -if (WARN_ON_ONCE (!shmem->vmap_use_count)) -return ; - -if (--shmem->vmap_use_count >0 ) -return ; -if (obj->import_attach) {dma_buf_vunmap (obj->import_attach->dmabuf, map); }else { +if (WARN_ON_ONCE (!shmem->vmap_use_count)) +return ; + +if (--shmem->vmap_use_count >0 ) +return ; +vunmap (shmem->vaddr);drm_gem_shmem_put_pages (shmem); }
@@ -637,7 +635,14 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem, struct drm_printer *p, unsigned int indent) { drm_printf_indent(p, indent, "pages_use_count=%u\n", shmem->pages_use_count); -drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); + +if (shmem->base.import_attach) +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->base.dma_buf->vmapping_counter); +else +drm_printf_indent (p, indent,"vmap_use_count=%u\n" , + shmem->vmap_use_count); +drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); }EXPORT_SYMBOL (drm_gem_shmem_print_info);
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 16/22 drm/shmem-helper: Use reservation lock
In reply to Dmitry OsipenkoReplace drm_gem_shmem locks with GEM reservation lock to make drm-shmem locks consistent with the new locking convention of dma-bufs which tells that dma-buf importers are responsible for holding reservation lock for all operations performed over dma-bufs. This prepares drm-shmem code for addition of the generic shmem shrinker framework.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 181 +++++++----------- drivers/gpu/drm/lima/lima_gem.c | 8 +- drivers/gpu/drm/lima/lima_sched.c | 4 +- drivers/gpu/drm/panfrost/panfrost_drv.c | 7 +- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 +- drivers/gpu/drm/panfrost/panfrost_mmu.c | 19 +- drivers/gpu/drm/panfrost/panfrost_perfcnt.c | 6 +- include/drm/drm_gem_shmem_helper.h | 14 +- 8 files changed, 97 insertions(+), 148 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +68 -113
@@ -86,8 +86,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)if (ret) goto err_release; -mutex_init (&shmem->pages_lock); -mutex_init (&shmem->vmap_lock);INIT_LIST_HEAD (&shmem->madv_list);if (!private) {
@@ -139,11 +137,13 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -WARN_ON (shmem->vmap_use_count); -if (obj->import_attach) {drm_prime_gem_destroy (obj, shmem->sgt); }else { +dma_resv_lock (shmem->base.resv,NULL ); + +WARN_ON (shmem->vmap_use_count); +if (shmem->sgt) {dma_unmap_sgtable (obj->dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );
@@ -152,18 +152,18 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }if (shmem->pages)drm_gem_shmem_put_pages (shmem); - } -WARN_ON (shmem->pages_use_count); +WARN_ON (shmem->pages_use_count); + +dma_resv_unlock (shmem->base.resv); + }drm_gem_object_release (obj); -mutex_destroy (&shmem->pages_lock); -mutex_destroy (&shmem->vmap_lock);kfree (shmem); }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages_locked (struct drm_gem_shmem_object *shmem) +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages;
@@ -194,35 +194,17 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem) }/* -* drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object +* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object * -* This function makes sure that backing pages exists for the shmem GEM object -* and increases the use count. -* -* Returns: -* 0 on success or a negative error code on failure. +* This function decreases the use count and puts the backing pages when use drops to zero. */ -int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) -{ -int ret; - -WARN_ON (shmem->base.import_attach); - - ret =mutex_lock_interruptible (&shmem->pages_lock); -if (ret) -return ret; - ret =drm_gem_shmem_get_pages_locked (shmem); -mutex_unlock (&shmem->pages_lock); - -return ret; -} -EXPORT_SYMBOL (drm_gem_shmem_get_pages); - -static void drm_gem_shmem_put_pages_locked (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; +dma_resv_assert_held (shmem->base.resv); +if (WARN_ON_ONCE (!shmem->pages_use_count))return ;
@@ -239,19 +221,6 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem) shmem->pages_mark_accessed_on_put); shmem->pages =NULL ; } - -/* -* drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object -* @shmem: shmem GEM object -* -* This function decreases the use count and puts the backing pages when use drops to zero. -*/ -void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) -{ -mutex_lock (&shmem->pages_lock); -drm_gem_shmem_put_pages_locked (shmem); -mutex_unlock (&shmem->pages_lock); -}EXPORT_SYMBOL (drm_gem_shmem_put_pages);/**
@@ -266,6 +235,8 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +dma_resv_assert_held (shmem->base.resv); +WARN_ON (shmem->base.import_attach);return drm_gem_shmem_get_pages (shmem);
@@ -281,14 +252,31 @@ EXPORT_SYMBOL(drm_gem_shmem_pin);*/ void drm_gem_shmem_unpin (struct drm_gem_shmem_object *shmem) { +dma_resv_assert_held (shmem->base.resv); +WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); }EXPORT_SYMBOL (drm_gem_shmem_unpin); -static int drm_gem_shmem_vmap_locked (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +/* +* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object +* @shmem: shmem GEM object +* @map: Returns the kernel virtual address of the SHMEM GEM object's backing +* store. +* +* This function makes sure that a contiguous kernel virtual address mapping +* exists for the buffer backing the shmem GEM object. It hides the differences +* between dma-buf imported and natively allocated objects. +* +* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap(). +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;int ret =0 ;
@@ -304,6 +292,8 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem, }else {pgprot_t prot =PAGE_KERNEL ; +dma_resv_assert_held (shmem->base.resv); +if (shmem->vmap_use_count++ >0 ) {iosys_map_set_vaddr (map, shmem->vaddr);return 0 ;
@@ -338,45 +328,30 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,return ret; } +EXPORT_SYMBOL (drm_gem_shmem_vmap);/* -* drm_gem_shmem_vmap - Create a virtual mapping for a shmem GEM object +* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object * @shmem: shmem GEM object -* @map: Returns the kernel virtual address of the SHMEM GEM object's backing -* store. -* -* This function makes sure that a contiguous kernel virtual address mapping -* exists for the buffer backing the shmem GEM object. It hides the differences -* between dma-buf imported and natively allocated objects. +* @map: Kernel virtual address where the SHMEM GEM object was mapped * -* Acquired mappings should be cleaned up by calling drm_gem_shmem_vunmap(). +* This function cleans up a kernel virtual address mapping acquired by +* drm_gem_shmem_vmap(). The mapping is only removed when the use count drops to +* zero. * -* Returns: -* 0 on success or a negative error code on failure. +* This function hides the differences between dma-buf imported and natively +* allocated objects. */ -int drm_gem_shmem_vmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) -{ -int ret; - - ret =mutex_lock_interruptible (&shmem->vmap_lock); -if (ret) -return ret; - ret =drm_gem_shmem_vmap_locked (shmem, map); -mutex_unlock (&shmem->vmap_lock); - -return ret; -} -EXPORT_SYMBOL (drm_gem_shmem_vmap); - -static void drm_gem_shmem_vunmap_locked (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) +void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, +struct iosys_map *map) {struct drm_gem_object *obj = &shmem->base;if (obj->import_attach) {dma_buf_vunmap (obj->import_attach->dmabuf, map); }else { +dma_resv_assert_held (shmem->base.resv); +if (WARN_ON_ONCE (!shmem->vmap_use_count))return ;
@@ -389,26 +364,6 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem, shmem->vaddr =NULL ; } - -/* -* drm_gem_shmem_vunmap - Unmap a virtual mapping for a shmem GEM object -* @shmem: shmem GEM object -* @map: Kernel virtual address where the SHMEM GEM object was mapped -* -* This function cleans up a kernel virtual address mapping acquired by -* drm_gem_shmem_vmap(). The mapping is only removed when the use count drops to -* zero. -* -* This function hides the differences between dma-buf imported and natively -* allocated objects. -*/ -void drm_gem_shmem_vunmap (struct drm_gem_shmem_object *shmem, -struct iosys_map *map) -{ -mutex_lock (&shmem->vmap_lock); -drm_gem_shmem_vunmap_locked (shmem, map); -mutex_unlock (&shmem->vmap_lock); -}EXPORT_SYMBOL (drm_gem_shmem_vunmap); static struct drm_gem_shmem_object *
@@ -441,24 +396,24 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,*/ int drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv) { -mutex_lock (&shmem->pages_lock); +dma_resv_assert_held (shmem->base.resv);if (shmem->madv >=0 ) shmem->madv = madv; madv = shmem->madv; -mutex_unlock (&shmem->pages_lock); -return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem) +void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct drm_device *dev = obj->dev; +dma_resv_assert_held (shmem->base.resv); +WARN_ON (!drm_gem_shmem_is_purgeable (shmem));dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 );
@@ -466,7 +421,7 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)kfree (shmem->sgt); shmem->sgt =NULL ; -drm_gem_shmem_put_pages_locked (shmem); +drm_gem_shmem_put_pages (shmem); shmem->madv =-1 ;
@@ -482,17 +437,6 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); } -EXPORT_SYMBOL (drm_gem_shmem_purge_locked); - -bool drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) -{ -if (!mutex_trylock (&shmem->pages_lock)) -return false; -drm_gem_shmem_purge_locked (shmem); -mutex_unlock (&shmem->pages_lock); - -return true; -}EXPORT_SYMBOL (drm_gem_shmem_purge);/**
@@ -548,7 +492,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ; -mutex_lock (&shmem->pages_lock); +dma_resv_lock (shmem->base.resv,NULL ); if (page_offset >= num_pages || WARN_ON_ONCE(!shmem->pages) ||
@@ -560,7 +504,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf) ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } -mutex_unlock (&shmem->pages_lock); +dma_resv_unlock (shmem->base.resv);return ret; }
@@ -573,8 +517,10 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)WARN_ON (shmem->base.import_attach); +dma_resv_lock (shmem->base.resv,NULL ); ret =drm_gem_shmem_get_pages (shmem);WARN_ON_ONCE (ret !=0 ); +dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma); }
@@ -584,7 +530,10 @@ static void drm_gem_shmem_vm_close(struct vm_area_struct *vma)struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); +dma_resv_lock (shmem->base.resv,NULL );drm_gem_shmem_put_pages (shmem); +dma_resv_unlock (shmem->base.resv); +drm_gem_vm_close (vma); }
@@ -700,9 +649,11 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)WARN_ON (obj->import_attach); +dma_resv_lock (shmem->base.resv,NULL ); + ret =drm_gem_shmem_get_pages (shmem);if (ret) -return ERR_PTR (ret); + goto err_unlock; sgt =drm_gem_shmem_get_sg_table (shmem);if (IS_ERR (sgt)) {
@@ -716,6 +667,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +dma_resv_unlock (shmem->base.resv); +return sgt; err_free_sgt:
@@ -723,6 +676,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)kfree (sgt); err_put_pages:drm_gem_shmem_put_pages (shmem); +err_unlock: +dma_resv_unlock (shmem->base.resv);return ERR_PTR (ret); }EXPORT_SYMBOL_GPL (drm_gem_shmem_get_pages_sgt);
drivers/gpu/drm/lima/lima_gem.c +4 -4
@@ -34,7 +34,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) new_size =min (new_size, bo->base.base.size); -mutex_lock (&bo->base.pages_lock); +dma_resv_lock (bo->base.base.resv,NULL );if (bo->base.pages) { pages = bo->base.pages;
@@ -42,7 +42,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm) pages =kvmalloc_array (bo->base.base.size >>PAGE_SHIFT ,sizeof (*pages),GFP_KERNEL | __GFP_ZERO);if (!pages) { -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv);return -ENOMEM ; }
@@ -56,13 +56,13 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)struct page *page =shmem_read_mapping_page (mapping, i);if (IS_ERR (page)) { -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv);return PTR_ERR (page); } pages[i] = page; } -mutex_unlock (&bo->base.pages_lock); +dma_resv_unlock (bo->base.base.resv); ret =sg_alloc_table_from_pages (&sgt, pages, i,0 , new_size,GFP_KERNEL );
drivers/gpu/drm/lima/lima_sched.c +2 -2
@@ -371,7 +371,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task) }else { buffer_chunk->size =lima_bo_size (bo); - ret =drm_gem_shmem_vmap (&bo->base, &map); + ret =drm_gem_vmap_unlocked (&bo->base.base, &map);if (ret) {kvfree (et); goto out;
@@ -379,7 +379,7 @@ static void lima_sched_build_error_task_list(struct lima_sched_task *task)memcpy (buffer_chunk +1 , map.vaddr, buffer_chunk->size); -drm_gem_shmem_vunmap (&bo->base, &map); +drm_gem_vunmap_unlocked (&bo->base.base, &map); } buffer_chunk = (void *)(buffer_chunk +1 ) + buffer_chunk->size;
drivers/gpu/drm/panfrost/panfrost_drv.c +6 -1
@@ -405,6 +405,10 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, bo =to_panfrost_bo (gem_obj); + ret =dma_resv_lock_interruptible (bo->base.base.resv,NULL ); +if (ret) + goto out_put_object; +mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {
@@ -442,7 +446,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, out_unlock_mappings:mutex_unlock (&bo->mappings.lock);mutex_unlock (&pfdev->shrinker_lock); - +dma_resv_unlock (bo->base.base.resv); +out_put_object:drm_gem_object_put (gem_obj);return ret; }
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +3 -3
@@ -48,14 +48,14 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)if (!mutex_trylock (&bo->mappings.lock))return false; -if (!mutex_trylock (&shmem->pages_lock)) +if (!dma_resv_trylock (shmem->base.resv)) goto unlock_mappings;panfrost_gem_teardown_mappings_locked (bo); -drm_gem_shmem_purge_locked (&bo->base); +drm_gem_shmem_purge (&bo->base); ret = true; -mutex_unlock (&shmem->pages_lock); +dma_resv_unlock (shmem->base.resv); unlock_mappings:mutex_unlock (&bo->mappings.lock);
drivers/gpu/drm/panfrost/panfrost_mmu.c +10 -9
@@ -424,6 +424,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,struct panfrost_gem_mapping *bomapping;struct panfrost_gem_object *bo;struct address_space *mapping; +struct drm_gem_object *obj;pgoff_t page_offset;struct sg_table *sgt;struct page **pages;
@@ -446,15 +447,16 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, page_offset = addr >>PAGE_SHIFT ; page_offset -= bomapping->mmnode.start; -mutex_lock (&bo->base.pages_lock); + obj = &bo->base.base; + +dma_resv_lock (obj->resv,NULL );if (!bo->base.pages) { bo->sgts =kvmalloc_array (bo->base.base.size /SZ_2M ,sizeof (struct sg_table ),GFP_KERNEL | __GFP_ZERO);if (!bo->sgts) { -mutex_unlock (&bo->base.pages_lock); ret = -ENOMEM ; - goto err_bo; + goto err_unlock; } pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
@@ -462,9 +464,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,if (!pages) {kvfree (bo->sgts); bo->sgts =NULL ; -mutex_unlock (&bo->base.pages_lock); ret = -ENOMEM ; - goto err_bo; + goto err_unlock; } bo->base.pages = pages; bo->base.pages_use_count =1 ;
@@ -472,7 +473,6 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as, pages = bo->base.pages;if (pages[page_offset]) {/* Pages are already mapped, bail out. */ -mutex_unlock (&bo->base.pages_lock); goto out; } }
@@ -483,14 +483,11 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,for (i = page_offset; i < page_offset +NUM_FAULT_PAGES ; i++) { pages[i] =shmem_read_mapping_page (mapping, i);if (IS_ERR (pages[i])) { -mutex_unlock (&bo->base.pages_lock); ret =PTR_ERR (pages[i]); goto err_pages; } } -mutex_unlock (&bo->base.pages_lock); - sgt = &bo->sgts[page_offset / (SZ_2M /PAGE_SIZE )]; ret =sg_alloc_table_from_pages (sgt, pages + page_offset,NUM_FAULT_PAGES ,0 ,SZ_2M ,GFP_KERNEL );
@@ -509,6 +506,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,dev_dbg (pfdev->dev,"mapped page fault @ AS%d %llx" , as, addr); out: +dma_resv_unlock (obj->resv); +panfrost_gem_mapping_put (bomapping);return 0 ;
@@ -517,6 +516,8 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,sg_free_table (sgt); err_pages:drm_gem_shmem_put_pages (&bo->base); +err_unlock: +dma_resv_unlock (obj->resv); err_bo:panfrost_gem_mapping_put (bomapping);return ret;
drivers/gpu/drm/panfrost/panfrost_perfcnt.c +3 -3
@@ -106,7 +106,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev, goto err_close_bo; } - ret =drm_gem_shmem_vmap (bo, &map); + ret =drm_gem_vmap_unlocked (&bo->base, &map);if (ret) goto err_put_mapping; perfcnt->buf = map.vaddr;
@@ -165,7 +165,7 @@ static int panfrost_perfcnt_enable_locked(struct panfrost_device *pfdev,return 0 ; err_vunmap: -drm_gem_shmem_vunmap (bo, &map); +drm_gem_vunmap_unlocked (&bo->base, &map); err_put_mapping:panfrost_gem_mapping_put (perfcnt->mapping); err_close_bo:
@@ -195,7 +195,7 @@ static int panfrost_perfcnt_disable_locked(struct panfrost_device *pfdev, GPU_PERFCNT_CFG_MODE(GPU_PERFCNT_CFG_MODE_OFF)); perfcnt->user =NULL ; -drm_gem_shmem_vunmap (&perfcnt->mapping->obj->base, &map); +drm_gem_vunmap_unlocked (&perfcnt->mapping->obj->base.base, &map); perfcnt->buf =NULL ;panfrost_gem_close (&perfcnt->mapping->obj->base.base, file_priv);panfrost_mmu_as_put (pfdev, perfcnt->mapping->mmu);
include/drm/drm_gem_shmem_helper.h +1 -13
@@ -26,11 +26,6 @@ struct drm_gem_shmem_object {*/ struct drm_gem_object base; -/** -* @pages_lock: Protects the page table and use count -*/ -struct mutex pages_lock; -/** * @pages: Page table */
@@ -79,11 +74,6 @@ struct drm_gem_shmem_object {*/ struct sg_table *sgt; -/** -* @vmap_lock: Protects the vmap address and use count -*/ -struct mutex vmap_lock; -/** * @vaddr: Kernel virtual address of the backing memory */
@@ -109,7 +99,6 @@ struct drm_gem_shmem_object {struct drm_gem_shmem_object *drm_gem_shmem_create (struct drm_device *dev,size_t size);void drm_gem_shmem_free (struct drm_gem_shmem_object *shmem); -int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem);int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem);void drm_gem_shmem_unpin (struct drm_gem_shmem_object *shmem);
@@ -128,8 +117,7 @@ static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem !shmem->base.dma_buf && !shmem->base.import_attach; } -void drm_gem_shmem_purge_locked (struct drm_gem_shmem_object *shmem); -bool drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 17/22 drm/shmem-helper: Add generic memory shrinker
In reply to Dmitry OsipenkoIntroduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new evict() shmem object callback.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable(shmem) and alike API functions to activate shrinking of shmem GEMs.
This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +497 -43
@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free
@@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {
@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);
@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }
@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object
@@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))
@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);
@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);
@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);
@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;
@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;
@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;
@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
@@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;
@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);
@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)
@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);
@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;
@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8 -1
@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {
@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }
drivers/gpu/drm/virtio/virtgpu_drv.h +3 -0
@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;
@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */
include/drm/drm_device.h +4 -0
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;
@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: *
include/drm/drm_gem_shmem_helper.h +82 -5
132 lines not on this page · raw message
Signature · 1 line
2.35.3
Dan Carpenter
build bot[kbuild] Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
CC: kbuild-all(a)lists.01.org BCC: lkp(a)intel.com In-Reply-To: <[email protected]> References: <[email protected]> TO…
CC: kbuild-all(a)lists.01.org
BCC: lkp(a)intel.com
In-Reply-To: <[email protected]>
References: <[email protected]>
TO: Dmitry Osipenko <[email protected]>
TO: David Airlie <[email protected]>
TO: Gerd Hoffmann <[email protected]>
TO: Gurchetan Singh <[email protected]>
TO: "Chia-I Wu" <[email protected]>
TO: Daniel Vetter <[email protected]>
TO: Daniel Almeida <[email protected]>
TO: Gert Wollny <[email protected]>
TO: Gustavo Padovan <[email protected]>
TO: Daniel Stone <[email protected]>
TO: Tomeu Vizoso <[email protected]>
TO: Maarten Lankhorst <[email protected]>
TO: Maxime Ripard <[email protected]>
TO: Thomas Zimmermann <[email protected]>
TO: Rob Herring <[email protected]>
TO: Steven Price <[email protected]>
TO: Alyssa Rosenzweig <[email protected]>
TO: Rob Clark <[email protected]>
TO: Emil Velikov <[email protected]>
TO: Robin Murphy <[email protected]>
TO: Qiang Yu <[email protected]>
TO: Sumit Semwal <[email protected]>
TO: "Christian König" <[email protected]>
TO: "Pan, Xinhui" <[email protected]>
TO: Thierry Reding <[email protected]>
TO: Tomasz Figa <[email protected]>
TO: Marek Szyprowski <[email protected]>
TO: Mauro Carvalho Chehab <[email protected]>
CC: linux-media(a)vger.kernel.org
TO: Alex Deucher <[email protected]>
TO: Jani Nikula <[email protected]>
Hi Dmitry,
I love your patch! Perhaps something to improve:
[auto build test WARNING on linus/master]
[also build test WARNING on next-20220527]
[cannot apply to drm/drm-next media-tree/master drm-intel/for-linux-next v5.18] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/intel-lab-lkp/linux/commits/Dmitry-Osipenko/Add-generic-memory-shrinker-to-VirtIO-GPU-and-Panfrost-DRM-drivers/20220527-075717 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cdeffe87f790dfd1baa193020411ce9a538446d7
:::::: branch date: 14 hours ago
:::::: commit date: 14 hours ago
config: i386-randconfig-m021 (https://download.01.org/0day-ci/archive/20220527/202205272155.MxDiru43-lkp(a)intel.com/config)
compiler: gcc-11 (Debian 11.3.0-1) 11.3.0
If you fix the issue, kindly add following tag where applicable
New smatch warnings:drivers/gpu/drm/drm_gem_shmem_helper.c:226 drm_gem_shmem_set_evictable() warn: inconsistent returns 'shmem->base.resv'. drivers/gpu/drm/drm_gem_shmem_helper.c:253 drm_gem_shmem_set_purgeable() warn: inconsistent returns 'shmem->base.resv'. drivers/gpu/drm/drm_gem_shmem_helper.c:703 drm_gem_shmem_fault() error: we previously assumed 'shmem->pages' could be null (see line 694)
Old smatch warnings:include/drm/drm_vma_manager.h:225 drm_vma_node_unmap() warn: should 'drm_vma_node_size(node) << 12' be a 64 bit type?
vim +226 drivers/gpu/drm/drm_gem_shmem_helper.c
2194a63a818db7 Noralf Trønnes 2019-03-12 204
875fd932de880a Dmitry Osipenko 2022-05-27 205 /**
875fd932de880a Dmitry Osipenko 2022-05-27 206 * drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker
875fd932de880a Dmitry Osipenko 2022-05-27 207 * @shmem: shmem GEM object
875fd932de880a Dmitry Osipenko 2022-05-27 208 *
875fd932de880a Dmitry Osipenko 2022-05-27 209 * Tell memory shrinker that this GEM can be evicted. Initially eviction is
875fd932de880a Dmitry Osipenko 2022-05-27 210 * disabled for all GEMs. If GEM was purged, then -ENOMEM is returned.
875fd932de880a Dmitry Osipenko 2022-05-27 211 *
875fd932de880a Dmitry Osipenko 2022-05-27 212 * Returns:
875fd932de880a Dmitry Osipenko 2022-05-27 213 * 0 on success or a negative error code on failure.
875fd932de880a Dmitry Osipenko 2022-05-27 214 */
875fd932de880a Dmitry Osipenko 2022-05-27 215 int drm_gem_shmem_set_evictable(struct drm_gem_shmem_object *shmem)
875fd932de880a Dmitry Osipenko 2022-05-27 216 {
875fd932de880a Dmitry Osipenko 2022-05-27 217 dma_resv_lock(shmem->base.resv, NULL);
875fd932de880a Dmitry Osipenko 2022-05-27 218
875fd932de880a Dmitry Osipenko 2022-05-27 219 if (shmem->madv < 0)
875fd932de880a Dmitry Osipenko 2022-05-27 220 return -ENOMEM;
875fd932de880a Dmitry Osipenko 2022-05-27 221
875fd932de880a Dmitry Osipenko 2022-05-27 222 shmem->eviction_enabled = true;
875fd932de880a Dmitry Osipenko 2022-05-27 223
875fd932de880a Dmitry Osipenko 2022-05-27 224 dma_resv_unlock(shmem->base.resv);
875fd932de880a Dmitry Osipenko 2022-05-27 225
875fd932de880a Dmitry Osipenko 2022-05-27 @226 return 0;
875fd932de880a Dmitry Osipenko 2022-05-27 227 }
875fd932de880a Dmitry Osipenko 2022-05-27 228 EXPORT_SYMBOL_GPL(drm_gem_shmem_set_evictable);
875fd932de880a Dmitry Osipenko 2022-05-27 229
875fd932de880a Dmitry Osipenko 2022-05-27 230 /**
875fd932de880a Dmitry Osipenko 2022-05-27 231 * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker
875fd932de880a Dmitry Osipenko 2022-05-27 232 * @shmem: shmem GEM object
875fd932de880a Dmitry Osipenko 2022-05-27 233 *
875fd932de880a Dmitry Osipenko 2022-05-27 234 * Tell memory shrinker that this GEM can be purged. Initially purging is
875fd932de880a Dmitry Osipenko 2022-05-27 235 * disabled for all GEMs. If GEM was purged, then -ENOMEM is returned.
875fd932de880a Dmitry Osipenko 2022-05-27 236 *
875fd932de880a Dmitry Osipenko 2022-05-27 237 * Returns:
875fd932de880a Dmitry Osipenko 2022-05-27 238 * 0 on success or a negative error code on failure.
875fd932de880a Dmitry Osipenko 2022-05-27 239 */
875fd932de880a Dmitry Osipenko 2022-05-27 240 int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem)
875fd932de880a Dmitry Osipenko 2022-05-27 241 {
875fd932de880a Dmitry Osipenko 2022-05-27 242 dma_resv_lock(shmem->base.resv, NULL);
875fd932de880a Dmitry Osipenko 2022-05-27 243
875fd932de880a Dmitry Osipenko 2022-05-27 244 if (shmem->madv < 0)
875fd932de880a Dmitry Osipenko 2022-05-27 245 return -ENOMEM;
875fd932de880a Dmitry Osipenko 2022-05-27 246
875fd932de880a Dmitry Osipenko 2022-05-27 247 shmem->purge_enabled = true;
875fd932de880a Dmitry Osipenko 2022-05-27 248
875fd932de880a Dmitry Osipenko 2022-05-27 249 drm_gem_shmem_update_pages_state(shmem);
875fd932de880a Dmitry Osipenko 2022-05-27 250
875fd932de880a Dmitry Osipenko 2022-05-27 251 dma_resv_unlock(shmem->base.resv);
875fd932de880a Dmitry Osipenko 2022-05-27 252
875fd932de880a Dmitry Osipenko 2022-05-27 @253 return 0;
875fd932de880a Dmitry Osipenko 2022-05-27 254 }
875fd932de880a Dmitry Osipenko 2022-05-27 255 EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable);875fd932de880a Dmitry Osipenko 2022-05-27 256
Signature · 2 lines
0-DAY CI Kernel Test Service https://01.org/lkp
Dmitry Osipenko
maintainerRe: [kbuild] Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 53 lines · Dan Carpenter
Hi Dmitry,
url: https://github.com/intel-lab-lkp/linux/commits/Dmitry-Osipenko/Add-generic-memory-shrinker-to-VirtIO-GPU-and-Panfrost-DRM-drivers/20220527-075717 base: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git cdeffe87f790dfd1baa193020411ce9a538446d7 config: i386-randconfig-m021 (https://download.01.org/0day-ci/archive/20220527/202205272155.MxDiru43-lkp(a)intel.com/config ) compiler: gcc-11 (Debian 11.3.0-1) 11.3.0If you fix the issue, kindly add following tag where applicable
Reported-by kernel test robot <[email protected]>Reported-by Dan Carpenter <[email protected]>New smatch warnings:
drivers/gpu/drm/drm_gem_shmem_helper.c:226drm_gem_shmem_set_evictable()warn: inconsistent returns 'shmem->base.resv'.drivers/gpu/drm/drm_gem_shmem_helper.c:253drm_gem_shmem_set_purgeable()warn: inconsistent returns 'shmem->base.resv'.drivers/gpu/drm/drm_gem_shmem_helper.c:703drm_gem_shmem_fault()error: we previously assumed 'shmem->pages' could be null (see line 694)vim +226
drivers/gpu/drm/drm_gem_shmem_helper.c875fd932de880a Dmitry Osipenko 2022-05-27 215 int drm_gem_shmem_set_evictable(struct drm_gem_shmem_object *shmem) 875fd932de880a Dmitry Osipenko 2022-05-27 216 { 875fd932de880a Dmitry Osipenko 2022-05-27 217 dma_resv_lock(shmem->base.resv, NULL); 875fd932de880a Dmitry Osipenko 2022-05-27 218 875fd932de880a Dmitry Osipenko 2022-05-27 219 if (shmem->madv < 0) 875fd932de880a Dmitry Osipenko 2022-05-27 220 return -ENOMEM;Unlock before returning?
875fd932de880a Dmitry Osipenko 2022-05-27 221 875fd932de880a Dmitry Osipenko 2022-05-27 222 shmem->eviction_enabled = true; 875fd932de880a Dmitry Osipenko 2022-05-27 223 875fd932de880a Dmitry Osipenko 2022-05-27 224 dma_resv_unlock(shmem->base.resv); 875fd932de880a Dmitry Osipenko 2022-05-27 225 875fd932de880a Dmitry Osipenko 2022-05-27 @226 return 0; 875fd932de880a Dmitry Osipenko 2022-05-27 227 } 875fd932de880a Dmitry Osipenko 2022-05-27 228 EXPORT_SYMBOL_GPL(drm_gem_shmem_set_evictable); 875fd932de880a Dmitry Osipenko 2022-05-27 229 875fd932de880a Dmitry Osipenko 2022-05-27 230 /** 875fd932de880a Dmitry Osipenko 2022-05-27 231 * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker 875fd932de880a Dmitry Osipenko 2022-05-27 232 * @shmem: shmem GEM object 875fd932de880a Dmitry Osipenko 2022-05-27 233 * 875fd932de880a Dmitry Osipenko 2022-05-27 234 * Tell memory shrinker that this GEM can be purged. Initially purging is 875fd932de880a Dmitry Osipenko 2022-05-27 235 * disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. 875fd932de880a Dmitry Osipenko 2022-05-27 236 * 875fd932de880a Dmitry Osipenko 2022-05-27 237 * Returns: 875fd932de880a Dmitry Osipenko 2022-05-27 238 * 0 on success or a negative error code on failure. 875fd932de880a Dmitry Osipenko 2022-05-27 239 */ 875fd932de880a Dmitry Osipenko 2022-05-27 240 int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem) 875fd932de880a Dmitry Osipenko 2022-05-27 241 { 875fd932de880a Dmitry Osipenko 2022-05-27 242 dma_resv_lock(shmem->base.resv, NULL); 875fd932de880a Dmitry Osipenko 2022-05-27 243 875fd932de880a Dmitry Osipenko 2022-05-27 244 if (shmem->madv < 0) 875fd932de880a Dmitry Osipenko 2022-05-27 245 return -ENOMEM;
Quoted 1 line
Same.
Indeed, thanks!
Signature · 2 lines
Best regards, Dmitry
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
In reply to Dmitry OsipenkoQuoted 12 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.
Quoted 5 lines
This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>
So I guess I get a price for being blind since forever, because this thing existed since at least 2013. I just stumbled over llist_lru.[hc], a purpose built list helper for shrinkers. I think we should try to adopt that so that our gpu shrinkers look more like shrinkers for everything else.
Apologies for this, since I fear this might cause a bit of churn. Hopefully it's all contained to the list manipulation code in shmem helpers, I don't think this should leak any further. -Daniel
Quoted 948 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +497
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --2.35.3
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 18 lines · Daniel Vetter
Quoted 15 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>So I guess I get a price for being blind since forever, because this thing existed since at least 2013. I just stumbled over llist_lru.[hc], a purpose built list helper for shrinkers. I think we should try to adopt that so that our gpu shrinkers look more like shrinkers for everything else.
followup from a bit of irc discussion w/ danvet about list_lru:
* It seems to be missing a way to bail out of iteration before nr_to_scan is hit.. which is going to be inconvenient if you want to allow active bos on the LRU but bail scanning once you encounter the first one.
* Not sure if the numa node awareness is super useful for GEM bos
First issue is perhaps not too hard to fix. But maybe a better idea is a drm_gem_lru helper type thing which is more tailored to GEM buffers?
BR,
-R
Quoted 959 lines
Apologies for this, since I fear this might cause a bit of churn. Hopefully it's all contained to the list manipulation code in shmem helpers, I don't think this should leak any further. -Daniel
Quoted 946 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +497
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --Quoted 2 lines
2.35.3
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 36 lines · Rob Clark
Quoted 18 lines · Daniel Vetter
Quoted 15 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>So I guess I get a price for being blind since forever, because this thing existed since at least 2013. I just stumbled over llist_lru.[hc], a purpose built list helper for shrinkers. I think we should try to adopt that so that our gpu shrinkers look more like shrinkers for everything else.
followup from a bit of irc discussion w/ danvet about
list_lru:* It seems to be missing a way to bail out of iteration before
nr_to_scanis hit.. which is going to be inconvenient if you want to allow active bos on the LRU but bail scanning once you encounter the first one.
Quoted 6 lines
* Not sure if the numa node awareness is super useful for GEM bos
First issue is perhaps not too hard to fix. But maybe a better idea is a
drm_gem_lruhelper type thing which is more tailored to GEM buffers?
Yeah I guess reusing list_lru isn't that good idea. So just open-coding it for now, and then drm_gem_bo_lru or so if we need to share it separately from shmem helpers with other drivers. Maybe will be needed for ttm or so.
-Daniel
Quoted 963 lines
BR,
-RQuoted 959 lines
Apologies for this, since I fear this might cause a bit of churn. Hopefully it's all contained to the list manipulation code in shmem helpers, I don't think this should leak any further. -Daniel
Quoted 946 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +497
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --Quoted 2 lines
2.35.3
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
In reply to Daniel VetterAm 05.06.22 um 18:47 schrieb Daniel Vetter:
Quoted 16 lines
Quoted 14 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>So I guess I get a price for being blind since forever, because this thing existed since at least 2013. I just stumbled over llist_lru.[hc], a purpose built list helper for shrinkers. I think we should try to adopt that so that our gpu shrinkers look more like shrinkers for everything else.
What the heck are you talking about?
I can't find any llist_lru.[hc] in the linux kernel sources.
Christian.
Quoted 955 lines
Apologies for this, since I fear this might cause a bit of churn. Hopefully it's all contained to the list manipulation code in shmem helpers, I don't think this should leak any further. -Daniel
Quoted 948 lines
--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +497
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --2.35.3
Dmitry Osipenko
maintainerRe: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 25 lines · Christian König
Am 05.06.22 um 18:47 schrieb Daniel Vetter:
Quoted 16 lines
Quoted 14 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>So I guess I get a price for being blind since forever, because this thing existed since at least 2013. I just stumbled over llist_lru.[hc], a purpose built list helper for shrinkers. I think we should try to adopt that so that our gpu shrinkers look more like shrinkers for everything else.
Quoted 3 lines
What the heck are you talking about?
I can't find any llist_lru.[hc] in the linux kernel sources.
I think Daniel meant this:
https://elixir.bootlin.com/linux/v5.19-rc1/source/include/linux/list_lru.h
https://elixir.bootlin.com/linux/v5.19-rc1/source/mm/list_lru.c
Signature · 2 lines
Best regards, Dmitry
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
In reply to Dmitry OsipenkoQuoted 518 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +268
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock))
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 954
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock);
As I mentioned on other thread, count_objects, being approximate but lockless and fast is the important thing. Otherwise when you start hitting the shrinker on many threads, you end up serializing them all, even if you have no pages to return to the system at that point.
Quoted 95 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +95
-0+ +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1059
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + }
I'm not sure this is a good idea to serialize on object_name_lock. Purgeable buffers should never be shared (imported or exported). So at best you are avoiding evicting and immediately swapping back in, in a rare case, at the cost of serializing multiple threads trying to reclaim pages in parallel.
BR,
-R
Quoted 332 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +114
-0+ +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan) + freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --2.35.3
Dmitry Osipenko
maintainerRe: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
On 6/19/22 20:53, Rob Clark wrote:
...
Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +16
-0+static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ;
Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 162
Quoted 3 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +3
-0+ } + +mutex_unlock (&gem_shrinker->lock);As I mentioned on other thread,
count_objects, being approximate but lockless and fast is the important thing. Otherwise when you start hitting the shrinker on many threads, you end up serializing them all, even if you have no pages to return to the system at that point.
Daniel's point for dropping the lockless variant was that we're already in trouble if we're hitting shrinker too often and extra optimizations won't bring much benefits to us.
Alright, I'll add back the lockless variant (or will use yours drm_gem_lru) in the next revision. The code difference is very small after all.
...
Quoted 11 lines
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1059
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-0+/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + }I'm not sure this is a good idea to serialize on
object_name_lock. Purgeable buffers should never be shared (imported or exported). So at best you are avoiding evicting and immediately swapping back in, in a rare case, at the cost of serializing multiple threads trying to reclaim pages in parallel.
The object_name_lock shouldn't cause contention in practice. But objects are also pinned on attachment, hence maybe this lock is indeed unnecessary.. I'll re-check it.
Signature · 2 lines
Best regards, Dmitry
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 28 lines · Dmitry Osipenko
On 6/19/22 20:53, Rob Clark wrote:
...Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +16
-0+static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ;Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 162
Quoted 3 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +3
-0+ } + +mutex_unlock (&gem_shrinker->lock);As I mentioned on other thread,
count_objects, being approximate but lockless and fast is the important thing. Otherwise when you start hitting the shrinker on many threads, you end up serializing them all, even if you have no pages to return to the system at that point.
Quoted 3 lines
Daniel's point for dropping the lockless variant was that we're already in trouble if we're hitting shrinker too often and extra optimizations won't bring much benefits to us.
At least with zram swap (which I highly recommend using even if you are not using a physical swap file/partition), swapin/out is actually quite fast. And if you are leaning on zram swap to fit 8GB of chrome browser on a 4GB device, the shrinker gets hit quite a lot. Lower spec (4GB RAM) chromebooks can be under constant memory pressure and can quite easily get into a situation where you are hitting the shrinker on many threads simultaneously. So it is pretty important for all shrinkers in the system (not just drm driver) to be as concurrent as possible. As long as you avoid serializing reclaim on all the threads, performance can still be quite good, but if you don't performance will fall off a cliff.
jfwiw, we are seeing pretty good results (iirc 40-70% increase in open tab counts) with the combination of eviction + multigen LRU[1] + sizing zram swap to be 2x physical RAM
[1] https://lwn.net/Articles/856931/
Quoted 17 lines
Alright, I'll add back the lockless variant (or will use yours
drm_gem_lru) in the next revision. The code difference is very small after all....
Quoted 11 lines
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1059
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-0+/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + }I'm not sure this is a good idea to serialize on
object_name_lock. Purgeable buffers should never be shared (imported or exported). So at best you are avoiding evicting and immediately swapping back in, in a rare case, at the cost of serializing multiple threads trying to reclaim pages in parallel.
Quoted 3 lines
The
object_name_lockshouldn't cause contention in practice. But objects are also pinned on attachment, hence maybe this lock is indeed unnecessary.. I'll re-check it.
I'm not worried about contention with export/import/etc, but contention between multiple threads hitting the shrinker in parallel. I guess since you are using trylock, it won't *block* the other threads hitting shrinker, but they'll just end up looping in do_shrink_slab() because they are hitting contention.
I'd have to do some experiments to see how it works out in practice, but my gut feel is that it isn't a good idea
BR,
-R
Quoted 3 lines
--
Best regards,
Dmitry
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
Quoted 73 lines · Rob Clark
Quoted 28 lines · Dmitry Osipenko
On 6/19/22 20:53, Rob Clark wrote:
...Quoted 16 lines
Quoted 16 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +16
-0+static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ;Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t · line 162
Quoted 3 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +3
-0+ } + +mutex_unlock (&gem_shrinker->lock);As I mentioned on other thread,
count_objects, being approximate but lockless and fast is the important thing. Otherwise when you start hitting the shrinker on many threads, you end up serializing them all, even if you have no pages to return to the system at that point.Quoted 3 lines
Daniel's point for dropping the lockless variant was that we're already in trouble if we're hitting shrinker too often and extra optimizations won't bring much benefits to us.
At least with zram swap (which I highly recommend using even if you are not using a physical swap file/partition), swapin/out is actually quite fast. And if you are leaning on zram swap to fit 8GB of chrome browser on a 4GB device, the shrinker gets hit quite a lot. Lower spec (4GB RAM) chromebooks can be under constant memory pressure and can quite easily get into a situation where you are hitting the shrinker on many threads simultaneously. So it is pretty important for all shrinkers in the system (not just drm driver) to be as concurrent as possible. As long as you avoid serializing reclaim on all the threads, performance can still be quite good, but if you don't performance will fall off a cliff.
jfwiw, we are seeing pretty good results (iirc 40-70% increase in open tab counts) with the combination of eviction + multigen LRU[1] + sizing zram swap to be 2x physical RAM
[1] https://lwn.net/Articles/856931/
Quoted 17 lines
Alright, I'll add back the lockless variant (or will use yours
drm_gem_lru) in the next revision. The code difference is very small after all....
Quoted 11 lines
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1059
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-0+/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + }I'm not sure this is a good idea to serialize on
object_name_lock. Purgeable buffers should never be shared (imported or exported). So at best you are avoiding evicting and immediately swapping back in, in a rare case, at the cost of serializing multiple threads trying to reclaim pages in parallel.Quoted 3 lines
The
object_name_lockshouldn't cause contention in practice. But objects are also pinned on attachment, hence maybe this lock is indeed unnecessary.. I'll re-check it.
Quoted 8 lines
I'm not worried about contention with export/import/etc, but contention between multiple threads hitting the shrinker in parallel. I guess since you are using trylock, it won't *block* the other threads hitting shrinker, but they'll just end up looping in
do_shrink_slab()because they are hitting contention.I'd have to do some experiments to see how it works out in practice, but my gut feel is that it isn't a good idea
Yeah trylock on anything else than the object lock is No Good in the shrinker. And it really shouldn't be needed, since import/export should pin stuff as needed. Which should be protected by the dma_resv object lock. If not, we need to fix that.
Picking a random drm-internal lock like this is definitely no good design. -Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
In reply to Rob ClarkQuoted 8 lines · Rob Clark
Quoted 1 line · Dmitry Osipenko
+
mutex_unlock(&gem_shrinker->lock);As I mentioned on other thread,
count_objects, being approximate but lockless and fast is the important thing. Otherwise when you start hitting the shrinker on many threads, you end up serializing them all, even if you have no pages to return to the system at that point.
Yeah agreed, seems like I was wrong here :-) Atomic counter or something would also be in link the the lru_list stuff.
It would be to record this in the kerneldoc for the shrinker structure though, to make sure this is all understood.
Quoted 11 lines
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1059
Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +5
-0+/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + }I'm not sure this is a good idea to serialize on
object_name_lock. Purgeable buffers should never be shared (imported or exported). So at best you are avoiding evicting and immediately swapping back in, in a rare case, at the cost of serializing multiple threads trying to reclaim pages in parallel.
Yeah this sounds really bad. Plus this is a per-device lock, and doing those with trylock means the shrinker will fail to find shrinkable memory way too often. We need to engineer this out somehow.
-Daniel
Signature · 3 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
In reply to Dmitry Osipenko()
Quoted 667 lines · Dmitry Osipenko
Introduce a common DRM SHMEM shrinker framework that allows to reduce code duplication among DRM drivers by replacing theirs custom shrinker implementations with the generic shrinker.
In order to start using DRM SHMEM shrinker drivers should:
1. Implement new
evict()shmem object callback.
2. Register shrinker usingdrm_gem_shmem_shrinker_register(drm_device).
3. Usedrm_gem_shmem_set_purgeable(shmem)and alike API functions to activate shrinking of shmem GEMs.This patch is based on a ideas borrowed from Rob's Clark MSM shrinker, Thomas' Zimmermann variant of SHMEM shrinker and Intel's i915 shrinker.
Signed-off-by Daniel Almeida <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/drm_gem_shmem_helper.c | 540 ++++++++++++++++-- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 9 +- drivers/gpu/drm/virtio/virtgpu_drv.h | 3 + include/drm/drm_device.h | 4 + include/drm/drm_gem_shmem_helper.h | 87 ++- 5 files changed, 594 insertions(+), 49 deletions(-)drivers/gpu/drm/drm_gem_shmem_helper.c +417
-43@@ -126,6 +126,42 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t }EXPORT_SYMBOL_GPL (drm_gem_shmem_create); +static bool drm_gem_shmem_is_evictable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >=0 ) && shmem->evict && + shmem->eviction_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && shmem->sgt && !shmem->evicted; +} + +static void +drm_gem_shmem_update_pages_state (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker; + +dma_resv_assert_held (shmem->base.resv); + +if (!gem_shrinker || obj->import_attach) +return ; + +mutex_lock (&gem_shrinker->lock); + +if (drm_gem_shmem_is_evictable (shmem) || +drm_gem_shmem_is_purgeable (shmem)) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evictable); +else if (shmem->madv <0 ) +list_del_init (&shmem->madv_list); +else if (shmem->evicted) +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_evicted); +else if (!shmem->pages) +list_del_init (&shmem->madv_list); +else +list_move_tail (&shmem->madv_list, &gem_shrinker->lru_pinned); + +mutex_unlock (&gem_shrinker->lock); +} +/** * drm_gem_shmem_free - Free resources associated with a shmem GEM object * @shmem: shmem GEM object to free @@ -142,6 +178,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }else {dma_resv_lock (shmem->base.resv,NULL ); +/* take out shmem GEM object from the memory shrinker */ +drm_gem_shmem_madvise (shmem,-1 ); +WARN_ON (shmem->vmap_use_count);if (shmem->sgt) {@@ -150,7 +189,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)sg_free_table (shmem->sgt);kfree (shmem->sgt); } -if (shmem->pages) +if (shmem->pages_use_count)drm_gem_shmem_put_pages (shmem);WARN_ON (shmem->pages_use_count);@@ -163,18 +202,82 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem) }EXPORT_SYMBOL_GPL (drm_gem_shmem_free); -static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_set_evictable() - Make GEM evictable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be evicted. Initially eviction is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->eviction_enabled = true; + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_evictable); + +/** +* drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker +* @shmem: shmem GEM object +* +* Tell memory shrinker that this GEM can be purged. Initially purging is +* disabled for all GEMs. If GEM was purged, then -ENOMEM is returned. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem) +{ +dma_resv_lock (shmem->base.resv,NULL ); + +if (shmem->madv <0 ) +return -ENOMEM ; + + shmem->purge_enabled = true; + +drm_gem_shmem_update_pages_state (shmem); + +dma_resv_unlock (shmem->base.resv); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_set_purgeable); + +static int +drm_gem_shmem_acquire_pages (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base;struct page **pages; -if (shmem->pages_use_count++ >0 ) +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) { +WARN_ON (shmem->pages); +return -ENOMEM ; + } + +if (shmem->pages) { +WARN_ON (!shmem->evicted);return 0 ; + } + +if (WARN_ON (!shmem->pages_use_count)) +return -EINVAL ; pages =drm_gem_get_pages (obj);if (IS_ERR (pages)) {DRM_DEBUG_KMS ("Failed to get pages (%ld)\n" ,PTR_ERR (pages)); - shmem->pages_use_count =0 ;return PTR_ERR (pages); }@@ -193,6 +296,58 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)return 0 ; } +static int drm_gem_shmem_get_pages (struct drm_gem_shmem_object *shmem) +{ +int err; + +dma_resv_assert_held (shmem->base.resv); + +if (shmem->madv <0 ) +return -ENOMEM ; + +if (shmem->pages_use_count++ >0 ) { + err =drm_gem_shmem_swap_in (shmem); +if (err) + goto err_zero_use; + +return 0 ; + } + + err =drm_gem_shmem_acquire_pages (shmem); +if (err) + goto err_zero_use; + +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; + +err_zero_use: + shmem->pages_use_count =0 ; + +return err; +} + +static void +drm_gem_shmem_release_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +if (!shmem->pages) { +WARN_ON (!shmem->evicted && shmem->madv >=0 ); +return ; + } + +#ifdef CONFIG_X86 +if (shmem->map_wc) +set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); +#endif + +drm_gem_put_pages (obj, shmem->pages, + shmem->pages_mark_dirty_on_put, + shmem->pages_mark_accessed_on_put); + shmem->pages =NULL ; +} +/* * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object * @shmem: shmem GEM object @@ -201,8 +356,6 @@ static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)*/ void drm_gem_shmem_put_pages (struct drm_gem_shmem_object *shmem) { -struct drm_gem_object *obj = &shmem->base; -dma_resv_assert_held (shmem->base.resv);if (WARN_ON_ONCE (!shmem->pages_use_count))@@ -211,15 +364,9 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)if (--shmem->pages_use_count >0 )return ; -#ifdef CONFIG_X86 -if (shmem->map_wc) -set_pages_array_wb (shmem->pages, obj->size >>PAGE_SHIFT ); -#endif +drm_gem_shmem_release_pages (shmem); -drm_gem_put_pages (obj, shmem->pages, - shmem->pages_mark_dirty_on_put, - shmem->pages_mark_accessed_on_put); - shmem->pages =NULL ; +drm_gem_shmem_update_pages_state (shmem); }EXPORT_SYMBOL (drm_gem_shmem_put_pages);@@ -235,11 +382,17 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);*/ int drm_gem_shmem_pin (struct drm_gem_shmem_object *shmem) { +int ret; +dma_resv_assert_held (shmem->base.resv);WARN_ON (shmem->base.import_attach); -return drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_get_pages (shmem); +if (!ret) + shmem->pages_pin_count++; + +return ret; }EXPORT_SYMBOL (drm_gem_shmem_pin);@@ -257,6 +410,8 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)WARN_ON (shmem->base.import_attach);drm_gem_shmem_put_pages (shmem); + + shmem->pages_pin_count--; }EXPORT_SYMBOL (drm_gem_shmem_unpin);@@ -299,7 +454,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,return 0 ; } - ret =drm_gem_shmem_get_pages (shmem); + ret =drm_gem_shmem_pin (shmem);if (ret) goto err_zero_use;@@ -322,7 +477,7 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, err_put_pages:if (!obj->import_attach) -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); err_zero_use: shmem->vmap_use_count =0 ;@@ -359,7 +514,7 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,return ;vunmap (shmem->vaddr); -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_unpin (shmem); } shmem->vaddr =NULL ;@@ -403,41 +558,77 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv) madv = shmem->madv; +drm_gem_shmem_update_pages_state (shmem); +return (madv >=0 ); }EXPORT_SYMBOL (drm_gem_shmem_madvise); -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +/** +* drm_gem_shmem_swap_in() - Moves shmem GEM back to memory and enables +* hardware access to the memory. +* @shmem: shmem GEM object +* +* This function moves shmem GEM back to memory if it was previously evicted +* by the memory shrinker. The GEM is ready to use on success. +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem) {struct drm_gem_object *obj = &shmem->base; -struct drm_device *dev = obj->dev; +struct sg_table *sgt; +int err;dma_resv_assert_held (shmem->base.resv); -WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); +if (shmem->evicted) { + err =drm_gem_shmem_acquire_pages (shmem); +if (err) +return err; + + sgt =drm_gem_shmem_get_sg_table (shmem); +if (IS_ERR (sgt)) +return PTR_ERR (sgt); + + err =dma_map_sgtable (obj->dev->dev, sgt, +DMA_BIDIRECTIONAL ,0 ); +if (err) { +sg_free_table (sgt); +kfree (sgt); +return err; + } -dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); -sg_free_table (shmem->sgt); -kfree (shmem->sgt); - shmem->sgt =NULL ; + shmem->sgt = sgt; + shmem->evicted = false; -drm_gem_shmem_put_pages (shmem); +drm_gem_shmem_update_pages_state (shmem); + } - shmem->madv =-1 ; +if (!shmem->pages) +return -ENOMEM ; -drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); -drm_gem_free_mmap_offset (obj); +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_swap_in); -/* Our goal here is to return as much of the memory as -* is possible back to the system as we are called from OOM. -* To do this we must instruct the shmfs to drop all of its -* backing pages, *now*. -*/ -shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); +static void drm_gem_shmem_unpin_pages (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; +struct drm_device *dev = obj->dev; -invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); +if (shmem->evicted) +return ; + +dma_unmap_sgtable (dev->dev, shmem->sgt,DMA_BIDIRECTIONAL ,0 ); +drm_gem_shmem_release_pages (shmem); +drm_vma_node_unmap (&obj->vma_node, dev->anon_inode->i_mapping); + +sg_free_table (shmem->sgt); +kfree (shmem->sgt); + shmem->sgt =NULL ; } -EXPORT_SYMBOL (drm_gem_shmem_purge);/** * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object @@ -488,22 +679,33 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)vm_fault_t ret;struct page *page;pgoff_t page_offset; +bool pages_unpinned; +int err;/* We don't use vmf->pgoff since that has the fake offset */ page_offset = (vmf->address - vma->vm_start) >>PAGE_SHIFT ;dma_resv_lock (shmem->base.resv,NULL ); -if (page_offset >= num_pages || -WARN_ON_ONCE (!shmem->pages) || - shmem->madv <0 ) { +/* Sanity-check that we have the pages pointer when it should present */ + pages_unpinned = (shmem->evicted || shmem->madv <0 || !shmem->pages_use_count); +WARN_ON_ONCE (!shmem->pages ^ pages_unpinned); + +if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) { ret =VM_FAULT_SIGBUS ; }else { + err =drm_gem_shmem_swap_in (shmem); +if (err) { + ret =VM_FAULT_OOM ; + goto unlock; + } + page = shmem->pages[page_offset]; ret =vmf_insert_pfn (vma, vmf->address,page_to_pfn (page)); } +unlock:dma_resv_unlock (shmem->base.resv);return ret;@@ -513,13 +715,15 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma) {struct drm_gem_object *obj = vma->vm_private_data;struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -int ret;WARN_ON (shmem->base.import_attach);dma_resv_lock (shmem->base.resv,NULL ); - ret =drm_gem_shmem_get_pages (shmem); -WARN_ON_ONCE (ret !=0 ); + +if (drm_gem_shmem_get_pages (shmem)) + shmem->pages_use_count++; + +drm_gem_shmem_update_pages_state (shmem);dma_resv_unlock (shmem->base.resv);drm_gem_vm_open (vma);@@ -583,6 +787,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_mmap);void drm_gem_shmem_print_info (const struct drm_gem_shmem_object *shmem,struct drm_printer *p,unsigned int indent) { +drm_printf_indent (p, indent,"eviction_enabled=%d\n" , shmem->eviction_enabled); +drm_printf_indent (p, indent,"purge_enabled=%d\n" , shmem->purge_enabled);drm_printf_indent (p, indent,"pages_use_count=%u\n" , shmem->pages_use_count);if (shmem->base.import_attach)@@ -592,7 +798,9 @@ void drm_gem_shmem_print_info(const struct drm_gem_shmem_object *shmem,drm_printf_indent (p, indent,"vmap_use_count=%u\n" , shmem->vmap_use_count); +drm_printf_indent (p, indent,"evicted=%d\n" , shmem->evicted);drm_printf_indent (p, indent,"vaddr=%p\n" , shmem->vaddr); +drm_printf_indent (p, indent,"madv=%d\n" , shmem->madv); }EXPORT_SYMBOL (drm_gem_shmem_print_info);@@ -667,6 +875,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem) shmem->sgt = sgt; +drm_gem_shmem_update_pages_state (shmem); +dma_resv_unlock (shmem->base.resv);return sgt;@@ -717,6 +927,250 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, }EXPORT_SYMBOL_GPL (drm_gem_shmem_prime_import_sg_table); +static struct drm_gem_shmem_shrinker * +to_drm_shrinker (struct shrinker *shrinker) +{ + return container_of(shrinker, struct drm_gem_shmem_shrinker, base); +} + +static unsigned long +drm_gem_shmem_shrinker_count_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +unsigned long count =0 ; + +if (!mutex_trylock (&gem_shrinker->lock)) +return 0 ; + +list_for_each_entry (shmem, &gem_shrinker->lru_evictable, madv_list) { + count += shmem->base.size; + +if (count >=SHRINK_EMPTY ) +break ; + } + +mutex_unlock (&gem_shrinker->lock); + +if (count >=SHRINK_EMPTY ) +return SHRINK_EMPTY -1 ; + +return count ?:SHRINK_EMPTY ; +} + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +WARN_ON (!drm_gem_shmem_is_evictable (shmem)); +WARN_ON (shmem->evicted); + +drm_gem_shmem_unpin_pages (shmem); + + shmem->evicted = true; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_evict); + +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem) +{ +struct drm_gem_object *obj = &shmem->base; + +WARN_ON (!drm_gem_shmem_is_purgeable (shmem)); + +drm_gem_shmem_unpin_pages (shmem); +drm_gem_free_mmap_offset (obj); + +/* Our goal here is to return as much of the memory as +* is possible back to the system as we are called from OOM. +* To do this we must instruct the shmfs to drop all of its +* backing pages, *now*. +*/ +shmem_truncate_range (file_inode (obj->filp),0 , (loff_t)-1 ); + +invalidate_mapping_pages (file_inode (obj->filp)->i_mapping,0 , (loff_t)-1 ); + + shmem->madv =-1 ; + shmem->evicted = false; +drm_gem_shmem_update_pages_state (shmem); + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_purge); + +static unsigned long +drm_gem_shmem_shrinker_run_objects_scan (struct shrinker *shrinker, +unsigned long nr_to_scan, +bool *lock_contention, +bool evict) +{ +struct drm_gem_shmem_shrinker *gem_shrinker =to_drm_shrinker (shrinker); +struct drm_gem_shmem_object *shmem; +struct list_head still_in_list; +struct drm_gem_object *obj; +unsigned long freed =0 ; +size_t page_count; +int err; + +INIT_LIST_HEAD (&still_in_list); + +mutex_lock (&gem_shrinker->lock); + +while (freed < nr_to_scan) { + shmem =list_first_entry_or_null (&gem_shrinker->lru_evictable, +typeof (*shmem), madv_list); +if (!shmem) +break ; + + obj = &shmem->base; + page_count = obj->size >>PAGE_SHIFT ; +list_move_tail (&shmem->madv_list, &still_in_list); + +if (evict) { +if (!drm_gem_shmem_is_evictable (shmem) || +get_nr_swap_pages () < page_count) +continue ; + }else { +if (!drm_gem_shmem_is_purgeable (shmem)) +continue ; + } + +/* +* If it's in the process of being freed, gem_object->free() +* may be blocked on lock waiting to remove it. So just +* skip it. +*/ +if (!kref_get_unless_zero (&obj->refcount)) +continue ; + +mutex_unlock (&gem_shrinker->lock); + +/* prevent racing with job-submission code paths */ +if (!dma_resv_trylock (obj->resv)) { + *lock_contention |= true; + goto shrinker_lock; + } + +/* prevent racing with the dma-buf importing/exporting */ +if (!mutex_trylock (&gem_shrinker->dev->object_name_lock)) { + *lock_contention |= true; + goto resv_unlock; + } + +/* check whether h/w uses this object */ +if (!dma_resv_test_signaled (obj->resv,DMA_RESV_USAGE_WRITE )) + goto object_name_unlock; + +/* re-check whether eviction status hasn't changed */ +if (!drm_gem_shmem_is_evictable (shmem) && + !drm_gem_shmem_is_purgeable (shmem)) + goto object_name_unlock; + + err = shmem->evict (shmem); +if (!err) + freed += obj->size >>PAGE_SHIFT ; + +object_name_unlock: +mutex_unlock (&gem_shrinker->dev->object_name_lock); +resv_unlock: +dma_resv_unlock (obj->resv); +shrinker_lock: +drm_gem_object_put (&shmem->base); +mutex_lock (&gem_shrinker->lock); + } + +list_splice_tail (&still_in_list, &gem_shrinker->lru_evictable); + +mutex_unlock (&gem_shrinker->lock); + +return freed; +} + +static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{
drivers/gpu/drm/drm_gem_shmem_helper.c drm_gem_shmem_prime_import_sg_table(struct drm_device *dev, · line 1103
Quoted 10 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +10
-0+unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */ +if (freed < nr_to_scan)
oh, one other small note, both in scan_objects() and count_objects(), you should check that get_nr_swap_pages()>0 before counting evictable/willneed objects. (And you probably want to keep separate LRUs for dontneed vs willneed to accomplish that.) At least for CrOS, inside the VM there is no swap enabled (but instead we rely on zram swap in the host.. plus vm-balloon to balance memory pressure between host and guest)
BR,
-R
Quoted 288 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +70
-0+ freed +=drm_gem_shmem_shrinker_run_objects_scan (shrinker, + nr_to_scan - freed, + &lock_contention, + true); + +return (!freed && !lock_contention) ?SHRINK_STOP : freed; +} + +/** +* drm_gem_shmem_shrinker_register() - Register shmem shrinker +* @dev: DRM device +* +* Returns: +* 0 on success or a negative error code on failure. +*/ +int drm_gem_shmem_shrinker_register (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker; +int err; + +if (WARN_ON (dev->shmem_shrinker)) +return -EBUSY ; + + gem_shrinker =kzalloc (sizeof (*gem_shrinker),GFP_KERNEL ); +if (!gem_shrinker) +return -ENOMEM ; + + gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects; + gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects; + gem_shrinker->base.seeks =DEFAULT_SEEKS ; + gem_shrinker->dev = dev; + +INIT_LIST_HEAD (&gem_shrinker->lru_evictable); +INIT_LIST_HEAD (&gem_shrinker->lru_evicted); +INIT_LIST_HEAD (&gem_shrinker->lru_pinned); +mutex_init (&gem_shrinker->lock); + + dev->shmem_shrinker = gem_shrinker; + + err =register_shrinker (&gem_shrinker->base); +if (err) { + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); +return err; + } + +return 0 ; +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_register); + +/** +* drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker +* @dev: DRM device +*/ +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev) +{ +struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker; + +if (gem_shrinker) { +unregister_shrinker (&gem_shrinker->base); +WARN_ON (!list_empty (&gem_shrinker->lru_evictable)); +WARN_ON (!list_empty (&gem_shrinker->lru_evicted)); +WARN_ON (!list_empty (&gem_shrinker->lru_pinned)); +mutex_destroy (&gem_shrinker->lock); + dev->shmem_shrinker =NULL ; +kfree (gem_shrinker); + } +} +EXPORT_SYMBOL_GPL (drm_gem_shmem_shrinker_unregister); +MODULE_DESCRIPTION ("DRM SHMEM memory-management helpers" );MODULE_IMPORT_NS (DMA_BUF );MODULE_LICENSE ("GPL v2" );drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c +8
-1@@ -15,6 +15,13 @@#include "panfrost_gem.h" #include "panfrost_mmu.h" +static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) +{ +return (shmem->madv >0 ) && + !shmem->pages_pin_count && shmem->sgt && + !shmem->base.dma_buf && !shmem->base.import_attach; +} +static unsigned long panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) {@@ -27,7 +34,7 @@ panfrost_gem_shrinker_count(struct shrinker *shrinker, struct shrink_control *screturn 0 ;list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (drm_gem_shmem_is_purgeable (shmem)) +if (panfrost_gem_shmem_is_purgeable (shmem)) count += shmem->base.size >>PAGE_SHIFT ; }drivers/gpu/drm/virtio/virtgpu_drv.h +3
-0@@ -89,6 +89,7 @@ struct virtio_gpu_object {uint32_t hw_res_handle;bool dumb;bool created; +bool detached;bool host3d_blob, guest_blob;uint32_t blob_mem, blob_flags;@@ -453,6 +454,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,bool virtio_gpu_is_shmem (struct virtio_gpu_object *bo); +int virtio_gpu_reattach_shmem_object (struct virtio_gpu_object *bo); +int virtio_gpu_resource_id_get (struct virtio_gpu_device *vgdev,uint32_t *resid);/* virtgpu_prime.c */ include/drm/drm_device.h +4
-0@@ -16,6 +16,7 @@ struct drm_vblank_crtc;struct drm_vma_offset_manager ;struct drm_vram_mm ;struct drm_fb_helper ; +struct drm_gem_shmem_shrinker ;struct inode ;@@ -277,6 +278,9 @@ struct drm_device {/** @vram_mm: VRAM MM memory manager */ struct drm_vram_mm *vram_mm; +/** @shmem_shrinker: SHMEM GEM memory shrinker */ +struct drm_gem_shmem_shrinker *shmem_shrinker; +/** * @switch_power_state: * include/drm/drm_gem_shmem_helper.h +82
-6@@ -6,6 +6,7 @@#include <linux/fs.h> #include <linux/mm.h> #include <linux/mutex.h> +#include <linux/shrinker.h> #include <drm/drm_file.h> #include <drm/drm_gem.h> @@ -15,6 +16,7 @@struct dma_buf_attachment ;struct drm_mode_create_dumb ;struct drm_printer ; +struct drm_device ;struct sg_table ;/** @@ -39,12 +41,21 @@ struct drm_gem_shmem_object {*/ unsigned int pages_use_count; +/** +* @pages_pin_count: +* +* Reference count on the pinned pages table. +* The pages can be evicted by memory shrinker +* when the count reaches zero. +*/ +unsigned int pages_pin_count; +/** * @madv: State for madvise * * 0 is active/inuse. +* 1 is not-needed/can-be-purged * A negative value is the object is purged. -* Positive values are driver specific and not used by the helpers. */ int madv;@@ -91,6 +102,39 @@ struct drm_gem_shmem_object {* @map_wc: map object write-combined (instead of using shmem defaults). */ bool map_wc; + +/** +* @eviction_enabled: +* +* The shmem pages can be evicted only if @eviction_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool eviction_enabled; + +/** +* @purge_enabled: +* +* The shmem pages can be purged only if @purge_enabled is set to true. +* Used internally by memory shrinker. +*/ +bool purge_enabled; + +/** +* @evicted: True if shmem pages are evicted by the memory shrinker. +* Used internally by memory shrinker. +*/ +bool evicted; + +/** +* @evict: +* +* Invoked by shmem shrinker before evicting shmem GEM from memory. +* GEM's DMA reservation is kept locked by the shrinker. This is +* optional callback that should be specified by drivers. +* +* Returns 0 on success, or -errno on error. +*/ +int (*evict)(struct drm_gem_shmem_object *shmem); };#define to_drm_gem_shmem_obj (obj) \@@ -110,14 +154,21 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_structint drm_gem_shmem_madvise (struct drm_gem_shmem_object *shmem,int madv); +int drm_gem_shmem_set_purgeable (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_set_evictable (struct drm_gem_shmem_object *shmem); +static inline bool drm_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) { -return (shmem->madv >0 ) && - !shmem->vmap_use_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; +return (shmem->madv >0 ) && shmem->evict && + shmem->purge_enabled && shmem->pages_use_count && + !shmem->pages_pin_count && !shmem->base.dma_buf && + !shmem->base.import_attach && (shmem->sgt || shmem->evicted); } -void drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_swap_in (struct drm_gem_shmem_object *shmem); + +int drm_gem_shmem_evict (struct drm_gem_shmem_object *shmem); +int drm_gem_shmem_purge (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_sg_table (struct drm_gem_shmem_object *shmem);struct sg_table *drm_gem_shmem_get_pages_sgt (struct drm_gem_shmem_object *shmem);@@ -260,6 +311,32 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct vreturn drm_gem_shmem_mmap (shmem, vma); } +/** +* struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs +*/ +struct drm_gem_shmem_shrinker { +/** @base: Shrinker for purging shmem GEM objects */ +struct shrinker base; + +/** @lock: Protects @lru_* */ +struct mutex lock; + +/** @lru_pinned: List of pinned shmem GEM objects */ +struct list_head lru_pinned; + +/** @lru_evictable: List of shmem GEM objects to be evicted */ +struct list_head lru_evictable; + +/** @lru_evicted: List of evicted shmem GEM objects */ +struct list_head lru_evicted; + +/** @dev: DRM device that uses this shrinker */ +struct drm_device *dev; +}; + +int drm_gem_shmem_shrinker_register (struct drm_device *dev); +void drm_gem_shmem_shrinker_unregister (struct drm_device *dev); +/* * Driver ops */ --2.35.3
Re: [PATCH v6 17/22] drm/shmem-helper: Add generic memory shrinker
20.06.2022 18:37, Rob Clark пишет:
Quoted 13 lines
Quoted 13 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +13
-0+static unsigned long +drm_gem_shmem_shrinker_scan_objects (struct shrinker *shrinker, +struct shrink_control *sc) +{ +unsigned long nr_to_scan = sc->nr_to_scan; +bool lock_contention = false; +unsigned long freed; + +/* purge as many objects as we can */ + freed =drm_gem_shmem_shrinker_run_objects_scan (shrinker, nr_to_scan, + &lock_contention, false); + +/* evict as many objects as we can */
Quoted 8 lines
Quoted 1 line
+ if (freed <
nr_to_scan)oh, one other small note, both in
scan_objects()andcount_objects(), you should check that get_nr_swap_pages()>0 before counting evictable/willneed objects. (And you probably want to keep separate LRUs for dontneed vs willneed to accomplish that.) At least for CrOS, inside the VM there is no swap enabled (but instead we rely on zram swap in the host.. plus vm-balloon to balance memory pressure between host and guest)
Noted, thank you!
Dmitry Osipenko
maintainerPATCH v6 18/22 drm/gem: Add drm_gem_pin_unlocked()
In reply to Dmitry OsipenkoAdd unlocked variants of drm_gem_un/pin() functions and make them public. These new helpers will take care of GEM dma-reservation locking for DRM drivers.
We are going to add memory shrinking support to the VirtIO-GPU driver that will need to pin framebuffers explicitly to prevent eviction of the actively used buffers by the shrinker. VirtIO-GPU driver will use these new generic helpers to pin shmem framebuffers.
--- drivers/gpu/drm/drm_gem.c | 29 +++++++++++++++++++++++++++++ include/drm/drm_gem.h | 3 +++ 2 files changed, 32 insertions(+)
drivers/gpu/drm/drm_gem.c +29 -0
@@ -1179,6 +1179,35 @@ void drm_gem_unpin(struct drm_gem_object *obj) obj->funcs->unpin (obj); } +int drm_gem_pin_unlocked (struct drm_gem_object *obj) +{ +int ret; + +if (!obj->funcs->pin) +return 0 ; + + ret =dma_resv_lock_interruptible (obj->resv,NULL ); +if (ret) +return ret; + + ret = obj->funcs->pin (obj); +dma_resv_unlock (obj->resv); + +return ret; +} +EXPORT_SYMBOL (drm_gem_pin_unlocked); + +void drm_gem_unpin_unlocked (struct drm_gem_object *obj) +{ +if (!obj->funcs->unpin) +return ; + +dma_resv_lock (obj->resv,NULL ); + obj->funcs->unpin (obj); +dma_resv_unlock (obj->resv); +} +EXPORT_SYMBOL (drm_gem_unpin_unlocked); +int drm_gem_vmap (struct drm_gem_object *obj,struct iosys_map *map) {int ret;
include/drm/drm_gem.h +3 -0
@@ -413,4 +413,7 @@ int drm_gem_dumb_map_offset(struct drm_file *file, struct drm_device *dev,int drm_gem_vmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map);void drm_gem_vunmap_unlocked (struct drm_gem_object *obj,struct iosys_map *map); +int drm_gem_pin_unlocked (struct drm_gem_object *obj); +void drm_gem_unpin_unlocked (struct drm_gem_object *obj); +#endif /* __DRM_GEM_H__ */
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 19/22 drm/virtio: Support memory shrinking
In reply to Dmitry OsipenkoSupport generic drm-shmem memory shrinker and add new madvise IOCTL to the VirtIO-GPU driver. BO cache manager of Mesa driver will mark BOs as "don't need" using the new IOCTL to let shrinker purge the marked BOs on OOM, the shrinker will also evict unpurgeable shmem BOs from memory if guest supports SWAP file or partition. Altogether this allows to prevent OOM kills of guest applications that use VirGL by lowering memory pressure.
--- drivers/gpu/drm/virtio/virtgpu_drv.h | 15 ++- drivers/gpu/drm/virtio/virtgpu_gem.c | 55 ++++++++++ drivers/gpu/drm/virtio/virtgpu_ioctl.c | 37 +++++++ drivers/gpu/drm/virtio/virtgpu_kms.c | 9 ++ drivers/gpu/drm/virtio/virtgpu_object.c | 138 +++++++++++++++++++----- drivers/gpu/drm/virtio/virtgpu_plane.c | 22 +++- drivers/gpu/drm/virtio/virtgpu_vq.c | 40 +++++++ include/uapi/drm/virtgpu_drm.h | 14 +++ 8 files changed, 300 insertions(+), 30 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +14 -1
@@ -275,7 +275,7 @@ struct virtio_gpu_fpriv { };/* virtgpu_ioctl.c */ -#define DRM_VIRTIO_NUM_IOCTLS 12 +#define DRM_VIRTIO_NUM_IOCTLS 13extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ];void virtio_gpu_create_context (struct drm_device *dev,struct drm_file *file);
@@ -311,6 +311,10 @@ void virtio_gpu_array_put_free(struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_delayed (struct virtio_gpu_device *vgdev,struct virtio_gpu_object_array *objs);void virtio_gpu_array_put_free_work (struct work_struct *work); +int virtio_gpu_array_prepare (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object_array *objs); +int virtio_gpu_gem_host_mem_release (struct virtio_gpu_object *bo); +int virtio_gpu_gem_madvise (struct virtio_gpu_object *obj,int madv);/* virtgpu_vq.c */ int virtio_gpu_alloc_vbufs (struct virtio_gpu_device *vgdev);
@@ -322,6 +326,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev, struct virtio_gpu_fence *fence);void virtio_gpu_cmd_unref_resource (struct virtio_gpu_device *vgdev,struct virtio_gpu_object *bo); +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo); void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev, uint64_t offset, uint32_t width, uint32_t height,
@@ -342,6 +348,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,struct virtio_gpu_object *obj, struct virtio_gpu_mem_entry *ents, unsigned int nents); +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence);int virtio_gpu_attach_status_page (struct virtio_gpu_device *vgdev);int virtio_gpu_detach_status_page (struct virtio_gpu_device *vgdev); void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -486,4 +495,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,struct sg_table *sgt, enum dma_data_direction dir); +/* virtgpu_gem_shrinker.c */ +int virtio_gpu_gem_shrinker_init (struct virtio_gpu_device *vgdev); +void virtio_gpu_gem_shrinker_fini (struct virtio_gpu_device *vgdev); +#endif
drivers/gpu/drm/virtio/virtgpu_gem.c +55 -0
58 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_ioctl.c +37 -0
@@ -217,6 +217,10 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data, ret =virtio_gpu_array_lock_resv (buflist);if (ret) goto out_memdup; + + ret =virtio_gpu_array_prepare (vgdev, buflist); +if (ret) + goto out_unresv; } out_fence =virtio_gpu_fence_alloc (vgdev, fence_ctx, ring_idx);
@@ -423,6 +427,10 @@ static int virtio_gpu_transfer_from_host_ioctl(struct drm_device *dev,if (ret !=0 ) goto err_put_free; + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 );if (!fence) { ret = -ENOMEM ;
@@ -482,6 +490,10 @@ static int virtio_gpu_transfer_to_host_ioctl(struct drm_device *dev, void *data,if (ret !=0 ) goto err_put_free; + ret =virtio_gpu_array_prepare (vgdev, objs); +if (ret) + goto err_unlock; + ret = -ENOMEM ; fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 );
@@ -836,6 +848,28 @@ static int virtio_gpu_context_init_ioctl(struct drm_device *dev,return ret; } +static int virtio_gpu_madvise_ioctl (struct drm_device *dev, +void *data, +struct drm_file *file) +{ +struct drm_virtgpu_madvise *args = data; +struct virtio_gpu_object *bo; +struct drm_gem_object *obj; + +if (args->madv >VIRTGPU_MADV_DONTNEED ) +return -EOPNOTSUPP ; + + obj =drm_gem_object_lookup (file, args->bo_handle); +if (!obj) +return -ENOENT ; + + bo =gem_to_virtio_gpu_obj (obj); + args->retained =virtio_gpu_gem_madvise (bo, args->madv); +drm_gem_object_put (obj); + +return 0 ; +} +struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS ] = {DRM_IOCTL_DEF_DRV (VIRTGPU_MAP , virtio_gpu_map_ioctl,DRM_RENDER_ALLOW ),
@@ -875,4 +909,7 @@ struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS] = {DRM_IOCTL_DEF_DRV (VIRTGPU_CONTEXT_INIT , virtio_gpu_context_init_ioctl,DRM_RENDER_ALLOW ), + +DRM_IOCTL_DEF_DRV (VIRTGPU_MADVISE , virtio_gpu_madvise_ioctl, + DRM_RENDER_ALLOW), };
drivers/gpu/drm/virtio/virtgpu_kms.c +9 -0
@@ -238,6 +238,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) goto err_scanouts; } + ret =drm_gem_shmem_shrinker_register (dev); +if (ret) { +DRM_ERROR ("shrinker init failed\n" ); + goto err_modeset; + } +virtio_device_ready (vgdev->vdev);if (num_capsets)
@@ -250,6 +256,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev) 5 * HZ);return 0 ; +err_modeset: +virtio_gpu_modeset_fini (vgdev); err_scanouts:virtio_gpu_free_vbufs (vgdev); err_vbufs:
@@ -289,6 +297,7 @@ void virtio_gpu_release(struct drm_device *dev)if (!vgdev)return ; +drm_gem_shmem_shrinker_unregister (dev);virtio_gpu_modeset_fini (vgdev);virtio_gpu_free_vbufs (vgdev);virtio_gpu_cleanup_cap_cache (vgdev);
drivers/gpu/drm/virtio/virtgpu_object.c +111 -27
172 lines not on this page · raw message
drivers/gpu/drm/virtio/virtgpu_plane.c +20 -2
@@ -246,20 +246,32 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,struct virtio_gpu_device *vgdev = dev->dev_private;struct virtio_gpu_framebuffer *vgfb;struct virtio_gpu_object *bo; +int err;if (!new_state->fb)return 0 ; vgfb =to_virtio_gpu_framebuffer (new_state->fb); bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); -if (!bo || (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)) + +if (virtio_gpu_is_shmem (bo)) { + err =drm_gem_pin_unlocked (&bo->base.base); +if (err) +return err; + } + +if (plane->type ==DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)return 0 ;if (bo->dumb && (plane->state->fb != new_state->fb)) { vgfb->fence =virtio_gpu_fence_alloc (vgdev, vgdev->fence_drv.context,0 ); -if (!vgfb->fence) +if (!vgfb->fence) { +if (virtio_gpu_is_shmem (bo)) +drm_gem_unpin_unlocked (&bo->base.base); +return -ENOMEM ; + } }return 0 ;
@@ -269,15 +281,21 @@ static void virtio_gpu_plane_cleanup_fb(struct drm_plane *plane, struct drm_plane_state *state) {struct virtio_gpu_framebuffer *vgfb; +struct virtio_gpu_object *bo;if (!state->fb)return ; vgfb =to_virtio_gpu_framebuffer (state->fb); + bo =gem_to_virtio_gpu_obj (vgfb->base.obj[0 ]); +if (vgfb->fence) {dma_fence_put (&vgfb->fence->f); vgfb->fence =NULL ; } + +if (virtio_gpu_is_shmem (bo)) +drm_gem_unpin_unlocked (&bo->base.base); } static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
drivers/gpu/drm/virtio/virtgpu_vq.c +40 -0
@@ -536,6 +536,21 @@ void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,virtio_gpu_cleanup_object (bo); } +int virtio_gpu_cmd_release_resource (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *bo) +{ +struct virtio_gpu_resource_unref *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_UNREF ); + cmd_p->resource_id =cpu_to_le32 (bo->hw_res_handle); + +return virtio_gpu_queue_ctrl_buffer (vgdev, vbuf); +} + void virtio_gpu_cmd_set_scanout(struct virtio_gpu_device *vgdev, uint32_t scanout_id, uint32_t resource_id, uint32_t width, uint32_t height,
@@ -636,6 +651,23 @@ virtio_gpu_cmd_resource_attach_backing(struct virtio_gpu_device *vgdev,virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); } +static void +virtio_gpu_cmd_resource_detach_backing (struct virtio_gpu_device *vgdev, +u32 resource_id, +struct virtio_gpu_fence *fence) +{ +struct virtio_gpu_resource_attach_backing *cmd_p; +struct virtio_gpu_vbuffer *vbuf; + + cmd_p =virtio_gpu_alloc_cmd (vgdev, &vbuf,sizeof (*cmd_p)); +memset (cmd_p,0 ,sizeof (*cmd_p)); + + cmd_p->hdr.type =cpu_to_le32 (VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING ); + cmd_p->resource_id =cpu_to_le32 (resource_id); + +virtio_gpu_queue_fenced_ctrl_buffer (vgdev, vbuf, fence); +} +static void virtio_gpu_cmd_get_display_info_cb (struct virtio_gpu_device *vgdev,struct virtio_gpu_vbuffer *vbuf) {
@@ -1099,6 +1131,14 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev, ents, nents, NULL); } +void virtio_gpu_object_detach (struct virtio_gpu_device *vgdev, +struct virtio_gpu_object *obj, +struct virtio_gpu_fence *fence) +{ +virtio_gpu_cmd_resource_detach_backing (vgdev, obj->hw_res_handle, + fence); +} +void virtio_gpu_cursor_ping (struct virtio_gpu_device *vgdev,struct virtio_gpu_output *output) {
include/uapi/drm/virtgpu_drm.h +14 -0
@@ -48,6 +48,7 @@ extern "C" {#define DRM_VIRTGPU_GET_CAPS 0x09#define DRM_VIRTGPU_RESOURCE_CREATE_BLOB 0x0a#define DRM_VIRTGPU_CONTEXT_INIT 0x0b +#define DRM_VIRTGPU_MADVISE 0x0c#define VIRTGPU_EXECBUF_FENCE_FD_IN 0x01#define VIRTGPU_EXECBUF_FENCE_FD_OUT 0x02
@@ -196,6 +197,15 @@ struct drm_virtgpu_context_init {__u64 ctx_set_params; }; +#define VIRTGPU_MADV_WILLNEED 0 +#define VIRTGPU_MADV_DONTNEED 1 +struct drm_virtgpu_madvise { +__u32 bo_handle; +__u32 retained;/* out, non-zero if BO can be used */ +__u32 madv; +__u32 pad; +}; +/* * Event code that's given when VIRTGPU_CONTEXT_PARAM_POLL_RINGS_MASK is in * effect. The event size is sizeof(drm_event), since there is no additional
@@ -246,6 +256,10 @@ struct drm_virtgpu_context_init {DRM_IOWR (DRM_COMMAND_BASE +DRM_VIRTGPU_CONTEXT_INIT , \ struct drm_virtgpu_context_init) +#define DRM_IOCTL_VIRTGPU_MADVISE \ + DRM_IOWR(DRM_COMMAND_BASE + DRM_VIRTGPU_MADVISE, \ + struct drm_virtgpu_madvise) +#if defined(__cplusplus) }#endif
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 20/22 drm/virtio: Use dev_is_pci()
In reply to Dmitry OsipenkoUse common dev_is_pci() helper to replace the strcmp("pci") used by driver.
--- drivers/gpu/drm/virtio/virtgpu_drv.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +1 -1
@@ -87,7 +87,7 @@ static int virtio_gpu_probe(struct virtio_device *vdev)return PTR_ERR (dev); vdev->priv = dev; -if (!strcmp (vdev->dev.parent->bus->name,"pci" )) { +if (dev_is_pci (vdev->dev.parent)) { ret =virtio_gpu_pci_quirk (dev);if (ret) goto err_free;
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 21/22 drm/virtio: Return proper error codes instead of -1
In reply to Dmitry OsipenkoDon't return -1 in error cases, return proper error code. The returned error codes propagate to error messages and to userspace and it's always good to have a meaningful error number for debugging purposes.
--- drivers/gpu/drm/virtio/virtgpu_vq.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/virtio/virtgpu_vq.c +3 -3
@@ -320,7 +320,7 @@ static int virtio_gpu_queue_ctrl_sgs(struct virtio_gpu_device *vgdev,if (fence && vbuf->objs)virtio_gpu_array_unlock_resv (vbuf->objs);free_vbuf (vgdev, vbuf); -return -1 ; +return -ENODEV ; }if (vgdev->has_indirect)
@@ -384,7 +384,7 @@ static int virtio_gpu_queue_fenced_ctrl_buffer(struct virtio_gpu_device *vgdev,if (!sgt) {if (fence && vbuf->objs)virtio_gpu_array_unlock_resv (vbuf->objs); -return -1 ; +return -ENOMEM ; } elemcnt += sg_ents;
@@ -750,7 +750,7 @@ static int virtio_get_edid_block(void *data, u8 *buf,size_t start = block *EDID_LENGTH ;if (start + len >le32_to_cpu (resp->size)) -return -1 ; +return -EINVAL ;memcpy (buf, resp->edid + start, len);return 0 ; }
Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerPATCH v6 22/22 drm/panfrost: Switch to generic memory shrinker
In reply to Dmitry OsipenkoReplace Panfrost's memory shrinker with a generic drm-shmem memory shrinker.
--- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 19 +-- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +++-- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 -- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 ------------------ drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++- 7 files changed, 42 insertions(+), 171 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
drivers/gpu/drm/panfrost/Makefile +0 -1
@@ -5,7 +5,6 @@ panfrost-y := \ panfrost_device.o \ panfrost_devfreq.o \ panfrost_gem.o \ - panfrost_gem_shrinker.o \ panfrost_gpu.o \ panfrost_job.o \ panfrost_mmu.o \
drivers/gpu/drm/panfrost/panfrost_device.h +0 -4
@@ -115,10 +115,6 @@ struct panfrost_device {atomic_t pending; } reset; -struct mutex shrinker_lock; -struct list_head shrinker_list; -struct shrinker shrinker; -struct panfrost_devfreq pfdevfreq; };
drivers/gpu/drm/panfrost/panfrost_drv.c +2 -17
@@ -160,7 +160,6 @@ panfrost_lookup_bos(struct drm_device *dev,break ; } -atomic_inc (&bo->gpu_usecount); job->mappings[i] = mapping; }
@@ -392,7 +391,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, {struct panfrost_file_priv *priv = file_priv->driver_priv;struct drm_panfrost_madvise *args = data; -struct panfrost_device *pfdev = dev->dev_private;struct drm_gem_object *gem_obj;struct panfrost_gem_object *bo;int ret =0 ;
@@ -409,7 +407,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (ret) goto out_put_object; -mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {struct panfrost_gem_mapping *first;
@@ -435,17 +432,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, args->retained =drm_gem_shmem_madvise (&bo->base, args->madv); -if (args->retained) { -if (args->madv ==PANFROST_MADV_DONTNEED ) -list_move_tail (&bo->base.madv_list, - &pfdev->shrinker_list); -else if (args->madv ==PANFROST_MADV_WILLNEED ) -list_del_init (&bo->base.madv_list); - } - out_unlock_mappings:mutex_unlock (&bo->mappings.lock); -mutex_unlock (&pfdev->shrinker_lock);dma_resv_unlock (bo->base.base.resv); out_put_object:drm_gem_object_put (gem_obj);
@@ -577,9 +565,6 @@ static int panfrost_probe(struct platform_device *pdev) ddev->dev_private = pfdev; pfdev->ddev = ddev; -mutex_init (&pfdev->shrinker_lock); -INIT_LIST_HEAD (&pfdev->shrinker_list); - err =panfrost_device_init (pfdev);if (err) {if (err != -EPROBE_DEFER )
@@ -601,7 +586,7 @@ static int panfrost_probe(struct platform_device *pdev)if (err <0 ) goto err_out1; -panfrost_gem_shrinker_init (ddev); +drm_gem_shmem_shrinker_register (ddev);return 0 ;
@@ -619,8 +604,8 @@ static int panfrost_remove(struct platform_device *pdev)struct panfrost_device *pfdev =platform_get_drvdata (pdev);struct drm_device *ddev = pfdev->ddev; +drm_gem_shmem_shrinker_unregister (ddev);drm_dev_unregister (ddev); -panfrost_gem_shrinker_cleanup (ddev);pm_runtime_get_sync (pfdev->dev);pm_runtime_disable (pfdev->dev);
drivers/gpu/drm/panfrost/panfrost_gem.c +23 -10
@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)struct panfrost_gem_object *bo =to_panfrost_bo (obj);struct panfrost_device *pfdev = obj->dev->dev_private; -/* -* Make sure the BO is no longer inserted in the shrinker list before -* taking care of the destruction itself. If we don't do that we have a -* race condition between this function and what's done in -* panfrost_gem_shrinker_scan(). -*/ -mutex_lock (&pfdev->shrinker_lock); -list_del_init (&bo->base.madv_list); -mutex_unlock (&pfdev->shrinker_lock); -/* * If we still have mappings attached to the BO, there's a problem in * our refcounting.
@@ -209,6 +199,25 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .vm_ops = &drm_gem_shmem_vm_ops, }; +static int panfrost_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +struct panfrost_gem_object *bo =to_panfrost_bo (&shmem->base); + +if (!drm_gem_shmem_is_purgeable (shmem)) +return -EOPNOTSUPP ; + +if (!mutex_trylock (&bo->mappings.lock)) +return -EBUSY ; + +panfrost_gem_teardown_mappings_locked (bo); + +drm_gem_shmem_purge (shmem); + +mutex_unlock (&bo->mappings.lock); + +return 0 ; +} +/** * panfrost_gem_create_object - Implementation of driver->gem_create_object. * @dev: DRM device
@@ -230,6 +239,7 @@ struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_tmutex_init (&obj->mappings.lock); obj->base.base.funcs = &panfrost_gem_funcs; obj->base.map_wc = !pfdev->coherent; + obj->base.evict = panfrost_shmem_evict;return &obj->base.base; }
@@ -266,6 +276,9 @@ panfrost_gem_create_with_handle(struct drm_file *file_priv,if (ret)return ERR_PTR (ret); +if (!bo->is_heap) +drm_gem_shmem_set_purgeable (shmem); +return bo; }
drivers/gpu/drm/panfrost/panfrost_gem.h +0 -9
@@ -30,12 +30,6 @@ struct panfrost_gem_object {struct mutex lock; } mappings; -/* -* Count the number of jobs referencing this BO so we don't let the -* shrinker reclaim this object prematurely. -*/ -atomic_t gpu_usecount; - bool noexec :1; bool is_heap :1; };
@@ -84,7 +78,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); -void panfrost_gem_shrinker_cleanup (struct drm_device *dev); -#endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0 -129
129 lines not on this page · raw message
drivers/gpu/drm/panfrost/panfrost_job.c +17 -1
@@ -271,6 +271,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,dma_resv_add_fence (bos[i]->resv, fence,DMA_RESV_USAGE_WRITE ); } +static int panfrost_objects_prepare (struct drm_gem_object **bos,int bo_count) +{ +struct panfrost_gem_object *bo; +int ret =0 ; + +while (!ret && bo_count--) { + bo =to_panfrost_bo (bos[bo_count]); + ret = bo->base.madv ? -ENOMEM :0 ; + } + +return ret; +} +int panfrost_job_push (struct panfrost_job *job) {struct panfrost_device *pfdev = job->pfdev;
@@ -282,6 +295,10 @@ int panfrost_job_push(struct panfrost_job *job)if (ret)return ret; + ret =panfrost_objects_prepare (job->bos, job->bo_count); +if (ret) + goto unlock; +mutex_lock (&pfdev->sched_lock);drm_sched_job_arm (&job->base);
@@ -323,7 +340,6 @@ static void panfrost_job_cleanup(struct kref *ref)if (!job->mappings[i])break ; -atomic_dec (&job->mappings[i]->obj->gpu_usecount);panfrost_gem_mapping_put (job->mappings[i]); }kvfree (job->mappings);
Signature · 1 line
2.35.3
Re: [PATCH v6 22/22] drm/panfrost: Switch to generic memory shrinker
Quoted 389 lines · Dmitry Osipenko
Replace Panfrost's memory shrinker with a generic drm-shmem memory shrinker.
Tested-by Steven Price <[email protected]>Signed-off-by Dmitry Osipenko <[email protected]>--- drivers/gpu/drm/panfrost/Makefile | 1 - drivers/gpu/drm/panfrost/panfrost_device.h | 4 - drivers/gpu/drm/panfrost/panfrost_drv.c | 19 +-- drivers/gpu/drm/panfrost/panfrost_gem.c | 33 +++-- drivers/gpu/drm/panfrost/panfrost_gem.h | 9 -- .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 129 ------------------ drivers/gpu/drm/panfrost/panfrost_job.c | 18 ++- 7 files changed, 42 insertions(+), 171 deletions(-) delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.cdrivers/gpu/drm/panfrost/Makefile +0
-1@@ -5,7 +5,6 @@ panfrost-y := \ panfrost_device.o \ panfrost_devfreq.o \ panfrost_gem.o \ - panfrost_gem_shrinker.o \ panfrost_gpu.o \ panfrost_job.o \ panfrost_mmu.o \drivers/gpu/drm/panfrost/panfrost_device.h +0
-4@@ -115,10 +115,6 @@ struct panfrost_device {atomic_t pending; } reset; -struct mutex shrinker_lock; -struct list_head shrinker_list; -struct shrinker shrinker; -struct panfrost_devfreq pfdevfreq; };drivers/gpu/drm/panfrost/panfrost_drv.c +2
-17@@ -160,7 +160,6 @@ panfrost_lookup_bos(struct drm_device *dev,break ; } -atomic_inc (&bo->gpu_usecount); job->mappings[i] = mapping; }@@ -392,7 +391,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, {struct panfrost_file_priv *priv = file_priv->driver_priv;struct drm_panfrost_madvise *args = data; -struct panfrost_device *pfdev = dev->dev_private;struct drm_gem_object *gem_obj;struct panfrost_gem_object *bo;int ret =0 ;@@ -409,7 +407,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,if (ret) goto out_put_object; -mutex_lock (&pfdev->shrinker_lock);mutex_lock (&bo->mappings.lock);if (args->madv ==PANFROST_MADV_DONTNEED ) {struct panfrost_gem_mapping *first;@@ -435,17 +432,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data, args->retained =drm_gem_shmem_madvise (&bo->base, args->madv); -if (args->retained) { -if (args->madv ==PANFROST_MADV_DONTNEED ) -list_move_tail (&bo->base.madv_list, - &pfdev->shrinker_list); -else if (args->madv ==PANFROST_MADV_WILLNEED ) -list_del_init (&bo->base.madv_list); - } - out_unlock_mappings:mutex_unlock (&bo->mappings.lock); -mutex_unlock (&pfdev->shrinker_lock);dma_resv_unlock (bo->base.base.resv); out_put_object:drm_gem_object_put (gem_obj);@@ -577,9 +565,6 @@ static int panfrost_probe(struct platform_device *pdev) ddev->dev_private = pfdev; pfdev->ddev = ddev; -mutex_init (&pfdev->shrinker_lock); -INIT_LIST_HEAD (&pfdev->shrinker_list); - err =panfrost_device_init (pfdev);if (err) {if (err != -EPROBE_DEFER )@@ -601,7 +586,7 @@ static int panfrost_probe(struct platform_device *pdev)if (err <0 ) goto err_out1; -panfrost_gem_shrinker_init (ddev); +drm_gem_shmem_shrinker_register (ddev);return 0 ;@@ -619,8 +604,8 @@ static int panfrost_remove(struct platform_device *pdev)struct panfrost_device *pfdev =platform_get_drvdata (pdev);struct drm_device *ddev = pfdev->ddev; +drm_gem_shmem_shrinker_unregister (ddev);drm_dev_unregister (ddev); -panfrost_gem_shrinker_cleanup (ddev);pm_runtime_get_sync (pfdev->dev);pm_runtime_disable (pfdev->dev);drivers/gpu/drm/panfrost/panfrost_gem.c +23
-10@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)struct panfrost_gem_object *bo =to_panfrost_bo (obj);struct panfrost_device *pfdev = obj->dev->dev_private; -/* -* Make sure the BO is no longer inserted in the shrinker list before -* taking care of the destruction itself. If we don't do that we have a -* race condition between this function and what's done in -* panfrost_gem_shrinker_scan(). -*/ -mutex_lock (&pfdev->shrinker_lock); -list_del_init (&bo->base.madv_list); -mutex_unlock (&pfdev->shrinker_lock); -/* * If we still have mappings attached to the BO, there's a problem in * our refcounting. @@ -209,6 +199,25 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = { .vm_ops = &drm_gem_shmem_vm_ops, }; +static int panfrost_shmem_evict (struct drm_gem_shmem_object *shmem) +{ +struct panfrost_gem_object *bo =to_panfrost_bo (&shmem->base); + +if (!drm_gem_shmem_is_purgeable (shmem)) +return -EOPNOTSUPP ; + +if (!mutex_trylock (&bo->mappings.lock)) +return -EBUSY ; + +panfrost_gem_teardown_mappings_locked (bo); + +drm_gem_shmem_purge (shmem); + +mutex_unlock (&bo->mappings.lock); + +return 0 ; +} +/** * panfrost_gem_create_object - Implementation of driver->gem_create_object. * @dev: DRM device @@ -230,6 +239,7 @@ struct drm_gem_object *panfrost_gem_create_object(struct drm_device *dev, size_tmutex_init (&obj->mappings.lock); obj->base.base.funcs = &panfrost_gem_funcs; obj->base.map_wc = !pfdev->coherent; + obj->base.evict = panfrost_shmem_evict;return &obj->base.base; }@@ -266,6 +276,9 @@ panfrost_gem_create_with_handle(struct drm_file *file_priv,if (ret)return ERR_PTR (ret); +if (!bo->is_heap) +drm_gem_shmem_set_purgeable (shmem); +return bo; }drivers/gpu/drm/panfrost/panfrost_gem.h +0
-9@@ -30,12 +30,6 @@ struct panfrost_gem_object {struct mutex lock; } mappings; -/* -* Count the number of jobs referencing this BO so we don't let the -* shrinker reclaim this object prematurely. -*/ -atomic_t gpu_usecount; - bool noexec :1; bool is_heap :1; };@@ -84,7 +78,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,void panfrost_gem_mapping_put (struct panfrost_gem_mapping *mapping);void panfrost_gem_teardown_mappings_locked (struct panfrost_gem_object *bo); -void panfrost_gem_shrinker_init (struct drm_device *dev); -void panfrost_gem_shrinker_cleanup (struct drm_device *dev); -#endif /* __PANFROST_GEM_H__ */ drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0
-129@@ -1,129 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* Copyright (C) 2019 Arm Ltd. -* -* Based on msm_gem_freedreno.c: -* Copyright (C) 2016 Red Hat -* Author: Rob Clark <[email protected]> -*/ - -#include <linux/list.h> - -#include <drm/drm_device.h> -#include <drm/drm_gem_shmem_helper.h> - -#include "panfrost_device.h" -#include "panfrost_gem.h" -#include "panfrost_mmu.h" - -static bool panfrost_gem_shmem_is_purgeable (struct drm_gem_shmem_object *shmem) -{ -return (shmem->madv >0 ) && - !shmem->pages_pin_count && shmem->sgt && - !shmem->base.dma_buf && !shmem->base.import_attach; -} - -static unsigned long -panfrost_gem_shrinker_count (struct shrinker *shrinker,struct shrink_control *sc) -{ -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); -struct drm_gem_shmem_object *shmem; -unsigned long count =0 ; - -if (!mutex_trylock (&pfdev->shrinker_lock)) -return 0 ; - -list_for_each_entry (shmem, &pfdev->shrinker_list, madv_list) { -if (panfrost_gem_shmem_is_purgeable (shmem)) - count += shmem->base.size >>PAGE_SHIFT ; - } - -mutex_unlock (&pfdev->shrinker_lock); - -return count; -} - -static bool panfrost_gem_purge (struct drm_gem_object *obj) -{ -struct drm_gem_shmem_object *shmem =to_drm_gem_shmem_obj (obj); -struct panfrost_gem_object *bo =to_panfrost_bo (obj); -bool ret = false; - -if (atomic_read (&bo->gpu_usecount)) -return false; - -if (!mutex_trylock (&bo->mappings.lock)) -return false; - -if (!dma_resv_trylock (shmem->base.resv)) - goto unlock_mappings; - -panfrost_gem_teardown_mappings_locked (bo); -drm_gem_shmem_purge (&bo->base); - ret = true; - -dma_resv_unlock (shmem->base.resv); - -unlock_mappings: -mutex_unlock (&bo->mappings.lock); -return ret; -} - -static unsigned long -panfrost_gem_shrinker_scan (struct shrinker *shrinker,struct shrink_control *sc) -{ -struct panfrost_device *pfdev = - container_of(shrinker, struct panfrost_device, shrinker); -struct drm_gem_shmem_object *shmem, *tmp; -unsigned long freed =0 ; - -if (!mutex_trylock (&pfdev->shrinker_lock)) -return SHRINK_STOP ; - -list_for_each_entry_safe (shmem, tmp, &pfdev->shrinker_list, madv_list) { -if (freed >= sc->nr_to_scan) -break ; -if (drm_gem_shmem_is_purgeable (shmem) && -panfrost_gem_purge (&shmem->base)) { - freed += shmem->base.size >>PAGE_SHIFT ; -list_del_init (&shmem->madv_list); - } - } - -mutex_unlock (&pfdev->shrinker_lock); - -if (freed >0 ) -pr_info_ratelimited ("Purging %lu bytes\n" , freed <<PAGE_SHIFT ); - -return freed; -} - -/** -* panfrost_gem_shrinker_init - Initialize panfrost shrinker -* @dev: DRM device -* -* This function registers and sets up the panfrost shrinker. -*/ -void panfrost_gem_shrinker_init (struct drm_device *dev) -{ -struct panfrost_device *pfdev = dev->dev_private; - pfdev->shrinker.count_objects = panfrost_gem_shrinker_count; - pfdev->shrinker.scan_objects = panfrost_gem_shrinker_scan; - pfdev->shrinker.seeks =DEFAULT_SEEKS ; -WARN_ON (register_shrinker (&pfdev->shrinker)); -} - -/** -* panfrost_gem_shrinker_cleanup - Clean up panfrost shrinker -* @dev: DRM device -* -* This function unregisters the panfrost shrinker. -*/ -void panfrost_gem_shrinker_cleanup (struct drm_device *dev) -{ -struct panfrost_device *pfdev = dev->dev_private; - -if (pfdev->shrinker.nr_deferred) { -unregister_shrinker (&pfdev->shrinker); - } -}drivers/gpu/drm/panfrost/panfrost_job.c +17
-1@@ -271,6 +271,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,dma_resv_add_fence (bos[i]->resv, fence,DMA_RESV_USAGE_WRITE ); } +static int panfrost_objects_prepare (struct drm_gem_object **bos,int bo_count) +{ +struct panfrost_gem_object *bo; +int ret =0 ; + +while (!ret && bo_count--) { + bo =to_panfrost_bo (bos[bo_count]); + ret = bo->base.madv ? -ENOMEM :0 ; + } + +return ret; +} +int panfrost_job_push (struct panfrost_job *job) {struct panfrost_device *pfdev = job->pfdev;@@ -282,6 +295,10 @@ int panfrost_job_push(struct panfrost_job *job)if (ret)return ret; + ret =panfrost_objects_prepare (job->bos, job->bo_count); +if (ret) + goto unlock; +mutex_lock (&pfdev->sched_lock);drm_sched_job_arm (&job->base);@@ -323,7 +340,6 @@ static void panfrost_job_cleanup(struct kref *ref)if (!job->mappings[i])break ; -atomic_dec (&job->mappings[i]->obj->gpu_usecount);panfrost_gem_mapping_put (job->mappings[i]); }kvfree (job->mappings);Signature · 1 line
2.35.3
Dmitry Osipenko
maintainerQuoted 189 lines · Dmitry Osipenko
Hello,
This patchset introduces memory shrinker for the VirtIO-GPU DRM driver and adds memory purging and eviction support to VirtIO-GPU driver.
The new dma-buf locking convention is introduced here as well.
During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL, it will also evict idling BOs to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.
The Panfrost driver is switched to use generic memory shrinker.
This patchset includes improvements and fixes for various things that I found while was working on the shrinker.
The Mesa and IGT patches will be kept on hold until this kernel series will be approved and merged.
This patchset was tested using Qemu and crosvm, including both cases of IOMMU off/on.
Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise IGT: https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvisehttps://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise
Changelog:
v6: - Added new VirtIO-related fix patch that previously was sent separately
and didn't get much attention:drm/gem: Properly annotate WW context on
drm_gem_lock_reservations()error- Added new patch that fixes mapping of imported dma-bufs for Tegra DRM and other affected drivers. It's also handy to have it for switching to the new dma-buf locking convention scheme:
drm/gem: Move mapping of imported dma-bufs to
drm_gem_mmap_obj()- Added new patch that fixes shrinker list corruption for stable Panfrost driver:
drm/panfrost: Fix shrinker list corruption by madvise IOCTL
- Added new minor patch-fix for drm-shmem:
drm/shmem-helper: Add missing vunmap on error
- Added fixes tag to the "Put mapping ..." patch like was suggested by Steven Price.
- Added new VirtIO-GPU driver improvement patch:
drm/virtio: Return proper error codes instead of -1
- Reworked shrinker patches like was suggested by Daniel Vetter:
- Introduced the new locking convention for dma-bufs. Tested on VirtIO-GPU, Panfrost, Lima, Tegra and Intel selftests.
- Dropped separate
purge()callback. Now singleevict()does everything.- Dropped
swap_in()callback from drm-shmem objects. DRM drivers now could and should restore only the required mappings.- Dropped dynamic counting of evictable pages. This simplifies code in exchange to *potentially* burning more CPU time on OOM.
v5: - Added new for-stable patch "drm/panfrost: Put mapping instead of
shmem obj onpanfrost_mmu_map_fault_addr()error" that corrects GEM's refcounting in case of error.- The drm_gem_shmem_v[un]map() now takes a separate
vmap_lockfor imported GEMs to avoid recursive locking of DMA reservations. This addresses v4 comment from Thomas Zimmermann about the potential deadlocking of vmapping.- Added ack from Thomas Zimmermann to "drm/shmem-helper: Correct doc-comment of
drm_gem_shmem_get_sg_table()" patch.- Dropped explicit shmem states from the generic shrinker patch as was requested by Thomas Zimmermann.
- Improved variable names and comments of the generic shrinker code.
- Extended
drm_gem_shmem_print_info()with the shrinker-state info in the "drm/virtio: Support memory shrinking" patch.- Moved evict()/swap_in()/purge() callbacks from
drm_gem_object_funcstodrm_gem_shmem_objectin the generic shrinker patch, for more consistency.- Corrected bisectability of the patches that was broken in v4 by accident.
- The
virtio_gpu_plane_prepare_fb()now usesdrm_gem_shmem_pin()instead ofdrm_gem_shmem_set_unpurgeable_and_unevictable()and does it only for shmem BOs in the "drm/virtio: Support memory shrinking" patch.- Made more functions private to
drm_gem_shmem_helper.cas was requested by Thomas Zimmermann. This minimizes number of the public shmem helpers.v4: - Corrected minor W=1 warnings reported by kernel test robot for v3.
- Renamed DRM_GEM_SHMEM_PAGES_STATE_ACTIVE/INACTIVE to PINNED/UNPINNED, for more clarity.
v3: - Hardened shrinker's
count()with usage ofREAD_ONCE()since we don't
use atomic type for counting and technically compiler is free to re-fetch counter's variable.- "Correct
drm_gem_shmem_get_sg_table()error handling" now usesPTR_ERR_OR_ZERO(), fixing typo that was made in v2.- Removed obsoleted shrinker from the Panfrost driver, which I missed to do in v2 by accident and Alyssa Rosenzweig managed to notice it.
- CCed stable kernels in all patches that make fixes, even the minor ones, like was suggested by Emil Velikov and added his r-b to the patches.
- Added t-b from Steven Price to the Panfrost's shrinker patch.
- Corrected doc-comment of drm_gem_shmem_object.madv, like was suggested by Steven Price. Comment now says that madv=1 means "object is purged" instead of saying that value is unused.
- Added more doc-comments to the new shmem shrinker API.
- The "Improve DMA API usage for shmem BOs" patch got more improvements by removing the obsoleted
drm_dev_set_unique()quirk and its comment.- Added patch that makes Virtio-GPU driver to use common
dev_is_pci()helper, which was suggested by Robin Murphy.- Added new "drm/shmem-helper: Take GEM reservation lock instead of
drm_gem_shmemlocks" patch, which was suggested by Daniel Vetter.- Added new "drm/virtio: Simplify error handling of
virtio_gpu_object_create()" patch.- Improved "Correct doc-comment of
drm_gem_shmem_get_sg_table()" patch, like was suggested by Daniel Vetter, by saying that function returnsERR_PTR()and not errno.-
virtio_gpu_purge_object()is fenced properly now, turned outvirtio_gpu_notify()doesn't do fencing as I was supposing before. Stress testing of memory eviction revealed that.- Added new patch that corrects
virtio_gpu_plane_cleanup_fb()to use appropriate atomic plane state.- SHMEM shrinker got eviction support.
- VirtIO-GPU driver now supports memory eviction. It's enabled for a non-blob GEMs only, i.e. for VirGL. The blobs don't support dynamic attaching/detaching of guest's memory, so it's not trivial to enable them.
- Added patch that removes obsoleted
drm_gem_shmem_purge()- Added patch that makes
drm_gem_shmem_get_pages()private.- Added patch that fixes lockup on
dma_resv_reserve_fences()error.v2: - Improved shrinker by using a more fine-grained locking to reduce
contention during scan of objects and dropped locking from the 'counting' callback by tracking count of shrinkable pages. This was suggested by Rob Clark in the comment to v1.- Factored out common shrinker code into
drm_gem_shmem_helper.cand switched Panfrost driver to use the new common memory shrinker. This was proposed by Thomas Zimmermann in his prototype series that he shared with us in the comment to v1. Note that I only compile-tested the Panfrost driver.- Shrinker now takes
object_name_lockduring scan to prevent racing with dma-buf exporting.- Shrinker now takes
vmap_lockduring scan to prevent racing with shmem vmap/unmap code.- Added "Correct doc-comment of
drm_gem_shmem_get_sg_table()" patch, which I sent out previously as a standalone change, since thedrm_gem_shmem_helper.cis now touched by this patchset anyways and it doesn't hurt to group all the patches together.
Quoted 16 lines
Dmitry Osipenko (22):
drm/gem: Properly annotate WW context ondrm_gem_lock_reservations()
error
drm/gem: Move mapping of imported dma-bufs todrm_gem_mmap_obj()drm/panfrost: Put mapping instead of shmem obj on
panfrost_mmu_map_fault_addr()error
drm/panfrost: Fix shrinker list corruption by madvise IOCTL drm/virtio: Correctdrm_gem_shmem_get_sg_table()error handling drm/virtio: Check whether transferred 2D BO is shmem drm/virtio: Unlock reservations onvirtio_gpu_object_shmem_init()
error
drm/virtio: Unlock reservations ondma_resv_reserve_fences()error drm/virtio: Use appropriate atomic state in
virtio_gpu_plane_cleanup_fb()
drm/shmem-helper: Add missing vunmap on error drm/shmem-helper: Correct doc-comment ofdrm_gem_shmem_get_sg_table()
...
Thomas, do you think it will be possible for you to take the fix patches 1-11 into the drm-fixes or would you prefer me to re-send them separately?
The VirtIO patches 12-13 also are good to go into drm-next, IMO.
I'm going to factor out the new dma-buf convention into a separate patchset, like was suggested by Christian. But it will take me some time to get the dma-buf patches ready and I also will be on a vacation soon.
At minimum nothing should hold the fixes, so will be great if they could land sooner.
Thank you!
Signature · 2 lines
Best regards, Dmitry
Quoted 7 lines · Dmitry Osipenko
Hello,
This patchset introduces memory shrinker for the VirtIO-GPU DRM driver and adds memory purging and eviction support to VirtIO-GPU driver.
The new dma-buf locking convention is introduced here as well.
Quoted 7 lines
During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL, it will also evict idling BOs to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.
The Panfrost driver is switched to use generic memory shrinker.
I think we still have some outstanding issues here - Alyssa reported some weirdness yesterday, so I just tried provoking a low-memory condition locally with this series applied and a few debug options enabled, and the results as below were... interesting.
Thanks,
Robin.
----->8-----
[ 68.295951] ======================================================
[ 68.295956] WARNING: possible circular locking dependency detected
[ 68.295963] 5.19.0-rc3+ #400 Not tainted
[ 68.295972] ------------------------------------------------------
[ 68.295977] cc1/295 is trying to acquire lock:
[ 68.295986] ffff000008d7f1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_gem_shmem_free+0x7c/0x198
[ 68.296036]
[ 68.296036] but task is already holding lock:
[ 68.296041] ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296080]
[ 68.296080] which lock already depends on the new lock.
[ 68.296080]
[ 68.296085]
[ 68.296085] the existing dependency chain (in reverse order) is:
[ 68.296090]
[ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}:
[ 68.296111] fs_reclaim_acquire+0xb8/0x150
[ 68.296130] dma_resv_lockdep+0x298/0x3fc
[ 68.296148] do_one_initcall+0xe4/0x5f8
[ 68.296163] kernel_init_freeable+0x414/0x49c
[ 68.296180] kernel_init+0x2c/0x148
[ 68.296195] ret_from_fork+0x10/0x20
[ 68.296207]
[ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}:
[ 68.296229] __lock_acquire+0x1724/0x2398
[ 68.296246] lock_acquire+0x218/0x5b0
[ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296277] ww_mutex_lock+0x7c/0x4d8
[ 68.296291] drm_gem_shmem_free+0x7c/0x198
[ 68.296304] panfrost_gem_free_object+0x118/0x138
[ 68.296318] drm_gem_object_free+0x40/0x68
[ 68.296334] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.296352] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.296368] do_shrink_slab+0x220/0x808
[ 68.296381] shrink_slab+0x11c/0x408
[ 68.296392] shrink_node+0x6ac/0xb90
[ 68.296403] do_try_to_free_pages+0x1dc/0x8d0
[ 68.296416] try_to_free_pages+0x1ec/0x5b0
[ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.296444] __alloc_pages+0x4e0/0x5b8
[ 68.296455] __folio_alloc+0x24/0x60
[ 68.296467] vma_alloc_folio+0xb8/0x2f8
[ 68.296483] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.296498] __handle_mm_fault+0x918/0x12a8
[ 68.296513] handle_mm_fault+0x130/0x300
[ 68.296527] do_page_fault+0x1d0/0x568
[ 68.296539] do_translation_fault+0xa0/0xb8
[ 68.296551] do_mem_abort+0x68/0xf8
[ 68.296562] el0_da+0x74/0x100
[ 68.296572] el0t_64_sync_handler+0x68/0xc0
[ 68.296585] el0t_64_sync+0x18c/0x190
[ 68.296596]
[ 68.296596] other info that might help us debug this:
[ 68.296596]
[ 68.296601] Possible unsafe locking scenario:
[ 68.296601]
[ 68.296604] CPU0 CPU1
[ 68.296608] ---- ----
[ 68.296612] lock(fs_reclaim);
[ 68.296622] lock(reservation_ww_class_mutex);
[ 68.296633] lock(fs_reclaim);
[ 68.296644] lock(reservation_ww_class_mutex);
[ 68.296654]
[ 68.296654] *** DEADLOCK ***
[ 68.296654]
[ 68.296658] 3 locks held by cc1/295:
[ 68.296666] #0: ffff00000616e898 (&mm->mmap_lock){++++}-{3:3}, at: do_page_fault+0x144/0x568
[ 68.296702] #1: ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296740] #2: ffff80000c1215b0 (shrinker_rwsem){++++}-{3:3}, at: shrink_slab+0xc0/0x408
[ 68.296774]
[ 68.296774] stack backtrace:
[ 68.296780] CPU: 2 PID: 295 Comm: cc1 Not tainted 5.19.0-rc3+ #400
[ 68.296794] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 68.296803] Call trace:
[ 68.296808] dump_backtrace+0x1e4/0x1f0
[ 68.296821] show_stack+0x20/0x70
[ 68.296832] dump_stack_lvl+0x8c/0xb8
[ 68.296849] dump_stack+0x1c/0x38
[ 68.296864] print_circular_bug.isra.0+0x284/0x378
[ 68.296881] check_noncircular+0x1d8/0x1f8
[ 68.296896] __lock_acquire+0x1724/0x2398
[ 68.296911] lock_acquire+0x218/0x5b0
[ 68.296926] __ww_mutex_lock.constprop.0+0x158/0x2378
[ 68.296942] ww_mutex_lock+0x7c/0x4d8
[ 68.296956] drm_gem_shmem_free+0x7c/0x198
[ 68.296970] panfrost_gem_free_object+0x118/0x138
[ 68.296984] drm_gem_object_free+0x40/0x68
[ 68.296999] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8
[ 68.297017] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170
[ 68.297033] do_shrink_slab+0x220/0x808
[ 68.297045] shrink_slab+0x11c/0x408
[ 68.297056] shrink_node+0x6ac/0xb90
[ 68.297068] do_try_to_free_pages+0x1dc/0x8d0
[ 68.297081] try_to_free_pages+0x1ec/0x5b0
[ 68.297094] __alloc_pages_slowpath.constprop.0+0x528/0x1470
[ 68.297110] __alloc_pages+0x4e0/0x5b8
[ 68.297122] __folio_alloc+0x24/0x60
[ 68.297134] vma_alloc_folio+0xb8/0x2f8
[ 68.297148] alloc_zeroed_user_highpage_movable+0x58/0x68
[ 68.297163] __handle_mm_fault+0x918/0x12a8
[ 68.297178] handle_mm_fault+0x130/0x300
[ 68.297193] do_page_fault+0x1d0/0x568
[ 68.297205] do_translation_fault+0xa0/0xb8
[ 68.297218] do_mem_abort+0x68/0xf8
[ 68.297229] el0_da+0x74/0x100
[ 68.297239] el0t_64_sync_handler+0x68/0xc0
[ 68.297252] el0t_64_sync+0x18c/0x190
[ 68.471812] arm-scmi firmware:scmi: timed out in resp(caller: scmi_power_state_set+0x11c/0x190)
[ 68.501947] arm-scmi firmware:scmi: Message for 119 type 0 is not expected!
[ 68.939686] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000915e2d34
[ 69.739386] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 70.415329] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 70.987166] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ffb7ff37
[ 71.914939] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 72.426987] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 73.578683] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 74.090555] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000075d00f9
[ 74.922709] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 75.434401] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000154189b
[ 76.394300] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 76.906236] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ee980c7e
[ 79.657234] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000f6d059fb
[ 80.168831] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000061a0f6bf
[ 80.808354] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000071ade02
[ 81.319967] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b0afea73
[ 81.831574] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000d78f36c2
[ 82.343160] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000f689397
[ 83.046689] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000412c2a2f
[ 83.558352] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000020e551b3
[ 84.261913] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 84.773576] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 85.317275] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 85.829035] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000000e92b26e
[ 86.660555] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000ac77ac55
[ 87.172126] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b940e406
[ 87.875846] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001c6fc094
[ 88.387443] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000009437aace
[ 89.059175] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000075dadb7f
[ 89.570960] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000005add546
[ 90.146687] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000cba2873c
[ 90.662497] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a4beb490
[ 95.392748] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000005b5fc4ec
[ 95.904179] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a17436ee
[ 96.416085] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000003888d2a7
[ 96.927874] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000093e04a98
[ 97.439742] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000c036a911
[ 97.954109] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000084c51113
[ 98.467374] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000664663ce
[ 98.975192] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=0000000060f2d45c
[ 99.487231] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000b29288f8
[ 99.998833] panfrost 2d000000.gpu: gpu sched timeout, js=0, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000002f07ab24
[ 100.510744] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000008c15c751
[ 100.511411] ==================================================================
[ 100.511419] BUG: KASAN: use-after-free in irq_work_single+0xa4/0x110
[ 100.511445] Write of size 4 at addr ffff0000107f5830 by task glmark2-es2-drm/280
[ 100.511458]
[ 100.511464] CPU: 1 PID: 280 Comm: glmark2-es2-drm Not tainted 5.19.0-rc3+ #400
[ 100.511479] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 100.511489] Call trace:
[ 100.511494] dump_backtrace+0x1e4/0x1f0
[ 100.511512] show_stack+0x20/0x70
[ 100.511523] dump_stack_lvl+0x8c/0xb8
[ 100.511543] print_report+0x16c/0x668
[ 100.511559] kasan_report+0x80/0x208
[ 100.511574] kasan_check_range+0x100/0x1b8
[ 100.511590] __kasan_check_write+0x34/0x60
[ 100.511607] irq_work_single+0xa4/0x110
[ 100.511619] irq_work_run_list+0x6c/0x88
[ 100.511632] irq_work_run+0x28/0x48
[ 100.511644] ipi_handler+0x254/0x468
[ 100.511664] handle_percpu_devid_irq+0x11c/0x518
[ 100.511681] generic_handle_domain_irq+0x50/0x70
[ 100.511699] gic_handle_irq+0xd4/0x118
[ 100.511711] call_on_irq_stack+0x2c/0x58
[ 100.511725] do_interrupt_handler+0xc0/0xc8
[ 100.511741] el1_interrupt+0x40/0x68
[ 100.511754] el1h_64_irq_handler+0x18/0x28
[ 100.511767] el1h_64_irq+0x64/0x68
[ 100.511778] irq_work_queue+0xc0/0xd8
[ 100.511790] drm_sched_entity_fini+0x2c4/0x3b0
[ 100.511805] drm_sched_entity_destroy+0x2c/0x40
[ 100.511818] panfrost_job_close+0x44/0x1c0
[ 100.511833] panfrost_postclose+0x38/0x60
[ 100.511845] drm_file_free.part.0+0x33c/0x4b8
[ 100.511862] drm_close_helper.isra.0+0xc0/0xd8
[ 100.511877] drm_release+0xe4/0x1e0
[ 100.511891] __fput+0xf8/0x390
[ 100.511904] ____fput+0x18/0x28
[ 100.511917] task_work_run+0xc4/0x1e0
[ 100.511929] do_exit+0x554/0x1168
[ 100.511945] do_group_exit+0x60/0x108
[ 100.511960] __arm64_sys_exit_group+0x34/0x38
[ 100.511977] invoke_syscall+0x64/0x180
[ 100.511993] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512012] do_el0_svc+0x48/0xe8
[ 100.512028] el0_svc+0x5c/0xe0
[ 100.512038] el0t_64_sync_handler+0xb8/0xc0
[ 100.512051] el0t_64_sync+0x18c/0x190
[ 100.512064]
[ 100.512068] Allocated by task 280:
[ 100.512075] kasan_save_stack+0x2c/0x58
[ 100.512091] __kasan_kmalloc+0x90/0xb8
[ 100.512105] kmem_cache_alloc_trace+0x1d4/0x330
[ 100.512118] panfrost_ioctl_submit+0x100/0x630
[ 100.512131] drm_ioctl_kernel+0x160/0x250
[ 100.512147] drm_ioctl+0x36c/0x628
[ 100.512161] __arm64_sys_ioctl+0xd8/0x120
[ 100.512178] invoke_syscall+0x64/0x180
[ 100.512194] el0_svc_common.constprop.0+0x13c/0x170
[ 100.512211] do_el0_svc+0x48/0xe8
[ 100.512226] el0_svc+0x5c/0xe0
[ 100.512236] el0t_64_sync_handler+0xb8/0xc0
[ 100.512248] el0t_64_sync+0x18c/0x190
[ 100.512259]
[ 100.512262] Freed by task 280:
[ 100.512268] kasan_save_stack+0x2c/0x58
[ 100.512283] kasan_set_track+0x2c/0x40
[ 100.512296] kasan_set_free_info+0x28/0x50
[ 100.512312] __kasan_slab_free+0xf0/0x170
[ 100.512326] kfree+0x124/0x418
[ 100.512337] panfrost_job_cleanup+0x1f0/0x298
[ 100.512350] panfrost_job_free+0x80/0xb0
[ 100.512363] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 100.512377] irq_work_single+0x88/0x110
[ 100.512389] irq_work_run_list+0x6c/0x88
[ 100.512401] irq_work_run+0x28/0x48
[ 100.512413] ipi_handler+0x254/0x468
[ 100.512427] handle_percpu_devid_irq+0x11c/0x518
[ 100.512443] generic_handle_domain_irq+0x50/0x70
[ 100.512460] gic_handle_irq+0xd4/0x118
[ 100.512471]
[ 100.512474] The buggy address belongs to the object at ffff0000107f5800
[ 100.512474] which belongs to the cache kmalloc-512 of size 512
[ 100.512484] The buggy address is located 48 bytes inside of
[ 100.512484] 512-byte region [ffff0000107f5800, ffff0000107f5a00)
[ 100.512497]
[ 100.512500] The buggy address belongs to the physical page:
[ 100.512506] page:000000000a626feb refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x907f4
[ 100.512520] head:000000000a626feb order:2 compound_mapcount:0 compound_pincount:0
[ 100.512530] flags: 0xffff00000010200(slab|head|node=0|zone=0|lastcpupid=0xffff)
[ 100.512556] raw: 0ffff00000010200 fffffc0000076400 dead000000000002 ffff000000002600
[ 100.512569] raw: 0000000000000000 0000000080100010 00000001ffffffff 0000000000000000
[ 100.512577] page dumped because: kasan: bad access detected
[ 100.512582]
[ 100.512585] Memory state around the buggy address:
[ 100.512592] ffff0000107f5700: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 100.512602] ffff0000107f5780: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
[ 100.512612] >ffff0000107f5800: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512619] ^
[ 100.512627] ffff0000107f5880: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512636] ffff0000107f5900: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[ 100.512643] ==================================================================
[ 101.022573] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000be4b1b31
[ 101.534469] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=00000000a8ff2c8a
[ 101.535981] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:870
[ 101.535994] in_atomic(): 1, irqs_disabled(): 128, non_block: 0, pid: 280, name: glmark2-es2-drm
[ 101.536006] preempt_count: 10000, expected: 0
[ 101.536012] RCU nest depth: 0, expected: 0
[ 101.536019] INFO: lockdep is turned off.
[ 101.536023] irq event stamp: 1666508
[ 101.536029] hardirqs last enabled at (1666507): [<ffff80000997ed70>] exit_to_kernel_mode.isra.0+0x40/0x140
[ 101.536056] hardirqs last disabled at (1666508): [<ffff800009985030>] __schedule+0xb38/0xea8
[ 101.536076] softirqs last enabled at (1664950): [<ffff800008010ac8>] __do_softirq+0x6b8/0x89c
[ 101.536092] softirqs last disabled at (1664941): [<ffff8000080e4fdc>] irq_exit_rcu+0x27c/0x2b0
[ 101.536118] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B
5.19.0-rc3+ #400
[ 101.536134] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.536143] Call trace:
[ 101.536147] dump_backtrace+0x1e4/0x1f0
[ 101.536161] show_stack+0x20/0x70
[ 101.536171] dump_stack_lvl+0x8c/0xb8
[ 101.536189] dump_stack+0x1c/0x38
[ 101.536204] __might_resched+0x1f0/0x2b0
[ 101.536220] __might_sleep+0x74/0xd0
[ 101.536234] ww_mutex_lock+0x40/0x4d8
[ 101.536249] drm_gem_shmem_free+0x7c/0x198
[ 101.536264] panfrost_gem_free_object+0x118/0x138
[ 101.536278] drm_gem_object_free+0x40/0x68
[ 101.536295] panfrost_job_cleanup+0x1bc/0x298
[ 101.536309] panfrost_job_free+0x80/0xb0
[ 101.536322] drm_sched_entity_kill_jobs_irq_work+0x80/0xa0
[ 101.536337] irq_work_single+0x88/0x110
[ 101.536351] irq_work_run_list+0x6c/0x88
[ 101.536364] irq_work_run+0x28/0x48
[ 101.536375] ipi_handler+0x254/0x468
[ 101.536392] handle_percpu_devid_irq+0x11c/0x518
[ 101.536409] generic_handle_domain_irq+0x50/0x70
[ 101.536428] gic_handle_irq+0xd4/0x118
[ 101.536439] call_on_irq_stack+0x2c/0x58
[ 101.536453] do_interrupt_handler+0xc0/0xc8
[ 101.536468] el1_interrupt+0x40/0x68
[ 101.536479] el1h_64_irq_handler+0x18/0x28
[ 101.536492] el1h_64_irq+0x64/0x68
[ 101.536503] __asan_load8+0x30/0xd0
[ 101.536519] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.536532] drm_sched_entity_destroy+0x2c/0x40
[ 101.536545] panfrost_job_close+0x44/0x1c0
[ 101.536559] panfrost_postclose+0x38/0x60
[ 101.536571] drm_file_free.part.0+0x33c/0x4b8
[ 101.536586] drm_close_helper.isra.0+0xc0/0xd8
[ 101.536601] drm_release+0xe4/0x1e0
[ 101.536615] __fput+0xf8/0x390
[ 101.536628] ____fput+0x18/0x28
[ 101.536640] task_work_run+0xc4/0x1e0
[ 101.536652] do_exit+0x554/0x1168
[ 101.536667] do_group_exit+0x60/0x108
[ 101.536682] __arm64_sys_exit_group+0x34/0x38
[ 101.536698] invoke_syscall+0x64/0x180
[ 101.536714] el0_svc_common.constprop.0+0x13c/0x170
[ 101.536733] do_el0_svc+0x48/0xe8
[ 101.536748] el0_svc+0x5c/0xe0
[ 101.536759] el0t_64_sync_handler+0xb8/0xc0
[ 101.536771] el0t_64_sync+0x18c/0x190
[ 101.541928] ------------[ cut here ]------------
[ 101.541934] kernel BUG at kernel/irq_work.c:235!
[ 101.541944] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 101.541961] Modules linked in:
[ 101.541978] CPU: 1 PID: 280 Comm: glmark2-es2-drm Tainted: G B W
5.19.0-rc3+ #400
[ 101.541997] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 101.542009] pstate: 40000005 (nZcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 101.542027] pc : irq_work_run_list+0x80/0x88
[ 101.542044] lr : irq_work_run+0x34/0x48
[ 101.542060] sp : ffff80000da37eb0
[ 101.542069] x29: ffff80000da37eb0 x28: ffff000006bb0000 x27: ffff000006bb0008
[ 101.542107] x26: ffff80000da37f20 x25: ffff8000080304d8 x24: 0000000000000001
[ 101.542142] x23: ffff80000abcd008 x22: ffff80000da37ed0 x21: ffff80001c0de000
[ 101.542177] x20: ffff80000abcd008 x19: ffff80000abdbad0 x18: 0000000000000000
[ 101.542212] x17: 616e202c30383220 x16: 3a646970202c3020 x15: ffff8000082df9d0
[ 101.542246] x14: ffff800008dfada8 x13: 0000000000000003 x12: 1fffe000018b2a06
[ 101.542280] x11: ffff6000018b2a06 x10: dfff800000000000 x9 : ffff00000c595033
[ 101.542315] x8 : ffff6000018b2a07 x7 : 0000000000000001 x6 : 00000000000000fb
[ 101.542349] x5 : ffff00000c595030 x4 : 0000000000000000 x3 : ffff00000c595030
[ 101.542382] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff000026cb9ad0
[ 101.542416] Call trace:
[ 101.542424] irq_work_run_list+0x80/0x88
[ 101.542441] ipi_handler+0x254/0x468
[ 101.542460] handle_percpu_devid_irq+0x11c/0x518
[ 101.542480] generic_handle_domain_irq+0x50/0x70
[ 101.542501] gic_handle_irq+0xd4/0x118
[ 101.542516] call_on_irq_stack+0x2c/0x58
[ 101.542534] do_interrupt_handler+0xc0/0xc8
[ 101.542553] el1_interrupt+0x40/0x68
[ 101.542568] el1h_64_irq_handler+0x18/0x28
[ 101.542584] el1h_64_irq+0x64/0x68
[ 101.542599] __asan_load8+0x30/0xd0
[ 101.542617] drm_sched_entity_fini+0x1e8/0x3b0
[ 101.542634] drm_sched_entity_destroy+0x2c/0x40
[ 101.542651] panfrost_job_close+0x44/0x1c0
[ 101.542669] panfrost_postclose+0x38/0x60
[ 101.542685] drm_file_free.part.0+0x33c/0x4b8
[ 101.542704] drm_close_helper.isra.0+0xc0/0xd8
[ 101.542723] drm_release+0xe4/0x1e0
[ 101.542740] __fput+0xf8/0x390
[ 101.542756] ____fput+0x18/0x28
[ 101.542773] task_work_run+0xc4/0x1e0
[ 101.542788] do_exit+0x554/0x1168
[ 101.542806] do_group_exit+0x60/0x108
[ 101.542825] __arm64_sys_exit_group+0x34/0x38
[ 101.542845] invoke_syscall+0x64/0x180
[ 101.542865] el0_svc_common.constprop.0+0x13c/0x170
[ 101.542887] do_el0_svc+0x48/0xe8
[ 101.542906] el0_svc+0x5c/0xe0
[ 101.542921] el0t_64_sync_handler+0xb8/0xc0
[ 101.542938] el0t_64_sync+0x18c/0x190
[ 101.542960] Code: a94153f3 a8c27bfd d50323bf d65f03c0 (d4210000)
[ 101.542979] ---[ end trace 0000000000000000 ]---
[ 101.678650] Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
[ 102.046301] panfrost 2d000000.gpu: gpu sched timeout, js=1, config=0x0, status=0x0, head=0x0, tail=0x0, sched_job=000000001da14c98
[ 103.227334] SMP: stopping secondary CPUs
[ 103.241055] Kernel Offset: disabled
[ 103.254316] CPU features: 0x800,00184810,00001086
[ 103.268904] Memory Limit: 800 MB
[ 103.411625] ---[ end Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt ]---Dmitry Osipenko
maintainerQuoted 59 lines · Robin Murphy
----->8-----
[ 68.295951] ====================================================== [ 68.295956] WARNING: possible circular locking dependency detected [ 68.295963] 5.19.0-rc3+ #400 Not tainted
[ 68.295972] ------------------------------------------------------ [ 68.295977] cc1/295 is trying to acquire lock:
[ 68.295986]ffff000008d7f1a0(reservation_ww_class_mutex){+.+.}-{3:3}, at:drm_gem_shmem_free+0x7c/0x198[ 68.296036]
[ 68.296036] but task is already holding lock:
[ 68.296041]ffff80000c14b820(fs_reclaim){+.+.}-{0:0}, at:
__alloc_pages_slowpath.constprop.0+0x4d8/0x1470
[ 68.296080]
[ 68.296080] which lock already depends on the new lock.
[ 68.296080]
[ 68.296085]
[ 68.296085] the existing dependency chain (in reverse order) is:
[ 68.296090]
[ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}:
[ 68.296111]fs_reclaim_acquire+0xb8/0x150
[ 68.296130]dma_resv_lockdep+0x298/0x3fc
[ 68.296148]do_one_initcall+0xe4/0x5f8
[ 68.296163]kernel_init_freeable+0x414/0x49c
[ 68.296180]kernel_init+0x2c/0x148
[ 68.296195]ret_from_fork+0x10/0x20
[ 68.296207]
[ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}:
[ 68.296229]__lock_acquire+0x1724/0x2398
[ 68.296246]lock_acquire+0x218/0x5b0
[ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378 [ 68.296277]ww_mutex_lock+0x7c/0x4d8
[ 68.296291]drm_gem_shmem_free+0x7c/0x198
[ 68.296304]panfrost_gem_free_object+0x118/0x138
[ 68.296318]drm_gem_object_free+0x40/0x68
[ 68.296334]drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8[ 68.296352]drm_gem_shmem_shrinker_scan_objects+0xa4/0x170[ 68.296368]do_shrink_slab+0x220/0x808
[ 68.296381]shrink_slab+0x11c/0x408
[ 68.296392]shrink_node+0x6ac/0xb90
[ 68.296403]do_try_to_free_pages+0x1dc/0x8d0
[ 68.296416]try_to_free_pages+0x1ec/0x5b0
[ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470 [ 68.296444]__alloc_pages+0x4e0/0x5b8
[ 68.296455]__folio_alloc+0x24/0x60
[ 68.296467]vma_alloc_folio+0xb8/0x2f8
[ 68.296483]alloc_zeroed_user_highpage_movable+0x58/0x68[ 68.296498]__handle_mm_fault+0x918/0x12a8
[ 68.296513]handle_mm_fault+0x130/0x300
[ 68.296527]do_page_fault+0x1d0/0x568
[ 68.296539]do_translation_fault+0xa0/0xb8
[ 68.296551]do_mem_abort+0x68/0xf8
[ 68.296562]el0_da+0x74/0x100
[ 68.296572]el0t_64_sync_handler+0x68/0xc0
[ 68.296585]el0t_64_sync+0x18c/0x190
[ 68.296596]
[ 68.296596] other info that might help us debug this:
[ 68.296596]
[ 68.296601] Possible unsafe locking scenario:
[ 68.296601]
Quoted 8 lines
[ 68.296604] CPU0 CPU1
[ 68.296608] ---- ----
[ 68.296612]lock(fs_reclaim);
[ 68.296622]lock(reservation_ww_class_mutex);
[ 68.296633]lock(fs_reclaim); [ 68.296644]lock(reservation_ww_class_mutex);
[ 68.296654]
[ 68.296654] *** DEADLOCK ***
This splat could be ignored for now. I'm aware about it, although haven't looked closely at how to fix it since it's a kind of a lockdep misreporting.
Signature · 2 lines
Best regards, Dmitry
Quoted 70 lines · Dmitry Osipenko
Quoted 59 lines · Robin Murphy
----->8-----
[ 68.295951] ====================================================== [ 68.295956] WARNING: possible circular locking dependency detected [ 68.295963] 5.19.0-rc3+ #400 Not tainted [ 68.295972] ------------------------------------------------------ [ 68.295977] cc1/295 is trying to acquire lock: [ 68.295986] ffff000008d7f1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_gem_shmem_free+0x7c/0x198 [ 68.296036] [ 68.296036] but task is already holding lock: [ 68.296041] ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470 [ 68.296080] [ 68.296080] which lock already depends on the new lock. [ 68.296080] [ 68.296085] [ 68.296085] the existing dependency chain (in reverse order) is: [ 68.296090] [ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}: [ 68.296111] fs_reclaim_acquire+0xb8/0x150 [ 68.296130] dma_resv_lockdep+0x298/0x3fc [ 68.296148] do_one_initcall+0xe4/0x5f8 [ 68.296163] kernel_init_freeable+0x414/0x49c [ 68.296180] kernel_init+0x2c/0x148 [ 68.296195] ret_from_fork+0x10/0x20 [ 68.296207] [ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}: [ 68.296229] __lock_acquire+0x1724/0x2398 [ 68.296246] lock_acquire+0x218/0x5b0 [ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378 [ 68.296277] ww_mutex_lock+0x7c/0x4d8 [ 68.296291] drm_gem_shmem_free+0x7c/0x198 [ 68.296304] panfrost_gem_free_object+0x118/0x138 [ 68.296318] drm_gem_object_free+0x40/0x68 [ 68.296334] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8 [ 68.296352] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170 [ 68.296368] do_shrink_slab+0x220/0x808 [ 68.296381] shrink_slab+0x11c/0x408 [ 68.296392] shrink_node+0x6ac/0xb90 [ 68.296403] do_try_to_free_pages+0x1dc/0x8d0 [ 68.296416] try_to_free_pages+0x1ec/0x5b0 [ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470 [ 68.296444] __alloc_pages+0x4e0/0x5b8 [ 68.296455] __folio_alloc+0x24/0x60 [ 68.296467] vma_alloc_folio+0xb8/0x2f8 [ 68.296483] alloc_zeroed_user_highpage_movable+0x58/0x68 [ 68.296498] __handle_mm_fault+0x918/0x12a8 [ 68.296513] handle_mm_fault+0x130/0x300 [ 68.296527] do_page_fault+0x1d0/0x568 [ 68.296539] do_translation_fault+0xa0/0xb8 [ 68.296551] do_mem_abort+0x68/0xf8 [ 68.296562] el0_da+0x74/0x100 [ 68.296572] el0t_64_sync_handler+0x68/0xc0 [ 68.296585] el0t_64_sync+0x18c/0x190 [ 68.296596] [ 68.296596] other info that might help us debug this: [ 68.296596] [ 68.296601] Possible unsafe locking scenario: [ 68.296601]Quoted 8 lines
[ 68.296604] CPU0 CPU1 [ 68.296608] ---- ---- [ 68.296612] lock(fs_reclaim); [ 68.296622] lock(reservation_ww_class_mutex); [ 68.296633] lock(fs_reclaim); [ 68.296644] lock(reservation_ww_class_mutex); [ 68.296654] [ 68.296654] *** DEADLOCK ***
Quoted 3 lines
This splat could be ignored for now. I'm aware about it, although haven't looked closely at how to fix it since it's a kind of a lockdep misreporting.
The lockdep splat could be fixed with something similar to what I've done in msm, ie. basically just not acquire the lock in the finalizer:
https://patchwork.freedesktop.org/patch/489364/
There is one gotcha to watch for, as danvet pointed out (scan_objects() could still see the obj in the LRU before the finalizer removes it), but if scan_objects() does the kref_get_unless_zero() trick, it is safe.
BR,
-R
Dmitry Osipenko
maintainerQuoted 79 lines · Rob Clark
Quoted 70 lines · Dmitry Osipenko
Quoted 59 lines · Robin Murphy
----->8-----
[ 68.295951] ====================================================== [ 68.295956] WARNING: possible circular locking dependency detected [ 68.295963] 5.19.0-rc3+ #400 Not tainted [ 68.295972] ------------------------------------------------------ [ 68.295977] cc1/295 is trying to acquire lock: [ 68.295986] ffff000008d7f1a0 (reservation_ww_class_mutex){+.+.}-{3:3}, at: drm_gem_shmem_free+0x7c/0x198 [ 68.296036] [ 68.296036] but task is already holding lock: [ 68.296041] ffff80000c14b820 (fs_reclaim){+.+.}-{0:0}, at: __alloc_pages_slowpath.constprop.0+0x4d8/0x1470 [ 68.296080] [ 68.296080] which lock already depends on the new lock. [ 68.296080] [ 68.296085] [ 68.296085] the existing dependency chain (in reverse order) is: [ 68.296090] [ 68.296090] -> #1 (fs_reclaim){+.+.}-{0:0}: [ 68.296111] fs_reclaim_acquire+0xb8/0x150 [ 68.296130] dma_resv_lockdep+0x298/0x3fc [ 68.296148] do_one_initcall+0xe4/0x5f8 [ 68.296163] kernel_init_freeable+0x414/0x49c [ 68.296180] kernel_init+0x2c/0x148 [ 68.296195] ret_from_fork+0x10/0x20 [ 68.296207] [ 68.296207] -> #0 (reservation_ww_class_mutex){+.+.}-{3:3}: [ 68.296229] __lock_acquire+0x1724/0x2398 [ 68.296246] lock_acquire+0x218/0x5b0 [ 68.296260] __ww_mutex_lock.constprop.0+0x158/0x2378 [ 68.296277] ww_mutex_lock+0x7c/0x4d8 [ 68.296291] drm_gem_shmem_free+0x7c/0x198 [ 68.296304] panfrost_gem_free_object+0x118/0x138 [ 68.296318] drm_gem_object_free+0x40/0x68 [ 68.296334] drm_gem_shmem_shrinker_run_objects_scan+0x42c/0x5b8 [ 68.296352] drm_gem_shmem_shrinker_scan_objects+0xa4/0x170 [ 68.296368] do_shrink_slab+0x220/0x808 [ 68.296381] shrink_slab+0x11c/0x408 [ 68.296392] shrink_node+0x6ac/0xb90 [ 68.296403] do_try_to_free_pages+0x1dc/0x8d0 [ 68.296416] try_to_free_pages+0x1ec/0x5b0 [ 68.296429] __alloc_pages_slowpath.constprop.0+0x528/0x1470 [ 68.296444] __alloc_pages+0x4e0/0x5b8 [ 68.296455] __folio_alloc+0x24/0x60 [ 68.296467] vma_alloc_folio+0xb8/0x2f8 [ 68.296483] alloc_zeroed_user_highpage_movable+0x58/0x68 [ 68.296498] __handle_mm_fault+0x918/0x12a8 [ 68.296513] handle_mm_fault+0x130/0x300 [ 68.296527] do_page_fault+0x1d0/0x568 [ 68.296539] do_translation_fault+0xa0/0xb8 [ 68.296551] do_mem_abort+0x68/0xf8 [ 68.296562] el0_da+0x74/0x100 [ 68.296572] el0t_64_sync_handler+0x68/0xc0 [ 68.296585] el0t_64_sync+0x18c/0x190 [ 68.296596] [ 68.296596] other info that might help us debug this: [ 68.296596] [ 68.296601] Possible unsafe locking scenario: [ 68.296601]Quoted 8 lines
[ 68.296604] CPU0 CPU1 [ 68.296608] ---- ---- [ 68.296612] lock(fs_reclaim); [ 68.296622] lock(reservation_ww_class_mutex); [ 68.296633] lock(fs_reclaim); [ 68.296644] lock(reservation_ww_class_mutex); [ 68.296654] [ 68.296654] *** DEADLOCK ***Quoted 3 lines
This splat could be ignored for now. I'm aware about it, although haven't looked closely at how to fix it since it's a kind of a lockdep misreporting.
The lockdep splat could be fixed with something similar to what I've done in msm, ie. basically just not acquire the lock in the finalizer:
Quoted 6 lines
https://patchwork.freedesktop.org/patch/489364/
There is one gotcha to watch for, as danvet pointed out (
scan_objects()could still see the obj in the LRU before the finalizer removes it), but ifscan_objects()does thekref_get_unless_zero()trick, it is safe.
Nice, thank you!
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerQuoted 43 lines · Robin Murphy
[ 100.511411]
==================================================================
[ 100.511419] BUG: KASAN: use-after-free inirq_work_single+0xa4/0x110[ 100.511445] Write of size 4 at addrffff0000107f5830by task glmark2-es2-drm/280
[ 100.511458]
[ 100.511464] CPU: 1 PID: 280 Comm: glmark2-es2-drm Not tainted 5.19.0-rc3+ #400
[ 100.511479] Hardware name: ARM LTD ARM Juno Development Platform/ARM Juno Development Platform, BIOS EDK II Sep 3 2019
[ 100.511489] Call trace:
[ 100.511494]dump_backtrace+0x1e4/0x1f0
[ 100.511512]show_stack+0x20/0x70
[ 100.511523]dump_stack_lvl+0x8c/0xb8
[ 100.511543]print_report+0x16c/0x668
[ 100.511559]kasan_report+0x80/0x208
[ 100.511574]kasan_check_range+0x100/0x1b8
[ 100.511590]__kasan_check_write+0x34/0x60
[ 100.511607]irq_work_single+0xa4/0x110
[ 100.511619]irq_work_run_list+0x6c/0x88
[ 100.511632]irq_work_run+0x28/0x48
[ 100.511644]ipi_handler+0x254/0x468
[ 100.511664]handle_percpu_devid_irq+0x11c/0x518
[ 100.511681]generic_handle_domain_irq+0x50/0x70
[ 100.511699]gic_handle_irq+0xd4/0x118
[ 100.511711]call_on_irq_stack+0x2c/0x58
[ 100.511725]do_interrupt_handler+0xc0/0xc8
[ 100.511741]el1_interrupt+0x40/0x68
[ 100.511754]el1h_64_irq_handler+0x18/0x28
[ 100.511767]el1h_64_irq+0x64/0x68
[ 100.511778]irq_work_queue+0xc0/0xd8
[ 100.511790]drm_sched_entity_fini+0x2c4/0x3b0
[ 100.511805]drm_sched_entity_destroy+0x2c/0x40
[ 100.511818]panfrost_job_close+0x44/0x1c0
[ 100.511833]panfrost_postclose+0x38/0x60
[ 100.511845] drm_file_free.part.0+0x33c/0x4b8
[ 100.511862] drm_close_helper.isra.0+0xc0/0xd8
[ 100.511877]drm_release+0xe4/0x1e0
[ 100.511891]__fput+0xf8/0x390
[ 100.511904]____fput+0x18/0x28
[ 100.511917]task_work_run+0xc4/0x1e0
[ 100.511929]do_exit+0x554/0x1168
[ 100.511945]do_group_exit+0x60/0x108
Quoted 8 lines
[ 100.511960]
__arm64_sys_exit_group+0x34/0x38[ 100.511977]invoke_syscall+0x64/0x180
[ 100.511993] el0_svc_common.constprop.0+0x13c/0x170 [ 100.512012]do_el0_svc+0x48/0xe8
[ 100.512028]el0_svc+0x5c/0xe0
[ 100.512038]el0t_64_sync_handler+0xb8/0xc0[ 100.512051]el0t_64_sync+0x18c/0x190
[ 100.512064]
This one shall be fixed by [1] that is not in the RC kernel yet, please use linux-next.
Signature · 2 lines
Best regards, Dmitry
Dmitry Osipenko
maintainerHello Robin,
Quoted 14 lines · Robin Murphy
Quoted 14 lines
Hello,
This patchset introduces memory shrinker for the VirtIO-GPU DRM driver and adds memory purging and eviction support to VirtIO-GPU driver.
The new dma-buf locking convention is introduced here as well.
During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL, it will also evict idling BOs to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark
the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.
Quoted 6 lines
Quoted 1 line
The Panfrost driver is switched to use generic memory shrinker.
I think we still have some outstanding issues here - Alyssa reported some weirdness yesterday, so I just tried provoking a low-memory condition locally with this series applied and a few debug options enabled, and the results as below were... interesting.
The warning and crash that you got actually are the minor issues.
Alyssa caught an interesting PREEMPT_DEBUG issue in the shrinker that I haven't seen before.
She is also experiencing another problem in the Panfrost driver with a bad shmem pages (I think). It is unrelated to this patchset and apparently require an extra setup for the reproduction.
Signature · 2 lines
Best regards, Dmitry