Skip to contentSkip to conversation

Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers

PATCH Superseded Dmitry Osipenko 5 participants 56 messages Apr 17, 2022 – Jun 19, 2022

Subsystems (6)

v4 of 19 Superseded Latest is v19 →

lore
Share & tools
lore

Dmitry Osipenko

author maintainer
#0 lore raw

v4 cover Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers

Hello,

This patchset introduces memory shrinker for the VirtIO-GPU DRM driver. During OOM, the shrinker will release BOs that are marked as "not needed" by userspace using the new madvise IOCTL, it will also evict idling BOs to SWAP. The userspace in this case is the Mesa VirGL driver, it will mark the cached BOs as "not needed", allowing kernel driver to release memory of the cached shmem BOs on lowmem situations, preventing OOM kills.

This patchset adds memory purging and eviction support to VirtIO-GPU driver.

The Panfrost driver is switched to use generic memory shrinker. Eviction support will come later on, after resolving the blocker bug in Panfrost.

This patchset also includes couple improvements and fixes for various minor things that I found while was working on the shrinker.

The Mesa and IGT patches will be kept on hold until this kernel series will be approved and merged.

This patchset was tested using Qemu and crosvm, including both cases of IOMMU off/on.

Mesa: https://gitlab.freedesktop.org/digetx/mesa/-/commits/virgl-madvise
IGT:  https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/virtio-madvise

https://gitlab.freedesktop.org/digetx/igt-gpu-tools/-/commits/panfrost-madvise

Changelog:

v4: - Corrected minor W=1 warnings reported by kernel test robot for v3.

- Renamed DRM_GEM_SHMEM_PAGES_STATE_ACTIVE/INACTIVE to PINNED/UNPINNED, for more clarity.

v3: - Hardened shrinker's count() with usage of READ_ONCE() since we don't
use atomic type for counting and technically compiler is free to re-fetch counter's variable.

- "Correct drm_gem_shmem_get_sg_table() error handling" now uses PTR_ERR_OR_ZERO(), fixing typo that was made in v2.

- Removed obsoleted shrinker from the Panfrost driver, which I missed to do in v2 by accident and Alyssa Rosenzweig managed to notice it.

- CCed stable kernels in all patches that make fixes, even the minor ones, like was suggested by Emil Velikov and added his r-b to the patches.

- Added t-b from Steven Price to the Panfrost's shrinker patch.

- Corrected doc-comment of drm_gem_shmem_object.madv, like was suggested by Steven Price. Comment now says that madv=1 means "object is purged" instead of saying that value is unused.

- Added more doc-comments to the new shmem shrinker API.

- The "Improve DMA API usage for shmem BOs" patch got more improvements by removing the obsoleted drm_dev_set_unique() quirk and its comment.

- Added patch that makes Virtio-GPU driver to use common dev_is_pci() helper, which was suggested by Robin Murphy.

- Added new "drm/shmem-helper: Take GEM reservation lock instead of drm_gem_shmem locks" patch, which was suggested by Daniel Vetter.

- Added new "drm/virtio: Simplify error handling of virtio_gpu_object_create()" patch.

- Improved "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, like was suggested by Daniel Vetter, by saying that function returns ERR_PTR() and not errno.

- virtio_gpu_purge_object() is fenced properly now, turned out virtio_gpu_notify() doesn't do fencing as I was supposing before. Stress testing of memory eviction revealed that.

- Added new patch that corrects virtio_gpu_plane_cleanup_fb() to use appropriate atomic plane state.

- SHMEM shrinker got eviction support.

- VirtIO-GPU driver now supports memory eviction. It's enabled for a non-blob GEMs only, i.e. for VirGL. The blobs don't support dynamic attaching/detaching of guest's memory, so it's not trivial to enable them.

- Added patch that removes obsoleted drm_gem_shmem_purge()

- Added patch that makes drm_gem_shmem_get_pages() private.

- Added patch that fixes lockup on dma_resv_reserve_fences() error.

v2: - Improved shrinker by using a more fine-grained locking to reduce
contention during scan of objects and dropped locking from the 'counting' callback by tracking count of shrinkable pages. This was suggested by Rob Clark in the comment to v1.

- Factored out common shrinker code into drm_gem_shmem_helper.c and switched Panfrost driver to use the new common memory shrinker. This was proposed by Thomas Zimmermann in his prototype series that he shared with us in the comment to v1. Note that I only compile-tested the Panfrost driver.

- Shrinker now takes object_name_lock during scan to prevent racing with dma-buf exporting.

- Shrinker now takes vmap_lock during scan to prevent racing with shmem vmap/unmap code.

- Added "Correct doc-comment of drm_gem_shmem_get_sg_table()" patch, which I sent out previously as a standalone change, since the drm_gem_shmem_helper.c is now touched by this patchset anyways and it doesn't hurt to group all the patches together.

Dmitry Osipenko (15):
drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling drm/virtio: Check whether transferred 2D BO is shmem drm/virtio: Unlock GEM reservations on virtio_gpu_object_shmem_init()
error
drm/virtio: Unlock reservations on dma_resv_reserve_fences() error drm/virtio: Use appropriate atomic state in
virtio_gpu_plane_cleanup_fb()
drm/virtio: Simplify error handling of virtio_gpu_object_create() drm/virtio: Improve DMA API usage for shmem BOs drm/virtio: Use dev_is_pci()
drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table() drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks drm/shmem-helper: Add generic memory shrinker
drm/virtio: Support memory shrinking
drm/panfrost: Switch to generic memory shrinker drm/shmem-helper: Make drm_gem_shmem_get_pages() private drm/shmem-helper: Remove drm_gem_shmem_purge()

 drivers/gpu/drm/drm_gem_shmem_helper.c        | 814 ++++++++++++++++--
 drivers/gpu/drm/lima/lima_gem.c               |   8 +-
 drivers/gpu/drm/panfrost/Makefile             |   1 -
 drivers/gpu/drm/panfrost/panfrost_device.h    |   4 -
 drivers/gpu/drm/panfrost/panfrost_drv.c       |  19 +-
 drivers/gpu/drm/panfrost/panfrost_gem.c       |  30 +-
 drivers/gpu/drm/panfrost/panfrost_gem.h       |   9 -
 .../gpu/drm/panfrost/panfrost_gem_shrinker.c  | 122 ---
 drivers/gpu/drm/panfrost/panfrost_job.c       |  18 +-
 drivers/gpu/drm/panfrost/panfrost_mmu.c       |  15 +-
 drivers/gpu/drm/virtio/virtgpu_drv.c          |  53 +-
 drivers/gpu/drm/virtio/virtgpu_drv.h          |  20 +-
 drivers/gpu/drm/virtio/virtgpu_gem.c          |  50 +-
 drivers/gpu/drm/virtio/virtgpu_ioctl.c        |  37 +
 drivers/gpu/drm/virtio/virtgpu_kms.c          |  16 +-
 drivers/gpu/drm/virtio/virtgpu_object.c       | 204 +++--
 drivers/gpu/drm/virtio/virtgpu_plane.c        |  23 +-
 drivers/gpu/drm/virtio/virtgpu_vq.c           |  55 +-
 include/drm/drm_device.h                      |   4 +
 include/drm/drm_gem.h                         |  35 +
 include/drm/drm_gem_shmem_helper.h            | 114 ++-
 include/uapi/drm/virtgpu_drm.h                |  14 +
 22 files changed, 1277 insertions(+), 388 deletions(-)
 delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
Signature · 1 line
2.35.1
author maintainer
#1 lore raw

PATCH v4 1/15 drm/virtio: Correct drm_gem_shmem_get_sg_table() error handling

drm_gem_shmem_get_sg_table() never ever returned NULL on error. Correct the error handling to avoid crash on OOM.

---
 drivers/gpu/drm/virtio/virtgpu_object.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/gpu/drm/virtio/virtgpu_object.c +4 -2
index f293e6ad52da..3d0c8d4d1c20 100644
@@ -168,9 +168,11 @@ static int virtio_gpu_object_shmem_init(struct virtio_gpu_device *vgdev,
 	 * since virtio_gpu doesn't support dma-buf import from other devices.
 	 */
 	shmem->pages = drm_gem_shmem_get_sg_table(&bo->base);
-	if (!shmem->pages) {
+	ret = PTR_ERR_OR_ZERO(shmem->pages);
+	if (ret) {
 		drm_gem_shmem_unpin(&bo->base);
-		return -EINVAL;
+		shmem->pages = NULL;
+		return ret;
 	}
 
 	if (use_dma_api) {
Signature · 1 line
2.35.1
author maintainer
#2 lore raw

PATCH v4 2/15 drm/virtio: Check whether transferred 2D BO is shmem

In reply to Dmitry Osipenko

Transferred 2D BO always must be a shmem BO. Add check for that to prevent NULL dereference if userspace passes a VRAM BO.

---
 drivers/gpu/drm/virtio/virtgpu_vq.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_vq.c +1 -1
index 7c052efe8836..2edf31806b74 100644
@@ -595,7 +595,7 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
 	bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
 	struct virtio_gpu_object_shmem *shmem = to_virtio_gpu_shmem(bo);
 
-	if (use_dma_api)
+	if (virtio_gpu_is_shmem(bo) && use_dma_api)
 		dma_sync_sgtable_for_device(vgdev->vdev->dev.parent,
 					    shmem->pages, DMA_TO_DEVICE);
 
Signature · 1 line
2.35.1
author maintainer
#3 lore raw

PATCH v4 3/15 drm/virtio: Unlock GEM reservations on virtio_gpu_object_shmem_init() error

In reply to Dmitry Osipenko

Unlock reservations in the error code path of virtio_gpu_object_create() to silence debug warning splat produced by ww_mutex_destroy(&obj->lock) when GEM is released with the held lock.

---
 drivers/gpu/drm/virtio/virtgpu_object.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/gpu/drm/virtio/virtgpu_object.c +2 -0
index 3d0c8d4d1c20..21c19cdedce0 100644
@@ -250,6 +250,8 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 
 	ret = virtio_gpu_object_shmem_init(vgdev, bo, &ents, &nents);
 	if (ret != 0) {
+		if (fence)
+			virtio_gpu_array_unlock_resv(objs);
 		virtio_gpu_array_put_free(objs);
 		virtio_gpu_free_object(&shmem_obj->base);
 		return ret;
Signature · 1 line
2.35.1
author maintainer
#4 lore raw

PATCH v4 4/15 drm/virtio: Unlock reservations on dma_resv_reserve_fences() error

In reply to Dmitry Osipenko

Unlock reservations on dma_resv_reserve_fences() error to fix recursive locking of the reservations when this error happens.

---
 drivers/gpu/drm/virtio/virtgpu_gem.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_gem.c +3 -1
index 580a78809836..7db48d17ee3a 100644
@@ -228,8 +228,10 @@ int virtio_gpu_array_lock_resv(struct virtio_gpu_object_array *objs)
 
 	for (i = 0; i < objs->nents; ++i) {
 		ret = dma_resv_reserve_fences(objs->objs[i]->resv, 1);
-		if (ret)
+		if (ret) {
+			virtio_gpu_array_unlock_resv(objs);
 			return ret;
+		}
 	}
 	return ret;
 }
Signature · 1 line
2.35.1
author maintainer
#5 lore raw

PATCH v4 5/15 drm/virtio: Use appropriate atomic state in virtio_gpu_plane_cleanup_fb()

In reply to Dmitry Osipenko

Make virtio_gpu_plane_cleanup_fb() to clean the state which DRM core wants to clean up and not the current plane's state. Normally the older atomic state is cleaned up, but the newer state could also be cleaned up in case of aborted commits.

---
 drivers/gpu/drm/virtio/virtgpu_plane.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/virtio/virtgpu_plane.c +3 -3
index 6d3cc9e238a4..7148f3813d8b 100644
@@ -266,14 +266,14 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,
 }
 
 static void virtio_gpu_plane_cleanup_fb(struct drm_plane *plane,
-					struct drm_plane_state *old_state)
+					struct drm_plane_state *state)
 {
 	struct virtio_gpu_framebuffer *vgfb;
 
-	if (!plane->state->fb)
+	if (!state->fb)
 		return;
 
-	vgfb = to_virtio_gpu_framebuffer(plane->state->fb);
+	vgfb = to_virtio_gpu_framebuffer(state->fb);
 	if (vgfb->fence) {
 		dma_fence_put(&vgfb->fence->f);
 		vgfb->fence = NULL;
Signature · 1 line
2.35.1
author maintainer
#6 lore raw

PATCH v4 6/15 drm/virtio: Simplify error handling of virtio_gpu_object_create()

In reply to Dmitry Osipenko

Change the order of SHMEM initialization and reservation locking to make code cleaner a tad and to prepare to transitioning of the common GEM SHMEM code to use the GEM's reservation lock instead of the shmem.page_lock.

There is no need to lock reservation during allocation of the SHMEM pages because the lock is needed only to avoid racing with the async host-side allocation. Hence we can safely move the SHMEM initialization out of the reservation lock.

---
 drivers/gpu/drm/virtio/virtgpu_object.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
drivers/gpu/drm/virtio/virtgpu_object.c +4 -9
index 21c19cdedce0..18f70ef6b4d0 100644
@@ -236,6 +236,10 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 
 	bo->dumb = params->dumb;
 
+	ret = virtio_gpu_object_shmem_init(vgdev, bo, &ents, &nents);
+	if (ret != 0)
+		goto err_put_id;
+
 	if (fence) {
 		ret = -ENOMEM;
 		objs = virtio_gpu_array_alloc(1);
@@ -248,15 +252,6 @@ int virtio_gpu_object_create(struct virtio_gpu_device *vgdev,
 			goto err_put_objs;
 	}
 
-	ret = virtio_gpu_object_shmem_init(vgdev, bo, &ents, &nents);
-	if (ret != 0) {
-		if (fence)
-			virtio_gpu_array_unlock_resv(objs);
-		virtio_gpu_array_put_free(objs);
-		virtio_gpu_free_object(&shmem_obj->base);
-		return ret;
-	}
-
 	if (params->blob) {
 		if (params->blob_mem == VIRTGPU_BLOB_MEM_GUEST)
 			bo->guest_blob = true;
Signature · 1 line
2.35.1
author maintainer
#7 lore raw

PATCH v4 7/15 drm/virtio: Improve DMA API usage for shmem BOs

In reply to Dmitry Osipenko

DRM API requires the DRM's driver to be backed with the device that can be used for generic DMA operations. The VirtIO-GPU device can't perform DMA operations if it uses PCI transport because PCI device driver creates a virtual VirtIO-GPU device that isn't associated with the PCI. Use PCI's GPU device for the DRM's device instead of the VirtIO-GPU device and drop DMA-related hacks from the VirtIO-GPU driver.

---
 drivers/gpu/drm/virtio/virtgpu_drv.c    | 51 ++++++----------------
 drivers/gpu/drm/virtio/virtgpu_drv.h    |  5 +--
 drivers/gpu/drm/virtio/virtgpu_kms.c    |  7 ++--
 drivers/gpu/drm/virtio/virtgpu_object.c | 56 +++++--------------------
 drivers/gpu/drm/virtio/virtgpu_vq.c     | 13 +++---
 5 files changed, 32 insertions(+), 100 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +12 -39
index 5f25a8d15464..0141b7df97ec 100644
@@ -46,12 +46,11 @@ static int virtio_gpu_modeset = -1;
 MODULE_PARM_DESC(modeset, "Disable/Enable modesetting");
 module_param_named(modeset, virtio_gpu_modeset, int, 0400);
 
-static int virtio_gpu_pci_quirk(struct drm_device *dev, struct virtio_device *vdev)
+static int virtio_gpu_pci_quirk(struct drm_device *dev)
 {
-	struct pci_dev *pdev = to_pci_dev(vdev->dev.parent);
+	struct pci_dev *pdev = to_pci_dev(dev->dev);
 	const char *pname = dev_name(&pdev->dev);
 	bool vga = (pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA;
-	char unique[20];
 	int ret;
 
 	DRM_INFO("pci: %s detected at %s\n",
@@ -63,39 +62,7 @@ static int virtio_gpu_pci_quirk(struct drm_device *dev, struct virtio_device *vd
 			return ret;
 	}
 
-	/*
-	 * Normally the drm_dev_set_unique() call is done by core DRM.
-	 * The following comment covers, why virtio cannot rely on it.
-	 *
-	 * Unlike the other virtual GPU drivers, virtio abstracts the
-	 * underlying bus type by using struct virtio_device.
-	 *
-	 * Hence the dev_is_pci() check, used in core DRM, will fail
-	 * and the unique returned will be the virtio_device "virtio0",
-	 * while a "pci:..." one is required.
-	 *
-	 * A few other ideas were considered:
-	 * - Extend the dev_is_pci() check [in drm_set_busid] to
-	 *   consider virtio.
-	 *   Seems like a bigger hack than what we have already.
-	 *
-	 * - Point drm_device::dev to the parent of the virtio_device
-	 *   Semantic changes:
-	 *   * Using the wrong device for i2c, framebuffer_alloc and
-	 *     prime import.
-	 *   Visual changes:
-	 *   * Helpers such as DRM_DEV_ERROR, dev_info, drm_printer,
-	 *     will print the wrong information.
-	 *
-	 * We could address the latter issues, by introducing
-	 * drm_device::bus_dev, ... which would be used solely for this.
-	 *
-	 * So for the moment keep things as-is, with a bulky comment
-	 * for the next person who feels like removing this
-	 * drm_dev_set_unique() quirk.
-	 */
-	snprintf(unique, sizeof(unique), "pci:%s", pname);
-	return drm_dev_set_unique(dev, unique);
+	return 0;
 }
 
 static int virtio_gpu_probe(struct virtio_device *vdev)
@@ -109,18 +76,24 @@ static int virtio_gpu_probe(struct virtio_device *vdev)
 	if (virtio_gpu_modeset == 0)
 		return -EINVAL;
 
-	dev = drm_dev_alloc(&driver, &vdev->dev);
+	/*
+	 * The virtio-gpu device is a virtual device that doesn't have DMA
+	 * ops assigned to it, nor DMA mask set and etc. Its parent device
+	 * is actual GPU device we want to use it for the DRM's device in
+	 * order to benefit from using generic DRM APIs.
+	 */
+	dev = drm_dev_alloc(&driver, vdev->dev.parent);
 	if (IS_ERR(dev))
 		return PTR_ERR(dev);
 	vdev->priv = dev;
 
 	if (!strcmp(vdev->dev.parent->bus->name, "pci")) {
-		ret = virtio_gpu_pci_quirk(dev, vdev);
+		ret = virtio_gpu_pci_quirk(dev);
 		if (ret)
 			goto err_free;
 	}
 
-	ret = virtio_gpu_init(dev);
+	ret = virtio_gpu_init(vdev, dev);
 	if (ret)
 		goto err_free;
 
drivers/gpu/drm/virtio/virtgpu_drv.h +1 -4
index 0a194aaad419..b2d93cb12ebf 100644
@@ -100,8 +100,6 @@ struct virtio_gpu_object {
 
 struct virtio_gpu_object_shmem {
 	struct virtio_gpu_object base;
-	struct sg_table *pages;
-	uint32_t mapped;
 };
 
 struct virtio_gpu_object_vram {
@@ -214,7 +212,6 @@ struct virtio_gpu_drv_cap_cache {
 };
 
 struct virtio_gpu_device {
-	struct device *dev;
 	struct drm_device *ddev;
 
 	struct virtio_device *vdev;
@@ -282,7 +279,7 @@ extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
 void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
 
 /* virtgpu_kms.c */
-int virtio_gpu_init(struct drm_device *dev);
+int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev);
 void virtio_gpu_deinit(struct drm_device *dev);
 void virtio_gpu_release(struct drm_device *dev);
 int virtio_gpu_driver_open(struct drm_device *dev, struct drm_file *file);
drivers/gpu/drm/virtio/virtgpu_kms.c +3 -4
index 3313b92db531..0d1e3eb61bee 100644
@@ -110,7 +110,7 @@ static void virtio_gpu_get_capsets(struct virtio_gpu_device *vgdev,
 	vgdev->num_capsets = num_capsets;
 }
 
-int virtio_gpu_init(struct drm_device *dev)
+int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 {
 	static vq_callback_t *callbacks[] = {
 		virtio_gpu_ctrl_ack, virtio_gpu_cursor_ack
@@ -123,7 +123,7 @@ int virtio_gpu_init(struct drm_device *dev)
 	u32 num_scanouts, num_capsets;
 	int ret = 0;
 
-	if (!virtio_has_feature(dev_to_virtio(dev->dev), VIRTIO_F_VERSION_1))
+	if (!virtio_has_feature(vdev, VIRTIO_F_VERSION_1))
 		return -ENODEV;
 
 	vgdev = kzalloc(sizeof(struct virtio_gpu_device), GFP_KERNEL);
@@ -132,8 +132,7 @@ int virtio_gpu_init(struct drm_device *dev)
 
 	vgdev->ddev = dev;
 	dev->dev_private = vgdev;
-	vgdev->vdev = dev_to_virtio(dev->dev);
-	vgdev->dev = dev->dev;
+	vgdev->vdev = vdev;
 
 	spin_lock_init(&vgdev->display_info_lock);
 	spin_lock_init(&vgdev->resource_export_lock);
drivers/gpu/drm/virtio/virtgpu_object.c +11 -45

82 lines not on this page · raw message

drivers/gpu/drm/virtio/virtgpu_vq.c +5 -8
index 2edf31806b74..06566e44307d 100644
@@ -593,11 +593,10 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
 	struct virtio_gpu_transfer_to_host_2d *cmd_p;
 	struct virtio_gpu_vbuffer *vbuf;
 	bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
-	struct virtio_gpu_object_shmem *shmem = to_virtio_gpu_shmem(bo);
 
 	if (virtio_gpu_is_shmem(bo) && use_dma_api)
-		dma_sync_sgtable_for_device(vgdev->vdev->dev.parent,
-					    shmem->pages, DMA_TO_DEVICE);
+		dma_sync_sgtable_for_device(&vgdev->vdev->dev,
+					    bo->base.sgt, DMA_TO_DEVICE);
 
 	cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
 	memset(cmd_p, 0, sizeof(*cmd_p));
@@ -1017,11 +1016,9 @@ void virtio_gpu_cmd_transfer_to_host_3d(struct virtio_gpu_device *vgdev,
 	struct virtio_gpu_vbuffer *vbuf;
 	bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev);
 
-	if (virtio_gpu_is_shmem(bo) && use_dma_api) {
-		struct virtio_gpu_object_shmem *shmem = to_virtio_gpu_shmem(bo);
-		dma_sync_sgtable_for_device(vgdev->vdev->dev.parent,
-					    shmem->pages, DMA_TO_DEVICE);
-	}
+	if (virtio_gpu_is_shmem(bo) && use_dma_api)
+		dma_sync_sgtable_for_device(&vgdev->vdev->dev,
+					    bo->base.sgt, DMA_TO_DEVICE);
 
 	cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
 	memset(cmd_p, 0, sizeof(*cmd_p));
Signature · 1 line
2.35.1
author maintainer
#8 lore raw

PATCH v4 8/15 drm/virtio: Use dev_is_pci()

In reply to Dmitry Osipenko

Use common dev_is_pci() helper to replace the strcmp("pci") used by driver.

---
 drivers/gpu/drm/virtio/virtgpu_drv.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/virtio/virtgpu_drv.c +1 -1
index 0141b7df97ec..0035affc3e59 100644
@@ -87,7 +87,7 @@ static int virtio_gpu_probe(struct virtio_device *vdev)
 		return PTR_ERR(dev);
 	vdev->priv = dev;
 
-	if (!strcmp(vdev->dev.parent->bus->name, "pci")) {
+	if (dev_is_pci(vdev->dev.parent)) {
 		ret = virtio_gpu_pci_quirk(dev);
 		if (ret)
 			goto err_free;
Signature · 1 line
2.35.1
author maintainer
#9 lore raw

PATCH v4 9/15 drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table()

In reply to Dmitry Osipenko

drm_gem_shmem_get_sg_table() never returns NULL on error, but a ERR_PTR. Correct the doc comment which says that it returns NULL on error.

---
 drivers/gpu/drm/drm_gem_shmem_helper.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +3 -2
index 8ad0e02991ca..30ee46348a99 100644
@@ -662,7 +662,7 @@ EXPORT_SYMBOL(drm_gem_shmem_print_info);
  * drm_gem_shmem_get_pages_sgt() instead.
  *
  * Returns:
- * A pointer to the scatter/gather table of pinned pages or NULL on failure.
+ * A pointer to the scatter/gather table of pinned pages or errno on failure.
  */
 struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem)
 {
@@ -688,7 +688,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
  * drm_gem_shmem_get_sg_table() should not be directly called by drivers.
  *
  * Returns:
- * A pointer to the scatter/gather table of pinned pages or errno on failure.
+ * A pointer to the scatter/gather table of pinned pages ERR_PTR()-encoded
+ * error code on failure.
  */
 struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
 {
Signature · 1 line
2.35.1
maintainer Acked-by on v4 9/15
#10 lore raw

Re: [PATCH v4 09/15] drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table()

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 15 lines

drm_gem_shmem_get_sg_table() never returns NULL on error, but a ERR_PTR. Correct the doc comment which says that it returns NULL on error.

---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +0 -0
index 8ad0e02991ca..30ee46348a99 100644

drivers/gpu/drm/drm_gem_shmem_helper.c EXPORT_SYMBOL(drm_gem_shmem_print_info); · line 665

Quoted 5 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -1
@@ -662,7 +662,7 @@ EXPORT_SYMBOL(drm_gem_shmem_print_info);
   * drm_gem_shmem_get_pages_sgt() instead.
   *
   * Returns:
- * A pointer to the scatter/gather table of pinned pages or NULL on failure.
+ * A pointer to the scatter/gather table of pinned pages or errno on failure.

', or an ERR_PTR()-encoded errno code on failure'

drivers/gpu/drm/drm_gem_shmem_helper.c EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table); · line 691

Quoted 9 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -1
   */
  struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem)
  {
@@ -688,7 +688,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
   * drm_gem_shmem_get_sg_table() should not be directly called by drivers.
   *
   * Returns:
- * A pointer to the scatter/gather table of pinned pages or errno on failure.
+ * A pointer to the scatter/gather table of pinned pages ERR_PTR()-encoded

', or an' before ERR_PTR

With the improved grammar:

Quoted 4 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -0
+ * error code on failure.
   */
  struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
  {
Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev
author maintainer
#11 lore raw

Re: [PATCH v4 09/15] drm/shmem-helper: Correct doc-comment of drm_gem_shmem_get_sg_table()

Quoted 45 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 20 lines

drm_gem_shmem_get_sg_table() never returns NULL on error, but a ERR_PTR. Correct the doc comment which says that it returns NULL on error.

---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 5 +++--

  1 file changed, 3 insertions(+), 2 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +0 -1
index 8ad0e02991ca..30ee46348a99 100644
@@ -662,7 +662,7 @@ EXPORT_SYMBOL(drm_gem_shmem_print_info);
    * drm_gem_shmem_get_pages_sgt() instead.
    *
    * Returns:
- * A pointer to the scatter/gather table of pinned pages or NULL on
Quoted 3 lines

failure.
+ * A pointer to the scatter/gather table of pinned pages or errno on failure.

', or an ERR_PTR()-encoded errno code on failure'

Quoted 6 lines

   */
  struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem)
  {

drivers/gpu/drm/drm_gem_shmem_helper.c +0 -0
@@ -688,7 +688,8 @@ EXPORT_SYMBOL_GPL(drm_gem_shmem_get_sg_table);
    * drm_gem_shmem_get_sg_table() should not be directly called by
Quoted 7 lines

drivers.
   *
   * Returns:
- * A pointer to the scatter/gather table of pinned pages or errno on failure.
+ * A pointer to the scatter/gather table of pinned pages ERR_PTR()-encoded

', or an' before ERR_PTR

Quoted 3 lines

With the improved grammar:

Thanks, something went wrong with these comments in this patch and I haven't noticed that :)

author maintainer
#12 lore raw

PATCH v4 10/15 drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

In reply to Dmitry Osipenko

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---
 drivers/gpu/drm/drm_gem_shmem_helper.c  | 38 ++++++++++++-------------
 drivers/gpu/drm/lima/lima_gem.c         |  8 +++---
 drivers/gpu/drm/panfrost/panfrost_mmu.c | 15 ++++++----
 include/drm/drm_gem_shmem_helper.h      | 10 -------
 4 files changed, 31 insertions(+), 40 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +18 -20
index 30ee46348a99..3ecef571eff3 100644
@@ -86,8 +86,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
 	if (ret)
 		goto err_release;
 
-	mutex_init(&shmem->pages_lock);
-	mutex_init(&shmem->vmap_lock);
 	INIT_LIST_HEAD(&shmem->madv_list);
 
 	if (!private) {
@@ -157,8 +155,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
 	WARN_ON(shmem->pages_use_count);
 
 	drm_gem_object_release(obj);
-	mutex_destroy(&shmem->pages_lock);
-	mutex_destroy(&shmem->vmap_lock);
 	kfree(shmem);
 }
 EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
@@ -209,11 +205,11 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
 
 	WARN_ON(shmem->base.import_attach);
 
-	ret = mutex_lock_interruptible(&shmem->pages_lock);
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
 	if (ret)
 		return ret;
 	ret = drm_gem_shmem_get_pages_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
 }
@@ -248,9 +244,9 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
  */
 void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
 {
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 	drm_gem_shmem_put_pages_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 }
 EXPORT_SYMBOL(drm_gem_shmem_put_pages);
 
@@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
 	} else {
 		pgprot_t prot = PAGE_KERNEL;
 
-		ret = drm_gem_shmem_get_pages(shmem);
+		ret = drm_gem_shmem_get_pages_locked(shmem);
 		if (ret)
 			goto err_zero_use;
 
@@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
 {
 	int ret;
 
-	ret = mutex_lock_interruptible(&shmem->vmap_lock);
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
 	if (ret)
 		return ret;
 	ret = drm_gem_shmem_vmap_locked(shmem, map);
-	mutex_unlock(&shmem->vmap_lock);
+	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
 }
@@ -385,7 +381,7 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
 		dma_buf_vunmap(obj->import_attach->dmabuf, map);
 	} else {
 		vunmap(shmem->vaddr);
-		drm_gem_shmem_put_pages(shmem);
+		drm_gem_shmem_put_pages_locked(shmem);
 	}
 
 	shmem->vaddr = NULL;
@@ -406,9 +402,11 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
 void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
 			  struct iosys_map *map)
 {
-	mutex_lock(&shmem->vmap_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 	drm_gem_shmem_vunmap_locked(shmem, map);
-	mutex_unlock(&shmem->vmap_lock);
+	dma_resv_unlock(shmem->base.resv);
+
+	drm_gem_shmem_update_purgeable_status(shmem);
 }
 EXPORT_SYMBOL(drm_gem_shmem_vunmap);
 
@@ -442,14 +440,14 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
  */
 int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
 {
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 
 	if (shmem->madv >= 0)
 		shmem->madv = madv;
 
 	madv = shmem->madv;
 
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
 	return (madv >= 0);
 }
@@ -487,10 +485,10 @@ EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
 bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
 {
-	if (!mutex_trylock(&shmem->pages_lock))
+	if (!dma_resv_trylock(shmem->base.resv))
 		return false;
 	drm_gem_shmem_purge_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
 	return true;
 }
@@ -549,7 +547,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
 	/* We don't use vmf->pgoff since that has the fake offset */
 	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
 
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 
 	if (page_offset >= num_pages ||
 	    WARN_ON_ONCE(!shmem->pages) ||
@@ -561,7 +559,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
 		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
 	}
 
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
 }
drivers/gpu/drm/lima/lima_gem.c +4 -4
index 0f1ca0b0db49..5008f0c2428f 100644
@@ -34,7 +34,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
 
 	new_size = min(new_size, bo->base.base.size);
 
-	mutex_lock(&bo->base.pages_lock);
+	dma_resv_lock(bo->base.base.resv, NULL);
 
 	if (bo->base.pages) {
 		pages = bo->base.pages;
@@ -42,7 +42,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
 		pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
 				       sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
 		if (!pages) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(bo->base.base.resv);
 			return -ENOMEM;
 		}
 
@@ -56,13 +56,13 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
 		struct page *page = shmem_read_mapping_page(mapping, i);
 
 		if (IS_ERR(page)) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(bo->base.base.resv);
 			return PTR_ERR(page);
 		}
 		pages[i] = page;
 	}
 
-	mutex_unlock(&bo->base.pages_lock);
+	dma_resv_unlock(bo->base.base.resv);
 
 	ret = sg_alloc_table_from_pages(&sgt, pages, i, 0,
 					new_size, GFP_KERNEL);
drivers/gpu/drm/panfrost/panfrost_mmu.c +9 -6
index d3f82b26a631..404b8f67e2df 100644
@@ -424,6 +424,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
 	struct panfrost_gem_mapping *bomapping;
 	struct panfrost_gem_object *bo;
 	struct address_space *mapping;
+	struct drm_gem_object *obj;
 	pgoff_t page_offset;
 	struct sg_table *sgt;
 	struct page **pages;
@@ -446,13 +447,15 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
 	page_offset = addr >> PAGE_SHIFT;
 	page_offset -= bomapping->mmnode.start;
 
-	mutex_lock(&bo->base.pages_lock);
+	obj = &bo->base.base;
+
+	dma_resv_lock(obj->resv, NULL);
 
 	if (!bo->base.pages) {
 		bo->sgts = kvmalloc_array(bo->base.base.size / SZ_2M,
 				     sizeof(struct sg_table), GFP_KERNEL | __GFP_ZERO);
 		if (!bo->sgts) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
 			ret = -ENOMEM;
 			goto err_bo;
 		}
@@ -462,7 +465,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
 		if (!pages) {
 			kvfree(bo->sgts);
 			bo->sgts = NULL;
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
 			ret = -ENOMEM;
 			goto err_bo;
 		}
@@ -472,7 +475,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
 		pages = bo->base.pages;
 		if (pages[page_offset]) {
 			/* Pages are already mapped, bail out. */
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
 			goto out;
 		}
 	}
@@ -483,13 +486,13 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
 	for (i = page_offset; i < page_offset + NUM_FAULT_PAGES; i++) {
 		pages[i] = shmem_read_mapping_page(mapping, i);
 		if (IS_ERR(pages[i])) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
 			ret = PTR_ERR(pages[i]);
 			goto err_pages;
 		}
 	}
 
-	mutex_unlock(&bo->base.pages_lock);
+	dma_resv_unlock(obj->resv);
 
 	sgt = &bo->sgts[page_offset / (SZ_2M / PAGE_SIZE)];
 	ret = sg_alloc_table_from_pages(sgt, pages + page_offset,
include/drm/drm_gem_shmem_helper.h +0 -10
index d0a57853c188..70889533962a 100644
@@ -26,11 +26,6 @@ struct drm_gem_shmem_object {
 	 */
 	struct drm_gem_object base;
 
-	/**
-	 * @pages_lock: Protects the page table and use count
-	 */
-	struct mutex pages_lock;
-
 	/**
 	 * @pages: Page table
 	 */
@@ -79,11 +74,6 @@ struct drm_gem_shmem_object {
 	 */
 	struct sg_table *sgt;
 
-	/**
-	 * @vmap_lock: Protects the vmap address and use count
-	 */
-	struct mutex vmap_lock;
-
 	/**
 	 * @vaddr: Kernel virtual address of the backing memory
 	 */
Signature · 1 line
2.35.1
#13 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 77 lines

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---
  drivers/gpu/drm/drm_gem_shmem_helper.c  | 38 ++++++++++++-------------
  drivers/gpu/drm/lima/lima_gem.c         |  8 +++---
  drivers/gpu/drm/panfrost/panfrost_mmu.c | 15 ++++++----
  include/drm/drm_gem_shmem_helper.h      | 10 -------
  4 files changed, 31 insertions(+), 40 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +5 -9
index 30ee46348a99..3ecef571eff3 100644
@@ -86,8 +86,6 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
  	if (ret)
  		goto err_release;
 
-	mutex_init(&shmem->pages_lock);
-	mutex_init(&shmem->vmap_lock);
  	INIT_LIST_HEAD(&shmem->madv_list);
 
  	if (!private) {
@@ -157,8 +155,6 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  	WARN_ON(shmem->pages_use_count);
 
  	drm_gem_object_release(obj);
-	mutex_destroy(&shmem->pages_lock);
-	mutex_destroy(&shmem->vmap_lock);
  	kfree(shmem);
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
@@ -209,11 +205,11 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
 
  	WARN_ON(shmem->base.import_attach);
 
-	ret = mutex_lock_interruptible(&shmem->pages_lock);
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
  	if (ret)
  		return ret;
  	ret = drm_gem_shmem_get_pages_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
  }
@@ -248,9 +244,9 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
   */
  void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
  {
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_put_pages_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
  }
  EXPORT_SYMBOL(drm_gem_shmem_put_pages);
 
@@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
  	} else {
  		pgprot_t prot = PAGE_KERNEL;
 
-		ret = drm_gem_shmem_get_pages(shmem);
+		ret = drm_gem_shmem_get_pages_locked(shmem);
  		if (ret)
  			goto err_zero_use;
 

drivers/gpu/drm/drm_gem_shmem_helper.c int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem, · line 362

Quoted 8 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -1
@@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
  {
  	int ret;
 
-	ret = mutex_lock_interruptible(&shmem->vmap_lock);
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
  	if (ret)
  		return ret;
  	ret = drm_gem_shmem_vmap_locked(shmem, map);

Within drm_gem_shmem_vmap_locked(), there's a call to dma_buf_vmap() for imported pages. If the exporter side also holds/acquires the same reservation lock as our object, the whole thing can deadlock. We cannot move dma_buf_vmap() out of the CS, because we still need to increment the reference counter. I honestly don't know how to easily fix this problem. There's a TODO item about replacing these locks at [1]. As Daniel suggested this patch, we should talk to him about the issue.

Best regards
Thomas

[1] https://www.kernel.org/doc/html/latest/gpu/todo.html#move-buffer-object-locking-to-dma-resv-lock

Quoted 206 lines
drivers/gpu/drm/drm_gem_shmem_helper.c +12 -10
-	mutex_unlock(&shmem->vmap_lock);
+	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
  }
@@ -385,7 +381,7 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
  		dma_buf_vunmap(obj->import_attach->dmabuf, map);
  	} else {
  		vunmap(shmem->vaddr);
-		drm_gem_shmem_put_pages(shmem);
+		drm_gem_shmem_put_pages_locked(shmem);
  	}
 
  	shmem->vaddr = NULL;
@@ -406,9 +402,11 @@ static void drm_gem_shmem_vunmap_locked(struct drm_gem_shmem_object *shmem,
  void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
  			  struct iosys_map *map)
  {
-	mutex_lock(&shmem->vmap_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_vunmap_locked(shmem, map);
-	mutex_unlock(&shmem->vmap_lock);
+	dma_resv_unlock(shmem->base.resv);
+
+	drm_gem_shmem_update_purgeable_status(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_vunmap);
 
@@ -442,14 +440,14 @@ drm_gem_shmem_create_with_handle(struct drm_file *file_priv,
   */
  int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
  {
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 
  	if (shmem->madv >= 0)
  		shmem->madv = madv;
 
  	madv = shmem->madv;
 
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
  	return (madv >= 0);
  }
@@ -487,10 +485,10 @@ EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
  bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
  {
-	if (!mutex_trylock(&shmem->pages_lock))
+	if (!dma_resv_trylock(shmem->base.resv))
  		return false;
  	drm_gem_shmem_purge_locked(shmem);
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
  	return true;
  }
@@ -549,7 +547,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
  	/* We don't use vmf->pgoff since that has the fake offset */
  	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
 
-	mutex_lock(&shmem->pages_lock);
+	dma_resv_lock(shmem->base.resv, NULL);
 
  	if (page_offset >= num_pages ||
  	    WARN_ON_ONCE(!shmem->pages) ||
@@ -561,7 +559,7 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
  		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
  	}
 
-	mutex_unlock(&shmem->pages_lock);
+	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
  }
drivers/gpu/drm/lima/lima_gem.c +4 -4
index 0f1ca0b0db49..5008f0c2428f 100644
@@ -34,7 +34,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
 
  	new_size = min(new_size, bo->base.base.size);
 
-	mutex_lock(&bo->base.pages_lock);
+	dma_resv_lock(bo->base.base.resv, NULL);
 
  	if (bo->base.pages) {
  		pages = bo->base.pages;
@@ -42,7 +42,7 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
  		pages = kvmalloc_array(bo->base.base.size >> PAGE_SHIFT,
  				       sizeof(*pages), GFP_KERNEL | __GFP_ZERO);
  		if (!pages) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(bo->base.base.resv);
  			return -ENOMEM;
  		}
 
@@ -56,13 +56,13 @@ int lima_heap_alloc(struct lima_bo *bo, struct lima_vm *vm)
  		struct page *page = shmem_read_mapping_page(mapping, i);
 
  		if (IS_ERR(page)) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(bo->base.base.resv);
  			return PTR_ERR(page);
  		}
  		pages[i] = page;
  	}
 
-	mutex_unlock(&bo->base.pages_lock);
+	dma_resv_unlock(bo->base.base.resv);
 
  	ret = sg_alloc_table_from_pages(&sgt, pages, i, 0,
  					new_size, GFP_KERNEL);
drivers/gpu/drm/panfrost/panfrost_mmu.c +9 -6
index d3f82b26a631..404b8f67e2df 100644
@@ -424,6 +424,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
  	struct panfrost_gem_mapping *bomapping;
  	struct panfrost_gem_object *bo;
  	struct address_space *mapping;
+	struct drm_gem_object *obj;
  	pgoff_t page_offset;
  	struct sg_table *sgt;
  	struct page **pages;
@@ -446,13 +447,15 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
  	page_offset = addr >> PAGE_SHIFT;
  	page_offset -= bomapping->mmnode.start;
 
-	mutex_lock(&bo->base.pages_lock);
+	obj = &bo->base.base;
+
+	dma_resv_lock(obj->resv, NULL);
 
  	if (!bo->base.pages) {
  		bo->sgts = kvmalloc_array(bo->base.base.size / SZ_2M,
  				     sizeof(struct sg_table), GFP_KERNEL | __GFP_ZERO);
  		if (!bo->sgts) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
  			ret = -ENOMEM;
  			goto err_bo;
  		}
@@ -462,7 +465,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
  		if (!pages) {
  			kvfree(bo->sgts);
  			bo->sgts = NULL;
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
  			ret = -ENOMEM;
  			goto err_bo;
  		}
@@ -472,7 +475,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
  		pages = bo->base.pages;
  		if (pages[page_offset]) {
  			/* Pages are already mapped, bail out. */
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
  			goto out;
  		}
  	}
@@ -483,13 +486,13 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
  	for (i = page_offset; i < page_offset + NUM_FAULT_PAGES; i++) {
  		pages[i] = shmem_read_mapping_page(mapping, i);
  		if (IS_ERR(pages[i])) {
-			mutex_unlock(&bo->base.pages_lock);
+			dma_resv_unlock(obj->resv);
  			ret = PTR_ERR(pages[i]);
  			goto err_pages;
  		}
  	}
 
-	mutex_unlock(&bo->base.pages_lock);
+	dma_resv_unlock(obj->resv);
 
  	sgt = &bo->sgts[page_offset / (SZ_2M / PAGE_SIZE)];
  	ret = sg_alloc_table_from_pages(sgt, pages + page_offset,
include/drm/drm_gem_shmem_helper.h +0 -10
index d0a57853c188..70889533962a 100644
@@ -26,11 +26,6 @@ struct drm_gem_shmem_object {
  	 */
  	struct drm_gem_object base;
 
-	/**
-	 * @pages_lock: Protects the page table and use count
-	 */
-	struct mutex pages_lock;
-
  	/**
  	 * @pages: Page table
  	 */
@@ -79,11 +74,6 @@ struct drm_gem_shmem_object {
  	 */
  	struct sg_table *sgt;
 
-	/**
-	 * @vmap_lock: Protects the vmap address and use count
-	 */
-	struct mutex vmap_lock;
-
  	/**
  	 * @vaddr: Kernel virtual address of the backing memory
  	 */
Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev
author maintainer
#14 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Hello,

Quoted 16 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 13 lines

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---

...

Quoted 26 lines
Quoted 9 lines

  @@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
      } else {
          pgprot_t prot = PAGE_KERNEL;
  -        ret = drm_gem_shmem_get_pages(shmem);
+        ret = drm_gem_shmem_get_pages_locked(shmem); if (ret)
              goto err_zero_use;
  @@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct

Quoted 8 lines

drm_gem_shmem_object *shmem,
  {
      int ret;
  -    ret = mutex_lock_interruptible(&shmem->vmap_lock); +    ret = dma_resv_lock_interruptible(shmem->base.resv, NULL); if (ret)
          return ret;
      ret = drm_gem_shmem_vmap_locked(shmem, map);

Within drm_gem_shmem_vmap_locked(), there's a call to dma_buf_vmap() for imported pages. If the exporter side also holds/acquires the same reservation lock as our object, the whole thing can deadlock. We cannot move dma_buf_vmap() out of the CS, because we still need to increment the reference counter. I honestly don't know how to easily fix this problem. There's a TODO item about replacing these locks at [1]. As Daniel suggested this patch, we should talk to him about the issue.

Quoted 5 lines

Best regards
Thomas

[1]
https://www.kernel.org/doc/html/latest/gpu/todo.html#move-buffer-object-locking-to-dma-resv-lock

Indeed, good catch! Perhaps we could simply use a separate lock for the vmapping of the *imported* GEMs? The vmap_use_count is used only by vmap/vunmap, so it doesn't matter which lock is used by these functions in the case of imported GEMs since we only need to protect the vmap_use_count.

#15 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 49 lines · Dmitry Osipenko

Hello,

Quoted 16 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 13 lines

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---

...

Quoted 29 lines
Quoted 9 lines

  @@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
      } else {
          pgprot_t prot = PAGE_KERNEL;
  -        ret = drm_gem_shmem_get_pages(shmem);
+        ret = drm_gem_shmem_get_pages_locked(shmem); if (ret)
              goto err_zero_use;
  @@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct

Quoted 8 lines

drm_gem_shmem_object *shmem,
  {
      int ret;
  -    ret = mutex_lock_interruptible(&shmem->vmap_lock); +    ret = dma_resv_lock_interruptible(shmem->base.resv, NULL); if (ret)
          return ret;
      ret = drm_gem_shmem_vmap_locked(shmem, map);

Within drm_gem_shmem_vmap_locked(), there's a call to dma_buf_vmap() for imported pages. If the exporter side also holds/acquires the same reservation lock as our object, the whole thing can deadlock. We cannot move dma_buf_vmap() out of the CS, because we still need to increment the reference counter. I honestly don't know how to easily fix this problem. There's a TODO item about replacing these locks at [1]. As Daniel suggested this patch, we should talk to him about the issue.

Best regards
Thomas

Quoted 8 lines
Quoted 2 lines

[1]
https://www.kernel.org/doc/html/latest/gpu/todo.html#move-buffer-object-locking-to-dma-resv-lock

Indeed, good catch! Perhaps we could simply use a separate lock for the vmapping of the *imported* GEMs? The vmap_use_count is used only by vmap/vunmap, so it doesn't matter which lock is used by these functions in the case of imported GEMs since we only need to protect the vmap_use_count.

Apologies for the late reply, I'm flooded.

I discussed this with Daniel Stone last week in a chat, roughly what we need to do is:

1. Pick a function from shmem helpers.

2. Go through all drivers that call this, and make sure that we acquire dma_resv_lock in the top level driver entry point for this.

3. Once all driver code paths are converted, add a dma_resv_assert_held() call to that function to make sure you have it all correctly.

4. Repeate 1-3 until all shmem helper functions are converted over.

5. Ditch the 3 different shmem helper locks.

The trouble is that I forgot that vmap is a thing, so that needs more work. I think there's two approaches here:
- Do the vmap at import time. This is the trick we used to untangle the dma_resv_lock issues around dma_buf_attachment_map()
- Change the dma_buf_vmap rules that callers must hold the dma_resv_lock.
- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

The trouble here is that trying to be clever and doing the conversion just in shmem helpers wont work, because there's a lot of cases where the drivers are all kinds of inconsistent with their locking.

Adding Daniel S, also maybe for questions it'd be fastest to chat on irc? -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#16 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Hello Daniel,

27.04.2022 17:50, Daniel Vetter пишет:

Quoted 64 lines · Dmitry Osipenko
Quoted 49 lines

Hello,

Quoted 16 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 13 lines

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---

...

Quoted 29 lines
Quoted 9 lines

  @@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
      } else {
          pgprot_t prot = PAGE_KERNEL;
  -        ret = drm_gem_shmem_get_pages(shmem);
+        ret = drm_gem_shmem_get_pages_locked(shmem); if (ret)
              goto err_zero_use;
  @@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct

Quoted 8 lines

drm_gem_shmem_object *shmem,
  {
      int ret;
  -    ret = mutex_lock_interruptible(&shmem->vmap_lock); +    ret = dma_resv_lock_interruptible(shmem->base.resv, NULL); if (ret)
          return ret;
      ret = drm_gem_shmem_vmap_locked(shmem, map);

Within drm_gem_shmem_vmap_locked(), there's a call to dma_buf_vmap() for imported pages. If the exporter side also holds/acquires the same reservation lock as our object, the whole thing can deadlock. We cannot move dma_buf_vmap() out of the CS, because we still need to increment the reference counter. I honestly don't know how to easily fix this problem. There's a TODO item about replacing these locks at [1]. As Daniel suggested this patch, we should talk to him about the issue.

Best regards
Thomas

Quoted 8 lines
Quoted 2 lines

[1]
https://www.kernel.org/doc/html/latest/gpu/todo.html#move-buffer-object-locking-to-dma-resv-lock

Indeed, good catch! Perhaps we could simply use a separate lock for the vmapping of the *imported* GEMs? The vmap_use_count is used only by vmap/vunmap, so it doesn't matter which lock is used by these functions in the case of imported GEMs since we only need to protect the vmap_use_count.

Apologies for the late reply, I'm flooded.

I discussed this with Daniel Stone last week in a chat, roughly what we need to do is:

Quoted 8 lines

1. Pick a function from shmem helpers.

2. Go through all drivers that call this, and make sure that we acquire dma_resv_lock in the top level driver entry point for this.

3. Once all driver code paths are converted, add a dma_resv_assert_held() call to that function to make sure you have it all correctly.
4. Repeate 1-3 until all shmem helper functions are converted over.

Somehow I didn't notice the existence of dma_resv_assert_held(), thank you for the suggestion :)

Quoted 9 lines

5. Ditch the 3 different shmem helper locks.

The trouble is that I forgot that vmap is a thing, so that needs more work. I think there's two approaches here:
- Do the vmap at import time. This is the trick we used to untangle the dma_resv_lock issues around dma_buf_attachment_map()

- Change the dma_buf_vmap rules that callers must hold the dma_resv_lock.

I'll consider this option for v6, thank you.

I see now that you actually want to define the new rules for the dma-bufs in general and not only in the context of the DRM code, this now makes much more sense to me.

Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

Quoted 5 lines

The trouble here is that trying to be clever and doing the conversion just in shmem helpers wont work, because there's a lot of cases where the drivers are all kinds of inconsistent with their locking.

Adding Daniel S, also maybe for questions it'd be fastest to chat on irc?

My nickname is digetx on the #dri-devel channel, feel free to ping me if needed. Right now yours suggestions are clear to me, hence no extra questions.

Thank you for the review.

#17 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 86 lines · Dmitry Osipenko

Hello Daniel,

27.04.2022 17:50, Daniel Vetter пишет:

Quoted 64 lines · Dmitry Osipenko
Quoted 49 lines

Hello,

Quoted 16 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 13 lines

Replace drm_gem_shmem locks with the reservation lock to make GEM lockings more consistent.

Previously drm_gem_shmem_vmap() and drm_gem_shmem_get_pages() were protected by separate locks, now it's the same lock, but it doesn't make any difference for the current GEM SHMEM users. Only Panfrost and Lima drivers use vmap() and they do it in the slow code paths, hence there was no practical justification for the usage of separate lock in the vmap().

---

...

Quoted 29 lines
Quoted 9 lines

  @@ -310,7 +306,7 @@ static int drm_gem_shmem_vmap_locked(struct drm_gem_shmem_object *shmem,
      } else {
          pgprot_t prot = PAGE_KERNEL;
  -        ret = drm_gem_shmem_get_pages(shmem);
+        ret = drm_gem_shmem_get_pages_locked(shmem); if (ret)
              goto err_zero_use;
  @@ -360,11 +356,11 @@ int drm_gem_shmem_vmap(struct

Quoted 8 lines

drm_gem_shmem_object *shmem,
  {
      int ret;
  -    ret = mutex_lock_interruptible(&shmem->vmap_lock); +    ret = dma_resv_lock_interruptible(shmem->base.resv, NULL); if (ret)
          return ret;
      ret = drm_gem_shmem_vmap_locked(shmem, map);

Within drm_gem_shmem_vmap_locked(), there's a call to dma_buf_vmap() for imported pages. If the exporter side also holds/acquires the same reservation lock as our object, the whole thing can deadlock. We cannot move dma_buf_vmap() out of the CS, because we still need to increment the reference counter. I honestly don't know how to easily fix this problem. There's a TODO item about replacing these locks at [1]. As Daniel suggested this patch, we should talk to him about the issue.

Best regards
Thomas

Quoted 8 lines
Quoted 2 lines

[1]
https://www.kernel.org/doc/html/latest/gpu/todo.html#move-buffer-object-locking-to-dma-resv-lock

Indeed, good catch! Perhaps we could simply use a separate lock for the vmapping of the *imported* GEMs? The vmap_use_count is used only by vmap/vunmap, so it doesn't matter which lock is used by these functions in the case of imported GEMs since we only need to protect the vmap_use_count.

Apologies for the late reply, I'm flooded.

I discussed this with Daniel Stone last week in a chat, roughly what we need to do is:

Quoted 8 lines

1. Pick a function from shmem helpers.

2. Go through all drivers that call this, and make sure that we acquire dma_resv_lock in the top level driver entry point for this.

3. Once all driver code paths are converted, add a dma_resv_assert_held() call to that function to make sure you have it all correctly.
4. Repeate 1-3 until all shmem helper functions are converted over.

Somehow I didn't notice the existence of dma_resv_assert_held(), thank you for the suggestion :)

Quoted 7 lines

5. Ditch the 3 different shmem helper locks.

The trouble is that I forgot that vmap is a thing, so that needs more work. I think there's two approaches here:
- Do the vmap at import time. This is the trick we used to untangle the dma_resv_lock issues around dma_buf_attachment_map()

Quoted 7 lines
Quoted 1 line

- Change the dma_buf_vmap rules that callers must hold the dma_resv_lock.

I'll consider this option for v6, thank you.

I see now that you actually want to define the new rules for the dma-bufs in general and not only in the context of the DRM code, this now makes much more sense to me.

Yeah dma-buf is a cross driver interface, so we should try to be consistent here. We didn't do this in the past, where the only reason you didn't get lockdep splats was because you normally didn't run all possible combinations of drivers and importer/exporter relationships in one system. But that means it becomes very tricky to reason about how dma-buf really works.

Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.
-Daniel

Quoted 11 lines
Quoted 5 lines

The trouble here is that trying to be clever and doing the conversion just in shmem helpers wont work, because there's a lot of cases where the drivers are all kinds of inconsistent with their locking.

Adding Daniel S, also maybe for questions it'd be fastest to chat on irc?

My nickname is digetx on the #dri-devel channel, feel free to ping me if needed. Right now yours suggestions are clear to me, hence no extra questions.

Thank you for the review.

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#18 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

Signature · 2 lines
Best regards,
Dmitry
#19 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 27 lines · Dmitry Osipenko

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

Quoted 6 lines

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

Hm that sounds like a bridge too far? I don't think we want to start adding moveable dma-bufs for shmem, thus far at least no one asked for that. Goal here is just to streamline the locking a bit and align across all the different ways of doing buffers in drm.

Or do you mean something else and I'm just completely lost?

Quoted 6 lines

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

Huh that sounds bad, how does this happen? Pretty much all pieces of dma-buf (cpu vmap, userspace mmap, heck even dma_buf_attach) are optional or at least can fail for various reasons. So exporters not providing mmap support is fine, but importers then dying is not.

Quoted 4 lines

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

So maybe we're completely talking past each another, or coffee is not working here on my end, but I've no idea what you mean.

We do have some helpers for taking care of the dma_resv_lock dance, and Christian König has an rfc patch set to maybe unify this further. But that should be fairly orthogonal to reworking shmem (it might help a bit with reworking shmem though).
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#20 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 35 lines · Daniel Vetter
Quoted 27 lines · Dmitry Osipenko

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

Quoted 6 lines

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

Quoted 6 lines

Hm that sounds like a bridge too far? I don't think we want to start adding moveable dma-bufs for shmem, thus far at least no one asked for that. Goal here is just to streamline the locking a bit and align across all the different ways of doing buffers in drm.

Or do you mean something else and I'm just completely lost?

I'm talking about aligning DRM locks with the dma-buf locks. The problem is that the convention of dma-bufs isn't specified yet. In particular there is no convention for the mapping operations.

If we want to switch vmapping of shmem to use reservation lock, then somebody will have to hold this lock for dma_buf_vmap() and the locking convention needs to be specified firmly.

In case of dynamic buffers, we will also need to specify whether dma_buf_vmap() should imply the implicit pinning by exporter or the buffer must be pinned explicitly by importer before dma_buf_vmap() is invoked.

Perhaps I indeed shouldn't care about this for this patchset. The complete locking model of dma-bufs must be specified first.

Quoted 11 lines
Quoted 6 lines

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

Huh that sounds bad, how does this happen? Pretty much all pieces of dma-buf (cpu vmap, userspace mmap, heck even dma_buf_attach) are optional or at least can fail for various reasons. So exporters not providing mmap support is fine, but importers then dying is not.

Those drivers that die don't have userspace that uses dma-bufs extensively. I noticed it only because was looking at this code too much for the last days.

Drivers that don't die either map imported BOs properly or don't allow mapping at all.

Quoted 5 lines
Quoted 4 lines

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

Quoted 7 lines

So maybe we're completely talking past each another, or coffee is not working here on my end, but I've no idea what you mean.

We do have some helpers for taking care of the dma_resv_lock dance, and Christian König has an rfc patch set to maybe unify this further. But that should be fairly orthogonal to reworking shmem (it might help a bit with reworking shmem though).

The reservation lock itself doesn't help much shmem, IMO. It should help only in the context of dynamic dma-bufs and today we don't have a need in the dynamic shmem dma-bufs.

You were talking about making DRM locks consistent with dma-buf locks, so I thought that yours main point of making use of reservation locks for shmem is to prepare to the new locking scheme.

I wanted to try to specify the dma-buf locking convention for mapping operations because it's missing right now and it should affect how DRM should take the reservation locks, but this is not easy to do as I see now.

Could you please point at the Christian's RFC patch? He posted too many patches, can't find it :) I'm curious to take a look.

Signature · 2 lines
Best regards,
Dmitry
#21 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 43 lines · Dmitry Osipenko
Quoted 35 lines · Daniel Vetter
Quoted 27 lines · Dmitry Osipenko

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

Quoted 6 lines

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

Quoted 6 lines

Hm that sounds like a bridge too far? I don't think we want to start adding moveable dma-bufs for shmem, thus far at least no one asked for that. Goal here is just to streamline the locking a bit and align across all the different ways of doing buffers in drm.

Or do you mean something else and I'm just completely lost?

Quoted 7 lines

I'm talking about aligning DRM locks with the dma-buf locks. The problem is that the convention of dma-bufs isn't specified yet. In particular there is no convention for the mapping operations.

If we want to switch vmapping of shmem to use reservation lock, then somebody will have to hold this lock for dma_buf_vmap() and the locking convention needs to be specified firmly.

Ah yes that makes sense.

Quoted 7 lines

In case of dynamic buffers, we will also need to specify whether dma_buf_vmap() should imply the implicit pinning by exporter or the buffer must be pinned explicitly by importer before dma_buf_vmap() is invoked.

Perhaps I indeed shouldn't care about this for this patchset. The complete locking model of dma-bufs must be specified first.

Hm I thought vmap is meant to pin itself, and not rely on any other pinning done already. And from a quick look through the long call chain for amd (which is currently the only driver supporting dynamic dma-buf) that seems to be the case.

But yeah the locking isn't specificied yet, and that makes it a bit a mess :-(

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

Huh that sounds bad, how does this happen? Pretty much all pieces of dma-buf (cpu vmap, userspace mmap, heck even dma_buf_attach) are optional or at least can fail for various reasons. So exporters not providing mmap support is fine, but importers then dying is not.

Quoted 6 lines

Those drivers that die don't have userspace that uses dma-bufs extensively. I noticed it only because was looking at this code too much for the last days.

Drivers that don't die either map imported BOs properly or don't allow mapping at all.

Ah yeah driver bugs as explanation makes sense :-/

Quoted 21 lines
Quoted 5 lines
Quoted 4 lines

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

Quoted 7 lines

So maybe we're completely talking past each another, or coffee is not working here on my end, but I've no idea what you mean.

We do have some helpers for taking care of the dma_resv_lock dance, and Christian König has an rfc patch set to maybe unify this further. But that should be fairly orthogonal to reworking shmem (it might help a bit with reworking shmem though).

The reservation lock itself doesn't help much shmem, IMO. It should help only in the context of dynamic dma-bufs and today we don't have a need in the dynamic shmem dma-bufs.

You were talking about making DRM locks consistent with dma-buf locks, so I thought that yours main point of making use of reservation locks for shmem is to prepare to the new locking scheme.

Quoted 6 lines

I wanted to try to specify the dma-buf locking convention for mapping operations because it's missing right now and it should affect how DRM should take the reservation locks, but this is not easy to do as I see now.

Could you please point at the Christian's RFC patch? He posted too many patches, can't find it :) I'm curious to take a look.

https://lore.kernel.org/dri-devel/[email protected]/

Wrt this patch series here I'm wondering whether we could do an interim solution that side-steps the dma_buf_vmap mess.

- in shmem helpers pin any vmapped buffer (it's how dma-buf works too), and that pinning would be done under dma_resv_lock (like with other drivers using dma_resv_lock for bo protection)

- switch over everything else except vmap code to dma_resv_lock, but leave vmap locking as-is

- shrinker then only needs to trylock dma_resv_trylock in the shrinker, which can check for pinned buffer and that's good enough to exclude vmap'ed buffer. And it avoids mixing the vmap locking into the new shrinker code and driver interfaces.

This still leaves the vmap locking mess as-is, but I think that's a mess that's orthogonal to shrinker work.

Thoughts?
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#22 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 59 lines · Daniel Vetter
Quoted 43 lines · Dmitry Osipenko
Quoted 35 lines · Daniel Vetter
Quoted 27 lines · Dmitry Osipenko

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

Quoted 6 lines

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

Quoted 6 lines

Hm that sounds like a bridge too far? I don't think we want to start adding moveable dma-bufs for shmem, thus far at least no one asked for that. Goal here is just to streamline the locking a bit and align across all the different ways of doing buffers in drm.

Or do you mean something else and I'm just completely lost?

Quoted 7 lines

I'm talking about aligning DRM locks with the dma-buf locks. The problem is that the convention of dma-bufs isn't specified yet. In particular there is no convention for the mapping operations.

If we want to switch vmapping of shmem to use reservation lock, then somebody will have to hold this lock for dma_buf_vmap() and the locking convention needs to be specified firmly.

Ah yes that makes sense.

Quoted 5 lines

In case of dynamic buffers, we will also need to specify whether dma_buf_vmap() should imply the implicit pinning by exporter or the buffer must be pinned explicitly by importer before dma_buf_vmap() is invoked.

Quoted 7 lines
Quoted 2 lines

Perhaps I indeed shouldn't care about this for this patchset. The complete locking model of dma-bufs must be specified first.

Hm I thought vmap is meant to pin itself, and not rely on any other pinning done already. And from a quick look through the long call chain for amd (which is currently the only driver supporting dynamic dma-buf) that seems to be the case.

The vmapping behaviour is implementation-defined until it's documented explicitly, IMO.

Quoted 69 lines

But yeah the locking isn't specificied yet, and that makes it a bit a mess :-(

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

Huh that sounds bad, how does this happen? Pretty much all pieces of dma-buf (cpu vmap, userspace mmap, heck even dma_buf_attach) are optional or at least can fail for various reasons. So exporters not providing mmap support is fine, but importers then dying is not.

Quoted 6 lines

Those drivers that die don't have userspace that uses dma-bufs extensively. I noticed it only because was looking at this code too much for the last days.

Drivers that don't die either map imported BOs properly or don't allow mapping at all.

Ah yeah driver bugs as explanation makes sense :-/

Quoted 21 lines
Quoted 5 lines
Quoted 4 lines

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

Quoted 7 lines

So maybe we're completely talking past each another, or coffee is not working here on my end, but I've no idea what you mean.

We do have some helpers for taking care of the dma_resv_lock dance, and Christian König has an rfc patch set to maybe unify this further. But that should be fairly orthogonal to reworking shmem (it might help a bit with reworking shmem though).

The reservation lock itself doesn't help much shmem, IMO. It should help only in the context of dynamic dma-bufs and today we don't have a need in the dynamic shmem dma-bufs.

You were talking about making DRM locks consistent with dma-buf locks, so I thought that yours main point of making use of reservation locks for shmem is to prepare to the new locking scheme.

Quoted 6 lines

I wanted to try to specify the dma-buf locking convention for mapping operations because it's missing right now and it should affect how DRM should take the reservation locks, but this is not easy to do as I see now.

Could you please point at the Christian's RFC patch? He posted too many patches, can't find it :) I'm curious to take a look.

https://lore.kernel.org/dri-devel/[email protected]/

Wrt this patch series here I'm wondering whether we could do an interim solution that side-steps the dma_buf_vmap mess.

- in shmem helpers pin any vmapped buffer (it's how dma-buf works too), and that pinning would be done under dma_resv_lock (like with other drivers using dma_resv_lock for bo protection)

- switch over everything else except vmap code to dma_resv_lock, but leave vmap locking as-is

- shrinker then only needs to trylock dma_resv_trylock in the shrinker, which can check for pinned buffer and that's good enough to exclude vmap'ed buffer. And it avoids mixing the vmap locking into the new shrinker code and driver interfaces.

Quoted 4 lines

This still leaves the vmap locking mess as-is, but I think that's a mess that's orthogonal to shrinker work.

Thoughts?

Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

There are actually very few drivers in kernel that use dma_buf_vmap() [1], so perhaps it's not really a big deal to first try to define the locking and pinning convention for the dma-bufs? At least for dma_buf_vmap()? Let me try to do this.

[1] https://elixir.bootlin.com/linux/v5.18-rc6/C/ident/dma_buf_vmap

I envision that the extra dma_resv_locks for dma-bufs potentially may create unnecessary bottlenecks for some drivers if locking isn't really necessary by a specific driver, so drivers will need to keep this in mind. On the other hand, I don't think that any of the today's drivers will notice the additional resv locks in practice.

Signature · 2 lines
Best regards,
Dmitry
#23 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 164 lines · Dmitry Osipenko
Quoted 59 lines · Daniel Vetter
Quoted 43 lines · Dmitry Osipenko
Quoted 35 lines · Daniel Vetter
Quoted 27 lines · Dmitry Osipenko

On 5/4/22 11:21, Daniel Vetter wrote:
...

Quoted 21 lines
Quoted 9 lines
Quoted 8 lines

- Maybe also do what you suggest and keep a separate lock for this, but the fundamental issue is that this doesn't really work - if you share buffers both ways with two drivers using shmem helpers, then the ordering of this vmap_count_mutex vs dma_resv_lock is inconsistent and you can get some nice deadlocks. So not a great approach (and also the reason why we really need to get everyone to move towards dma_resv_lock as _the_ buffer object lock, since otherwise we'll never get a consistent lock nesting hierarchy).

Quoted 4 lines

The separate locks should work okay because it will be always the exporter that takes the dma_resv_lock. But I agree that it's less ideal than defining the new rules for dma-bufs since sometime you will take the resv lock and sometime not, potentially hiding bugs related to lockings.

That's the issue, some importers need to take the dma_resv_lock for dma_buf_vmap too (e.g. to first nail the buffer in place when it's a dynamic memory manager). In practice it'll work as well as what we have currently, which is similarly inconsistent, except with per-driver locks instead of shared locks from shmem helpers or dma-buf, so less obvious that things are inconsistent.

Quoted 3 lines

So yeah if it's too messy maybe the approach is to have a separate lock for vmap for now, land things, and then fix up dma_buf_vmap in a follow up series.

Quoted 6 lines

The amdgpu driver was the fist who introduced the concept of movable memory for dma-bufs. Now we want to support it for DRM SHMEM too. For both amdgpu ttm and shmem drivers we will want to hold the reservation lock when we're touching moveable buffers. The current way of denoting that dma-buf is movable is to implement the pin/unpin callbacks of the dma-buf ops, should be doable for shmem.

Quoted 6 lines

Hm that sounds like a bridge too far? I don't think we want to start adding moveable dma-bufs for shmem, thus far at least no one asked for that. Goal here is just to streamline the locking a bit and align across all the different ways of doing buffers in drm.

Or do you mean something else and I'm just completely lost?

Quoted 7 lines

I'm talking about aligning DRM locks with the dma-buf locks. The problem is that the convention of dma-bufs isn't specified yet. In particular there is no convention for the mapping operations.

If we want to switch vmapping of shmem to use reservation lock, then somebody will have to hold this lock for dma_buf_vmap() and the locking convention needs to be specified firmly.

Ah yes that makes sense.

Quoted 5 lines

In case of dynamic buffers, we will also need to specify whether dma_buf_vmap() should imply the implicit pinning by exporter or the buffer must be pinned explicitly by importer before dma_buf_vmap() is invoked.

Quoted 7 lines
Quoted 2 lines

Perhaps I indeed shouldn't care about this for this patchset. The complete locking model of dma-bufs must be specified first.

Hm I thought vmap is meant to pin itself, and not rely on any other pinning done already. And from a quick look through the long call chain for amd (which is currently the only driver supporting dynamic dma-buf) that seems to be the case.

The vmapping behaviour is implementation-defined until it's documented explicitly, IMO.

Quoted 69 lines

But yeah the locking isn't specificied yet, and that makes it a bit a mess :-(

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

A day ago I found that mapping of imported dma-bufs is broken at least for the Tegra DRM driver (and likely for others too) because driver doesn't assume that anyone will try to mmap imported buffer and just doesn't handle this case at all, so we're getting a hard lockup on touching mapped memory because we're mapping something else than the dma-buf.

Huh that sounds bad, how does this happen? Pretty much all pieces of dma-buf (cpu vmap, userspace mmap, heck even dma_buf_attach) are optional or at least can fail for various reasons. So exporters not providing mmap support is fine, but importers then dying is not.

Quoted 6 lines

Those drivers that die don't have userspace that uses dma-bufs extensively. I noticed it only because was looking at this code too much for the last days.

Drivers that don't die either map imported BOs properly or don't allow mapping at all.

Ah yeah driver bugs as explanation makes sense :-/

Quoted 21 lines
Quoted 5 lines
Quoted 4 lines

My plan is to move the dma-buf management code to the level of DRM core and make it aware of the reservation locks for the dynamic dma-bufs. This way we will get the proper locking for dma-bufs and fix mapping of imported dma-bufs for Tegra and other drivers.

Quoted 7 lines

So maybe we're completely talking past each another, or coffee is not working here on my end, but I've no idea what you mean.

We do have some helpers for taking care of the dma_resv_lock dance, and Christian König has an rfc patch set to maybe unify this further. But that should be fairly orthogonal to reworking shmem (it might help a bit with reworking shmem though).

The reservation lock itself doesn't help much shmem, IMO. It should help only in the context of dynamic dma-bufs and today we don't have a need in the dynamic shmem dma-bufs.

You were talking about making DRM locks consistent with dma-buf locks, so I thought that yours main point of making use of reservation locks for shmem is to prepare to the new locking scheme.

Quoted 6 lines

I wanted to try to specify the dma-buf locking convention for mapping operations because it's missing right now and it should affect how DRM should take the reservation locks, but this is not easy to do as I see now.

Could you please point at the Christian's RFC patch? He posted too many patches, can't find it :) I'm curious to take a look.

https://lore.kernel.org/dri-devel/[email protected]/

Wrt this patch series here I'm wondering whether we could do an interim solution that side-steps the dma_buf_vmap mess.

- in shmem helpers pin any vmapped buffer (it's how dma-buf works too), and that pinning would be done under dma_resv_lock (like with other drivers using dma_resv_lock for bo protection)

- switch over everything else except vmap code to dma_resv_lock, but leave vmap locking as-is

- shrinker then only needs to trylock dma_resv_trylock in the shrinker, which can check for pinned buffer and that's good enough to exclude vmap'ed buffer. And it avoids mixing the vmap locking into the new shrinker code and driver interfaces.

Quoted 4 lines

This still leaves the vmap locking mess as-is, but I think that's a mess that's orthogonal to shrinker work.

Thoughts?

Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Quoted 6 lines

There are actually very few drivers in kernel that use dma_buf_vmap() [1], so perhaps it's not really a big deal to first try to define the locking and pinning convention for the dma-bufs? At least for dma_buf_vmap()? Let me try to do this.

[1] https://elixir.bootlin.com/linux/v5.18-rc6/C/ident/dma_buf_vmap

Yeah looking through the code there's largely two classes of drivers that need vmap:

- display drivers that need to do cpu upload (usb, spi, i2c displays). Those generally set up the vmap at import time or when creating the drm_framebuffer object (e.g. see drm_gem_cma_prime_import_sg_table_vmap()), because that's really the only place where you can safely do that without running into locking inversion issues sooner or later

- lots of other drivers (and shmem helpers) seem to do dma_buf_vmap just because they can, but only actually ever use vmap on native objects, never on imported objects. Or at least I think so.

So maybe another approach here:

1. In general drivers which need a vmap need to set that up at dma_buf import time - the same way we pin the buffers at import time for non-dynamic importers because that's the only place where across all drivers it's ok to just take dma_resv_lock.

2. We remove the "just because we can" dma_buf_vmap support from helpers/drivers - the paths all already can cope with NULL since dma_buf_vmap can fail. vmap will only work on native objects, not imported ones.

3. If there is any driver using shmem helpers that absolutely needs vmap to also work on imported it needs a special import function (like cma helpers) which sets up the vmap at import time.

So since this is all very tricky ... what did I miss this time around?

Quoted 5 lines

I envision that the extra dma_resv_locks for dma-bufs potentially may create unnecessary bottlenecks for some drivers if locking isn't really necessary by a specific driver, so drivers will need to keep this in mind. On the other hand, I don't think that any of the today's drivers will notice the additional resv locks in practice.

Nah I don't think the extra locking will ever create a bottleneck, especially not for vmap. Generally vmap is a fallback or at least cpu operation, so at that point you're already going very slow. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#24 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Regards,
Christian.

Quoted 48 lines
Quoted 7 lines

There are actually very few drivers in kernel that use dma_buf_vmap() [1], so perhaps it's not really a big deal to first try to define the locking and pinning convention for the dma-bufs? At least for dma_buf_vmap()? Let me try to do this.

[1] https://elixir.bootlin.com/linux/v5.18-rc6/C/ident/dma_buf_vmap

Yeah looking through the code there's largely two classes of drivers that need vmap:

- display drivers that need to do cpu upload (usb, spi, i2c displays). Those generally set up the vmap at import time or when creating the drm_framebuffer object (e.g. see drm_gem_cma_prime_import_sg_table_vmap()), because that's really the only place where you can safely do that without running into locking inversion issues sooner or later

- lots of other drivers (and shmem helpers) seem to do dma_buf_vmap just because they can, but only actually ever use vmap on native objects, never on imported objects. Or at least I think so.

So maybe another approach here:

1. In general drivers which need a vmap need to set that up at dma_buf import time - the same way we pin the buffers at import time for non-dynamic importers because that's the only place where across all drivers it's ok to just take dma_resv_lock.

2. We remove the "just because we can" dma_buf_vmap support from helpers/drivers - the paths all already can cope with NULL since dma_buf_vmap can fail. vmap will only work on native objects, not imported ones.

3. If there is any driver using shmem helpers that absolutely needs vmap to also work on imported it needs a special import function (like cma helpers) which sets up the vmap at import time.

So since this is all very tricky ... what did I miss this time around?

Quoted 5 lines

I envision that the extra dma_resv_locks for dma-bufs potentially may create unnecessary bottlenecks for some drivers if locking isn't really necessary by a specific driver, so drivers will need to keep this in mind. On the other hand, I don't think that any of the today's drivers will notice the additional resv locks in practice.

Nah I don't think the extra locking will ever create a bottleneck, especially not for vmap. Generally vmap is a fallback or at least cpu operation, so at that point you're already going very slow. -Daniel

#25 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 37 lines · Christian König

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

Quoted 8 lines

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Yeah I think it's doable or at least a lot less work than the map/unmap side, which really was unfixable without just pinning at import time to avoid the locking fun. But vmap is used a lot less, and mostly by display drivers (where locking is a lot easier against dma_resv_lock), so it might be possible to pull off.
-Daniel

Quoted 53 lines

Regards,
Christian.

Quoted 48 lines
Quoted 7 lines

There are actually very few drivers in kernel that use dma_buf_vmap() [1], so perhaps it's not really a big deal to first try to define the locking and pinning convention for the dma-bufs? At least for dma_buf_vmap()? Let me try to do this.

[1] https://elixir.bootlin.com/linux/v5.18-rc6/C/ident/dma_buf_vmap

Yeah looking through the code there's largely two classes of drivers that need vmap:

- display drivers that need to do cpu upload (usb, spi, i2c displays). Those generally set up the vmap at import time or when creating the drm_framebuffer object (e.g. see drm_gem_cma_prime_import_sg_table_vmap()), because that's really the only place where you can safely do that without running into locking inversion issues sooner or later

- lots of other drivers (and shmem helpers) seem to do dma_buf_vmap just because they can, but only actually ever use vmap on native objects, never on imported objects. Or at least I think so.

So maybe another approach here:

1. In general drivers which need a vmap need to set that up at dma_buf import time - the same way we pin the buffers at import time for non-dynamic importers because that's the only place where across all drivers it's ok to just take dma_resv_lock.

2. We remove the "just because we can" dma_buf_vmap support from helpers/drivers - the paths all already can cope with NULL since dma_buf_vmap can fail. vmap will only work on native objects, not imported ones.

3. If there is any driver using shmem helpers that absolutely needs vmap to also work on imported it needs a special import function (like cma helpers) which sets up the vmap at import time.

So since this is all very tricky ... what did I miss this time around?

Quoted 5 lines

I envision that the extra dma_resv_locks for dma-bufs potentially may create unnecessary bottlenecks for some drivers if locking isn't really necessary by a specific driver, so drivers will need to keep this in mind. On the other hand, I don't think that any of the today's drivers will notice the additional resv locks in practice.

Nah I don't think the extra locking will ever create a bottleneck, especially not for vmap. Generally vmap is a fallback or at least cpu operation, so at that point you're already going very slow. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#26 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

In reply to Christian König
Quoted 37 lines · Christian König

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

Quoted 5 lines

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

I've doubts about the need to move out the dma_resv_lock() into the callers of dma_buf_vmap()..

I looked through all the dma_buf_vmap() users and neither of them interacts with dma_resv_lock() at all, i.e. nobody takes the lock in/outside of dma_buf_vmap(). Hence it's easy and more practical to make dma_buf_mmap/vmap() to take the dma_resv_lock by themselves.

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Signature · 2 lines
Best regards,
Dmitry
#27 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 44 lines · Dmitry Osipenko
Quoted 37 lines · Christian König

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

Quoted 5 lines

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

Quoted 7 lines

I've doubts about the need to move out the dma_resv_lock() into the callers of dma_buf_vmap()..

I looked through all the dma_buf_vmap() users and neither of them interacts with dma_resv_lock() at all, i.e. nobody takes the lock in/outside of dma_buf_vmap(). Hence it's easy and more practical to make dma_buf_mmap/vmap() to take the dma_resv_lock by themselves.

i915_gem_dmabuf_vmap -> i915_gem_object_pin_map_unlocked ->
i915_gem_object_lock -> dma_resv_lock

And all the ttm drivers should work similarly. So there's definitely drivers which grab dma_resv_lock from their vmap callback.

Quoted 9 lines

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Quoted 6 lines
Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Unfortunately not really :-/ Only way really is to grab a driver which needs vmap (those are mostly display drivers) on an imported buffer, and see what happens.

2nd best is liberally sprinkling lockdep annotations all over the place and throwing it at intel ci (not sure amd ci is accessible to the public) and then hoping that's good enough. Stuff like might_lock and dma_resv_assert_held.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#28 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 50 lines · Daniel Vetter
Quoted 42 lines · Dmitry Osipenko
Quoted 36 lines · Christian König

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

Quoted 5 lines

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

Quoted 7 lines

I've doubts about the need to move out the dma_resv_lock() into the callers of dma_buf_vmap()..

I looked through all the dma_buf_vmap() users and neither of them interacts with dma_resv_lock() at all, i.e. nobody takes the lock in/outside of dma_buf_vmap(). Hence it's easy and more practical to make dma_buf_mmap/vmap() to take the dma_resv_lock by themselves.

Quoted 5 lines

i915_gem_dmabuf_vmap -> i915_gem_object_pin_map_unlocked ->
i915_gem_object_lock -> dma_resv_lock

And all the ttm drivers should work similarly. So there's definitely drivers which grab dma_resv_lock from their vmap callback.

Grr.. I'll take another look.

Quoted 8 lines
Quoted 8 lines

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Are you okay with this rule?

Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Quoted 8 lines

Unfortunately not really :-/ Only way really is to grab a driver which needs vmap (those are mostly display drivers) on an imported buffer, and see what happens.

2nd best is liberally sprinkling lockdep annotations all over the place and throwing it at intel ci (not sure amd ci is accessible to the public) and then hoping that's good enough. Stuff like might_lock and dma_resv_assert_held.

Alright

Signature · 2 lines
Best regards,
Dmitry
#29 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 65 lines · Dmitry Osipenko
Quoted 50 lines · Daniel Vetter
Quoted 42 lines · Dmitry Osipenko
Quoted 36 lines · Christian König

Am 11.05.22 um 15:00 schrieb Daniel Vetter:

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines

[SNIP]
Since vmapping implies implicit pinning, we can't use a separate lock in drm_gem_shmem_vmap() because we need to protect the drm_gem_shmem_get_pages(), which is invoked by drm_gem_shmem_vmap() to pin the pages and requires the dma_resv_lock to be locked.

Hence the problem is:

1. If dma-buf importer holds the dma_resv_lock and invokes dma_buf_vmap() -> drm_gem_shmem_vmap(), then drm_gem_shmem_vmap() shall not take the dma_resv_lock.

2. Since dma-buf locking convention isn't specified, we can't assume that dma-buf importer holds the dma_resv_lock around dma_buf_vmap().

The possible solutions are:

1. Specify the dma_resv_lock convention for dma-bufs and make all drivers to follow it.

Quoted 8 lines
Quoted 7 lines

2. Make only DRM drivers to hold dma_resv_lock around dma_buf_vmap(). Other non-DRM drivers will get the lockdep warning.

3. Make drm_gem_shmem_vmap() to take the dma_resv_lock and get deadlock if dma-buf importer holds the lock.

...

Yeah this is all very annoying.

Ah, yes that topic again :)

I think we could relatively easily fix that by just defining and enforcing that the dma_resv_lock must have be taken by the caller when dma_buf_vmap() is called.

Quoted 5 lines

A two step approach should work:
1. Move the call to dma_resv_lock() into the dma_buf_vmap() function and remove all lock taking from the vmap callback implementations.
2. Move the call to dma_resv_lock() into the callers of dma_buf_vmap() and enforce that the function is called with the lock held.

Quoted 7 lines

I've doubts about the need to move out the dma_resv_lock() into the callers of dma_buf_vmap()..

I looked through all the dma_buf_vmap() users and neither of them interacts with dma_resv_lock() at all, i.e. nobody takes the lock in/outside of dma_buf_vmap(). Hence it's easy and more practical to make dma_buf_mmap/vmap() to take the dma_resv_lock by themselves.

Quoted 5 lines

i915_gem_dmabuf_vmap -> i915_gem_object_pin_map_unlocked ->
i915_gem_object_lock -> dma_resv_lock

And all the ttm drivers should work similarly. So there's definitely drivers which grab dma_resv_lock from their vmap callback.

Grr.. I'll take another look.

Quoted 6 lines
Quoted 6 lines

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Are you okay with this rule?

Yeah I think long-term it's where we want to be, just trying to find clever ways to get there.

And I think Christian agrees with that?

Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Unfortunately not really :-/ Only way really is to grab a driver which needs vmap (those are mostly display drivers) on an imported buffer, and see what happens.

Quoted 6 lines
Quoted 4 lines

2nd best is liberally sprinkling lockdep annotations all over the place and throwing it at intel ci (not sure amd ci is accessible to the public) and then hoping that's good enough. Stuff like might_lock and dma_resv_assert_held.

Alright

So throwing it at intel-gfx-ci can't hurt I think, but that only covers i915 so doesn't really help with the bigger issue of catching all the drivers.

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#30 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Am 11.05.22 um 21:05 schrieb Daniel Vetter:

Quoted 7 lines

[SNIP]

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Are you okay with this rule?

Yeah I think long-term it's where we want to be, just trying to find clever ways to get there.

And I think Christian agrees with that?

Yes, completely.

A design where most DMA-buf functions are supposed to be called with the reservation lock held is exactly what I have in mind for the long term.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Unfortunately not really :-/ Only way really is to grab a driver which needs vmap (those are mostly display drivers) on an imported buffer, and see what happens.

Quoted 8 lines
Quoted 5 lines
Quoted 4 lines

2nd best is liberally sprinkling lockdep annotations all over the place and throwing it at intel ci (not sure amd ci is accessible to the public) and then hoping that's good enough. Stuff like might_lock and dma_resv_assert_held.

Alright

So throwing it at intel-gfx-ci can't hurt I think, but that only covers i915 so doesn't really help with the bigger issue of catching all the drivers.

BTW: We have now somebody working on converting the existing libdrm_amdgpu unit tests over to igt.

Regards,
Christian.

Quoted 2 lines

Cheers, Daniel

#31 lore raw

Re: [PATCH v4 10/15] drm/shmem-helper: Take reservation lock instead of drm_gem_shmem locks

Quoted 34 lines · Christian König

Am 11.05.22 um 21:05 schrieb Daniel Vetter:

Quoted 7 lines

[SNIP]

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

It's unclear to me which driver may ever want to do the mapping under the dma_resv_lock. But if we will ever have such a driver that will need to map imported buffer under dma_resv_lock, then we could always add the dma_buf_vmap_locked() variant of the function. In this case the locking rule will sound like this:

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

"All dma-buf importers are responsible for holding the dma-reservation lock around the dmabuf->ops->mmap/vmap() calls."

Are you okay with this rule?

Yeah I think long-term it's where we want to be, just trying to find clever ways to get there.

And I think Christian agrees with that?

Yes, completely.

A design where most DMA-buf functions are supposed to be called with the reservation lock held is exactly what I have in mind for the long term.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Quoted 2 lines

It shouldn't be that hard to clean up. The last time I looked into it my main problem was that we didn't had any easy unit test for it.

Do we have any tests for dma-bufs at all? It's unclear to me what you are going to test in regards to the reservation locks, could you please clarify?

Unfortunately not really :-/ Only way really is to grab a driver which needs vmap (those are mostly display drivers) on an imported buffer, and see what happens.

2nd best is liberally sprinkling lockdep annotations all over the place and throwing it at intel ci (not sure amd ci is accessible to the public) and then hoping that's good enough. Stuff like might_lock and dma_resv_assert_held.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Alright

So throwing it at intel-gfx-ci can't hurt I think, but that only covers i915 so doesn't really help with the bigger issue of catching all the drivers.

BTW: We have now somebody working on converting the existing libdrm_amdgpu unit tests over to igt.

This sounds awesome.

/me throws a happy dance

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#32 lore raw

PATCH v4 11/15 drm/shmem-helper: Add generic memory shrinker

In reply to Dmitry Osipenko

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

---
 drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++-
 include/drm/drm_device.h               |   4 +
 include/drm/drm_gem.h                  |  35 ++
 include/drm/drm_gem_shmem_helper.h     | 105 +++-
 4 files changed, 877 insertions(+), 32 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +738 -27
index 3ecef571eff3..3838fb8d6f3a 100644
@@ -88,6 +88,13 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
 
 	INIT_LIST_HEAD(&shmem->madv_list);
 
+	/*
+	 * Eviction and purging are disabled by default, shmem user must enable
+	 * them explicitly using drm_gem_shmem_set_evictable/purgeable().
+	 */
+	shmem->eviction_disable_count = 1;
+	shmem->purging_disable_count = 1;
+
 	if (!private) {
 		/*
 		 * Our buffers are kept pinned, so allocating them
@@ -126,6 +133,107 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t
 }
 EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
 
+static void
+drm_gem_shmem_add_pages_to_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (!shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count + page_count < page_count);
+		gem_shrinker->shrinkable_count += page_count;
+		shmem->pages_shrinkable = true;
+	}
+}
+
+static void
+drm_gem_shmem_remove_pages_from_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count < page_count);
+		gem_shrinker->shrinkable_count -= page_count;
+		shmem->pages_shrinkable = false;
+	}
+}
+
+static void
+drm_gem_shmem_set_pages_state_locked(struct drm_gem_shmem_object *shmem,
+				     enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	lockdep_assert_held(&gem_shrinker->lock);
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (new_state >= DRM_GEM_SHMEM_PAGES_STATE_PINNED) {
+		if (drm_gem_shmem_is_evictable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE;
+
+		if (drm_gem_shmem_is_purgeable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+
+		if (!shmem->pages)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_UNPINNED;
+
+		if (shmem->evicted)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTED;
+	}
+
+	if (shmem->pages_state == new_state)
+		return;
+
+	switch (new_state) {
+	case DRM_GEM_SHMEM_PAGES_STATE_UNPINNED:
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_del_init(&shmem->madv_list);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PINNED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_active);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_purgeable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evictable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evicted);
+		break;
+	}
+
+	shmem->pages_state = new_state;
+}
+
+static void
+drm_gem_shmem_set_pages_state(struct drm_gem_shmem_object *shmem,
+			      enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	if (!gem_shrinker)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+	mutex_unlock(&gem_shrinker->lock);
+}
+
 /**
  * drm_gem_shmem_free - Free resources associated with a shmem GEM object
  * @shmem: shmem GEM object to free
@@ -137,6 +245,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
 {
 	struct drm_gem_object *obj = &shmem->base;
 
+	/* take out shmem GEM object from the memory shrinker */
+	drm_gem_shmem_madvise(shmem, -1);
+
 	WARN_ON(shmem->vmap_use_count);
 
 	if (obj->import_attach) {
@@ -148,7 +259,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
 			sg_free_table(shmem->sgt);
 			kfree(shmem->sgt);
 		}
-		if (shmem->pages)
+		if (shmem->pages_use_count)
 			drm_gem_shmem_put_pages(shmem);
 	}
 
@@ -159,18 +270,226 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
 }
 EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
 
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_update_pages_state_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	enum drm_gem_shmem_pages_state new_state;
+
+	if (!gem_shrinker || obj->import_attach)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (!shmem->madv)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	else if (shmem->madv > 0)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+	else
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGED;
+
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+
+	mutex_unlock(&gem_shrinker->lock);
+}
+
+static void drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+	dma_resv_lock(shmem->base.resv, NULL);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static int
+drm_gem_shmem_set_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->eviction_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->evicted) {
+		err = obj->funcs->swap_in(obj);
+		if (err)
+			return err;
+	}
+
+	shmem->eviction_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->purging_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged. Initially purging is
+ * disabled for all GEMs. Each set_pureable() call must have corresponding
+ * set_unpureable() call. If GEM was purged, then -ENOMEM is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable);
+
+static int
+drm_gem_shmem_set_unpurgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	shmem->purging_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_and_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_evictable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_purgeable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_unevictable_locked(shmem);
+	}
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unpurgeable_and_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_unevictable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_purgeable_locked(shmem);
+	}
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable_and_evictable() - Make GEM unpurgeable and
+ * 						 unevictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can't be purged and evicted. Each
+ * set_purgeable_and_evictable() call must have corresponding
+ * unpurgeable_and_unevictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable_and_evictable);
+
+/**
+ * drm_gem_shmem_set_unpurgeable_and_unevictable() - Make GEM purgeable and
+ * 						     evictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged and evicted. Each
+ * unpurgeable_and_unevictable() call must have corresponding
+ * set_purgeable_and_evictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
+	if (ret)
+		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_unpurgeable_and_unevictable);
+
+static int
+drm_gem_shmem_acquire_pages_locked(struct drm_gem_shmem_object *shmem)
 {
 	struct drm_gem_object *obj = &shmem->base;
 	struct page **pages;
 
-	if (shmem->pages_use_count++ > 0)
+	if (shmem->madv < 0) {
+		WARN_ON(shmem->pages);
+		return -ENOMEM;
+	}
+
+	if (shmem->pages) {
+		WARN_ON(!shmem->evicted);
 		return 0;
+	}
 
 	pages = drm_gem_get_pages(obj);
 	if (IS_ERR(pages)) {
 		DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
-		shmem->pages_use_count = 0;
 		return PTR_ERR(pages);
 	}
 
@@ -189,6 +508,25 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
 	return 0;
 }
 
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->pages_use_count++ > 0)
+		return 0;
+
+	err = drm_gem_shmem_acquire_pages_locked(shmem);
+	if (err) {
+		shmem->pages_use_count = 0;
+		return err;
+	}
+
+	return 0;
+}
+
 /*
  * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
  * @shmem: shmem GEM object
@@ -209,21 +547,38 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
 	if (ret)
 		return ret;
 	ret = drm_gem_shmem_get_pages_locked(shmem);
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
 	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
 }
 EXPORT_SYMBOL(drm_gem_shmem_get_pages);
 
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_get_pages_no_fail(struct drm_gem_shmem_object *shmem)
 {
-	struct drm_gem_object *obj = &shmem->base;
+	WARN_ON(shmem->base.import_attach);
 
-	if (WARN_ON_ONCE(!shmem->pages_use_count))
-		return;
+	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (--shmem->pages_use_count > 0)
+	if (drm_gem_shmem_get_pages_locked(shmem))
+		shmem->pages_use_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static void
+drm_gem_shmem_release_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	if (!shmem->pages) {
+		WARN_ON(!shmem->evicted && shmem->madv >= 0);
 		return;
+	}
 
 #ifdef CONFIG_X86
 	if (shmem->map_wc)
@@ -236,6 +591,21 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
 	shmem->pages = NULL;
 }
 
+static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (WARN_ON(!shmem->pages_use_count))
+		return;
+
+	if (--shmem->pages_use_count > 0)
+		return;
+
+	drm_gem_shmem_release_pages_locked(shmem);
+}
+
 /*
  * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
  * @shmem: shmem GEM object
@@ -246,6 +616,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
 {
 	dma_resv_lock(shmem->base.resv, NULL);
 	drm_gem_shmem_put_pages_locked(shmem);
+	drm_gem_shmem_update_pages_state_locked(shmem);
 	dma_resv_unlock(shmem->base.resv);
 }
 EXPORT_SYMBOL(drm_gem_shmem_put_pages);
@@ -262,9 +633,21 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);
  */
 int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
 {
+	int err;
+
 	WARN_ON(shmem->base.import_attach);
 
-	return drm_gem_shmem_get_pages(shmem);
+	err = drm_gem_shmem_set_unpurgeable_and_unevictable(shmem);
+	if (err)
+		return err;
+
+	err = drm_gem_shmem_get_pages(shmem);
+	if (err) {
+		drm_gem_shmem_set_purgeable_and_evictable(shmem);
+		return err;
+	}
+
+	return 0;
 }
 EXPORT_SYMBOL(drm_gem_shmem_pin);
 
@@ -280,6 +663,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
 	WARN_ON(shmem->base.import_attach);
 
 	drm_gem_shmem_put_pages(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable(shmem);
 }
 EXPORT_SYMBOL(drm_gem_shmem_unpin);
 
@@ -359,7 +743,18 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
 	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
 	if (ret)
 		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	if (ret)
+		goto unlock;
+
 	ret = drm_gem_shmem_vmap_locked(shmem, map);
+	if (ret)
+		drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	else
+		drm_gem_shmem_update_pages_state_locked(shmem);
+
+unlock:
 	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
@@ -404,9 +799,9 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
 {
 	dma_resv_lock(shmem->base.resv, NULL);
 	drm_gem_shmem_vunmap_locked(shmem, map);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
 	dma_resv_unlock(shmem->base.resv);
-
-	drm_gem_shmem_update_purgeable_status(shmem);
 }
 EXPORT_SYMBOL(drm_gem_shmem_vunmap);
 
@@ -447,29 +842,140 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
 
 	madv = shmem->madv;
 
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
 	dma_resv_unlock(shmem->base.resv);
 
 	return (madv >= 0);
 }
 EXPORT_SYMBOL(drm_gem_shmem_madvise);
 
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+/**
+ * drm_gem_shmem_swap_in_pages_locked() - Moves shmem pages back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves pages back to memory if they were previously evicted
+ * by the memory shrinker.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct sg_table *sgt;
+	int ret;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted) {
+		ret = drm_gem_shmem_acquire_pages_locked(shmem);
+		if (ret)
+			return ret;
+
+		sgt = drm_gem_shmem_get_sg_table(shmem);
+		if (IS_ERR(sgt))
+			return PTR_ERR(sgt);
+
+		ret = dma_map_sgtable(obj->dev->dev, sgt,
+				      DMA_BIDIRECTIONAL, 0);
+		if (ret) {
+			sg_free_table(sgt);
+			kfree(sgt);
+			return ret;
+		}
+
+		shmem->sgt = sgt;
+		shmem->evicted = false;
+		shmem->pages_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+
+		drm_gem_shmem_update_pages_state_locked(shmem);
+	}
+
+	return shmem->pages ? 0 : -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_pages_locked);
+
+/**
+ * drm_gem_shmem_swap_in_locked() - Moves shmem GEM back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves shmem GEM back to memory if it was previously evicted
+ * by the memory shrinker. The GEM is ready to use on success.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted)
+		return obj->funcs->swap_in(obj);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_locked);
+
+static void drm_gem_shmem_unpin_pages_locked(struct drm_gem_shmem_object *shmem)
 {
 	struct drm_gem_object *obj = &shmem->base;
 	struct drm_device *dev = obj->dev;
 
-	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	if (shmem->evicted)
+		return;
 
 	dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+	drm_gem_shmem_release_pages_locked(shmem);
+	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+
 	sg_free_table(shmem->sgt);
 	kfree(shmem->sgt);
 	shmem->sgt = NULL;
+}
 
-	drm_gem_shmem_put_pages_locked(shmem);
+/**
+ * drm_gem_shmem_evict_locked - Evict shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function unpins shmem pages, allowing them to be swapped out from
+ * memory.
+ */
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
 
-	shmem->madv = -1;
+	lockdep_assert_held(&obj->resv->lock.base);
 
-	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+	WARN_ON(!drm_gem_shmem_is_evictable(shmem));
+	WARN_ON(shmem->madv < 0);
+	WARN_ON(shmem->evicted);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
+
+	shmem->evicted = true;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_EVICTED);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_evict_locked);
+
+/**
+ * drm_gem_shmem_purge_locked - Purge shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function permanently releases shmem pages.
+ */
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	WARN_ON(shmem->madv < 0);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
 	drm_gem_free_mmap_offset(obj);
 
 	/* Our goal here is to return as much of the memory as
@@ -480,6 +986,9 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
 	shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
 
 	invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0, (loff_t)-1);
+
+	shmem->madv = -1;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_PURGED);
 }
 EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
@@ -543,22 +1052,31 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
 	vm_fault_t ret;
 	struct page *page;
 	pgoff_t page_offset;
+	bool pages_inactive;
+	int err;
 
 	/* We don't use vmf->pgoff since that has the fake offset */
 	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
 
 	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (page_offset >= num_pages ||
-	    WARN_ON_ONCE(!shmem->pages) ||
-	    shmem->madv < 0) {
+	pages_inactive = shmem->pages_state < DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	WARN_ON_ONCE(!shmem->pages ^ pages_inactive);
+
+	if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) {
 		ret = VM_FAULT_SIGBUS;
 	} else {
+		err = drm_gem_shmem_swap_in_locked(shmem);
+		if (err) {
+			ret = VM_FAULT_OOM;
+			goto unlock;
+		}
+
 		page = shmem->pages[page_offset];
 
 		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
 	}
-
+unlock:
 	dma_resv_unlock(shmem->base.resv);
 
 	return ret;
@@ -568,13 +1086,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
 {
 	struct drm_gem_object *obj = vma->vm_private_data;
 	struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
-	int ret;
-
-	WARN_ON(shmem->base.import_attach);
-
-	ret = drm_gem_shmem_get_pages(shmem);
-	WARN_ON_ONCE(ret != 0);
 
+	drm_gem_shmem_get_pages_no_fail(shmem);
 	drm_gem_vm_open(vma);
 }
 
@@ -716,6 +1229,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
 
 	shmem->sgt = sgt;
 
+	drm_gem_shmem_update_pages_state(shmem);
+
 	return sgt;
 
 err_free_sgt:
@@ -762,6 +1277,202 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
 }
 EXPORT_SYMBOL_GPL(drm_gem_shmem_prime_import_sg_table);
 
+static struct drm_gem_shmem_shrinker *
+to_drm_shrinker(struct shrinker *shrinker)
+{
+	return container_of(shrinker, struct drm_gem_shmem_shrinker, base);
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_count_objects(struct shrinker *shrinker,
+				     struct shrink_control *sc)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	u64 count = READ_ONCE(gem_shrinker->shrinkable_count);
+
+	if (count >= SHRINK_EMPTY)
+		return SHRINK_EMPTY - 1;
+
+	return count ?: SHRINK_EMPTY;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_run_objects_scan(struct shrinker *shrinker,
+					unsigned long nr_to_scan,
+					bool *lock_contention,
+					bool evict)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	struct drm_gem_shmem_object *shmem;
+	struct list_head still_in_list;
+	struct drm_gem_object *obj;
+	unsigned long freed = 0;
+	struct list_head *lru;
+	size_t page_count;
+
+	INIT_LIST_HEAD(&still_in_list);
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (evict)
+		lru = &gem_shrinker->lru_evictable;
+	else
+		lru = &gem_shrinker->lru_purgeable;
+
+	while (freed < nr_to_scan) {
+		shmem = list_first_entry_or_null(lru, typeof(*shmem), madv_list);
+		if (!shmem)
+			break;
+
+		obj = &shmem->base;
+		page_count = obj->size >> PAGE_SHIFT;
+		list_move_tail(&shmem->madv_list, &still_in_list);
+
+		if (evict && get_nr_swap_pages() < page_count)
+			continue;
+
+		/*
+		 * If it's in the process of being freed, gem_object->free()
+		 * may be blocked on lock waiting to remove it.  So just
+		 * skip it.
+		 */
+		if (!kref_get_unless_zero(&obj->refcount))
+			continue;
+
+		mutex_unlock(&gem_shrinker->lock);
+
+		/* prevent racing with job-submission code paths */
+		if (!dma_resv_trylock(obj->resv)) {
+			*lock_contention |= true;
+			goto shrinker_lock;
+		}
+
+		/* prevent racing with the dma-buf exporting */
+		if (!mutex_trylock(&gem_shrinker->dev->object_name_lock)) {
+			*lock_contention |= true;
+			goto resv_unlock;
+		}
+
+		/* check whether h/w uses this object */
+		if (!dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_WRITE))
+			goto object_name_unlock;
+
+		/* GEM may've become unpurgeable while shrinker was unlocked */
+		if (evict) {
+			if (!drm_gem_shmem_is_evictable(shmem))
+				goto object_name_unlock;
+		} else {
+			if (!drm_gem_shmem_is_purgeable(shmem))
+				goto object_name_unlock;
+		}
+
+		if (evict)
+			freed += obj->funcs->evict(obj);
+		else
+			freed += obj->funcs->purge(obj);
+object_name_unlock:
+		mutex_unlock(&gem_shrinker->dev->object_name_lock);
+resv_unlock:
+		dma_resv_unlock(obj->resv);
+shrinker_lock:
+		drm_gem_object_put(&shmem->base);
+		mutex_lock(&gem_shrinker->lock);
+	}
+
+	list_splice_tail(&still_in_list, lru);
+
+	mutex_unlock(&gem_shrinker->lock);
+
+	return freed;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_scan_objects(struct shrinker *shrinker,
+				    struct shrink_control *sc)
+{
+	unsigned long nr_to_scan = sc->nr_to_scan;
+	bool lock_contention = false;
+	unsigned long freed;
+
+	/* purge as many objects as we can */
+	freed = drm_gem_shmem_shrinker_run_objects_scan(shrinker, nr_to_scan,
+							&lock_contention, false);
+	nr_to_scan -= freed;
+
+	/* evict as many objects as we can */
+	if (freed < nr_to_scan)
+		freed += drm_gem_shmem_shrinker_run_objects_scan(shrinker,
+								 nr_to_scan,
+								 &lock_contention,
+								 true);
+
+	return (!freed && !lock_contention) ? SHRINK_STOP : freed;
+}
+
+/**
+ * drm_gem_shmem_shrinker_register() - Register shmem shrinker
+ * @dev: DRM device
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_shrinker_register(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker;
+	int err;
+
+	if (WARN_ON(dev->shmem_shrinker))
+		return -EBUSY;
+
+	gem_shrinker = kzalloc(sizeof(*gem_shrinker), GFP_KERNEL);
+	if (!gem_shrinker)
+		return -ENOMEM;
+
+	gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects;
+	gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects;
+	gem_shrinker->base.seeks = DEFAULT_SEEKS;
+	gem_shrinker->dev = dev;
+
+	INIT_LIST_HEAD(&gem_shrinker->lru_purgeable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evictable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evicted);
+	INIT_LIST_HEAD(&gem_shrinker->lru_active);
+	mutex_init(&gem_shrinker->lock);
+
+	dev->shmem_shrinker = gem_shrinker;
+
+	err = register_shrinker(&gem_shrinker->base);
+	if (err) {
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+		return err;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_register);
+
+/**
+ * drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker
+ * @dev: DRM device
+ */
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker;
+
+	if (gem_shrinker) {
+		unregister_shrinker(&gem_shrinker->base);
+		WARN_ON(!list_empty(&gem_shrinker->lru_purgeable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evictable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evicted));
+		WARN_ON(!list_empty(&gem_shrinker->lru_active));
+		mutex_destroy(&gem_shrinker->lock);
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_unregister);
+
 MODULE_DESCRIPTION("DRM SHMEM memory-management helpers");
 MODULE_IMPORT_NS(DMA_BUF);
 MODULE_LICENSE("GPL v2");
include/drm/drm_device.h +4 -0
index 9923c7a6885e..929546cad894 100644
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;
 struct drm_vma_offset_manager;
 struct drm_vram_mm;
 struct drm_fb_helper;
+struct drm_gem_shmem_shrinker;
 
 struct inode;
 
@@ -277,6 +278,9 @@ struct drm_device {
 	/** @vram_mm: VRAM MM memory manager */
 	struct drm_vram_mm *vram_mm;
 
+	/** @shmem_shrinker: SHMEM GEM memory shrinker */
+	struct drm_gem_shmem_shrinker *shmem_shrinker;
+
 	/**
 	 * @switch_power_state:
 	 *
include/drm/drm_gem.h +35 -0
index 9d7c61a122dc..390d1ce08ed3 100644
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
 	 * This is optional but necessary for mmap support.
 	 */
 	const struct vm_operations_struct *vm_ops;
+
+	/**
+	 * @purge:
+	 *
+	 * Releases the GEM object's allocated backing storage to the system.
+	 *
+	 * Returns the number of pages that have been freed by purging the GEM object.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*purge)(struct drm_gem_object *obj);
+
+	/**
+	 * @evict:
+	 *
+	 * Unpins the GEM object's allocated backing storage, allowing shmem pages
+	 * to be swapped out.
+	 *
+	 * Returns the number of pages that have been unpinned.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*evict)(struct drm_gem_object *obj);
+
+	/**
+	 * @swap_in:
+	 *
+	 * Pins GEM object's allocated backing storage if it was previously evicted,
+	 * moving swapped out pages back to memory.
+	 *
+	 * Returns 0 on success, or -errno on error.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	int (*swap_in)(struct drm_gem_object *obj);
 };
 
 /**
include/drm/drm_gem_shmem_helper.h +100 -5

149 lines not on this page · raw message

Signature · 1 line
2.35.1
#33 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 1,017 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++-
  include/drm/drm_device.h               |   4 +
  include/drm/drm_gem.h                  |  35 ++
  include/drm/drm_gem_shmem_helper.h     | 105 +++-
  4 files changed, 877 insertions(+), 32 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +738 -27
index 3ecef571eff3..3838fb8d6f3a 100644
@@ -88,6 +88,13 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
 
  	INIT_LIST_HEAD(&shmem->madv_list);
 
+	/*
+	 * Eviction and purging are disabled by default, shmem user must enable
+	 * them explicitly using drm_gem_shmem_set_evictable/purgeable().
+	 */
+	shmem->eviction_disable_count = 1;
+	shmem->purging_disable_count = 1;
+
  	if (!private) {
  		/*
  		 * Our buffers are kept pinned, so allocating them
@@ -126,6 +133,107 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
 
+static void
+drm_gem_shmem_add_pages_to_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (!shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count + page_count < page_count);
+		gem_shrinker->shrinkable_count += page_count;
+		shmem->pages_shrinkable = true;
+	}
+}
+
+static void
+drm_gem_shmem_remove_pages_from_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count < page_count);
+		gem_shrinker->shrinkable_count -= page_count;
+		shmem->pages_shrinkable = false;
+	}
+}
+
+static void
+drm_gem_shmem_set_pages_state_locked(struct drm_gem_shmem_object *shmem,
+				     enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	lockdep_assert_held(&gem_shrinker->lock);
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (new_state >= DRM_GEM_SHMEM_PAGES_STATE_PINNED) {
+		if (drm_gem_shmem_is_evictable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE;
+
+		if (drm_gem_shmem_is_purgeable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+
+		if (!shmem->pages)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_UNPINNED;
+
+		if (shmem->evicted)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTED;
+	}
+
+	if (shmem->pages_state == new_state)
+		return;
+
+	switch (new_state) {
+	case DRM_GEM_SHMEM_PAGES_STATE_UNPINNED:
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_del_init(&shmem->madv_list);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PINNED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_active);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_purgeable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evictable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evicted);
+		break;
+	}
+
+	shmem->pages_state = new_state;
+}
+
+static void
+drm_gem_shmem_set_pages_state(struct drm_gem_shmem_object *shmem,
+			      enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	if (!gem_shrinker)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+	mutex_unlock(&gem_shrinker->lock);
+}
+
  /**
   * drm_gem_shmem_free - Free resources associated with a shmem GEM object
   * @shmem: shmem GEM object to free
@@ -137,6 +245,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
 
+	/* take out shmem GEM object from the memory shrinker */
+	drm_gem_shmem_madvise(shmem, -1);
+
  	WARN_ON(shmem->vmap_use_count);
 
  	if (obj->import_attach) {
@@ -148,7 +259,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  			sg_free_table(shmem->sgt);
  			kfree(shmem->sgt);
  		}
-		if (shmem->pages)
+		if (shmem->pages_use_count)
  			drm_gem_shmem_put_pages(shmem);
  	}
 
@@ -159,18 +270,226 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
 
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_update_pages_state_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	enum drm_gem_shmem_pages_state new_state;
+
+	if (!gem_shrinker || obj->import_attach)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (!shmem->madv)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	else if (shmem->madv > 0)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+	else
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGED;
+
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+
+	mutex_unlock(&gem_shrinker->lock);
+}
+
+static void drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+	dma_resv_lock(shmem->base.resv, NULL);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static int
+drm_gem_shmem_set_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->eviction_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->evicted) {
+		err = obj->funcs->swap_in(obj);
+		if (err)
+			return err;
+	}
+
+	shmem->eviction_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->purging_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged. Initially purging is
+ * disabled for all GEMs. Each set_pureable() call must have corresponding
+ * set_unpureable() call. If GEM was purged, then -ENOMEM is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable);
+
+static int
+drm_gem_shmem_set_unpurgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	shmem->purging_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_and_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_evictable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_purgeable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_unevictable_locked(shmem);
+	}
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unpurgeable_and_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_unevictable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_purgeable_locked(shmem);
+	}
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable_and_evictable() - Make GEM unpurgeable and
+ * 						 unevictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can't be purged and evicted. Each
+ * set_purgeable_and_evictable() call must have corresponding
+ * unpurgeable_and_unevictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable_and_evictable);
+
+/**
+ * drm_gem_shmem_set_unpurgeable_and_unevictable() - Make GEM purgeable and
+ * 						     evictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged and evicted. Each
+ * unpurgeable_and_unevictable() call must have corresponding
+ * set_purgeable_and_evictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
+	if (ret)
+		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_unpurgeable_and_unevictable);
+
+static int
+drm_gem_shmem_acquire_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct page **pages;
 
-	if (shmem->pages_use_count++ > 0)
+	if (shmem->madv < 0) {
+		WARN_ON(shmem->pages);
+		return -ENOMEM;
+	}
+
+	if (shmem->pages) {
+		WARN_ON(!shmem->evicted);
  		return 0;
+	}
 
  	pages = drm_gem_get_pages(obj);
  	if (IS_ERR(pages)) {
  		DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
-		shmem->pages_use_count = 0;
  		return PTR_ERR(pages);
  	}
 
@@ -189,6 +508,25 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
  	return 0;
  }
 
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->pages_use_count++ > 0)
+		return 0;
+
+	err = drm_gem_shmem_acquire_pages_locked(shmem);
+	if (err) {
+		shmem->pages_use_count = 0;
+		return err;
+	}
+
+	return 0;
+}
+
  /*
   * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -209,21 +547,38 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
  	if (ret)
  		return ret;
  	ret = drm_gem_shmem_get_pages_locked(shmem);
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
  }
  EXPORT_SYMBOL(drm_gem_shmem_get_pages);
 
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_get_pages_no_fail(struct drm_gem_shmem_object *shmem)
  {
-	struct drm_gem_object *obj = &shmem->base;
+	WARN_ON(shmem->base.import_attach);
 
-	if (WARN_ON_ONCE(!shmem->pages_use_count))
-		return;
+	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (--shmem->pages_use_count > 0)
+	if (drm_gem_shmem_get_pages_locked(shmem))
+		shmem->pages_use_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static void
+drm_gem_shmem_release_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	if (!shmem->pages) {
+		WARN_ON(!shmem->evicted && shmem->madv >= 0);
  		return;
+	}
 
  #ifdef CONFIG_X86
  	if (shmem->map_wc)
@@ -236,6 +591,21 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
  	shmem->pages = NULL;
  }
 
+static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (WARN_ON(!shmem->pages_use_count))
+		return;
+
+	if (--shmem->pages_use_count > 0)
+		return;
+
+	drm_gem_shmem_release_pages_locked(shmem);
+}
+
  /*
   * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -246,6 +616,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_put_pages_locked(shmem);
+	drm_gem_shmem_update_pages_state_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
  }
  EXPORT_SYMBOL(drm_gem_shmem_put_pages);
@@ -262,9 +633,21 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);
   */
  int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
  {
+	int err;
+
  	WARN_ON(shmem->base.import_attach);
 
-	return drm_gem_shmem_get_pages(shmem);
+	err = drm_gem_shmem_set_unpurgeable_and_unevictable(shmem);
+	if (err)
+		return err;
+
+	err = drm_gem_shmem_get_pages(shmem);
+	if (err) {
+		drm_gem_shmem_set_purgeable_and_evictable(shmem);
+		return err;
+	}
+
+	return 0;
  }
  EXPORT_SYMBOL(drm_gem_shmem_pin);
 
@@ -280,6 +663,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
  	WARN_ON(shmem->base.import_attach);
 
  	drm_gem_shmem_put_pages(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_unpin);
 
@@ -359,7 +743,18 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
  	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
  	if (ret)
  		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	if (ret)
+		goto unlock;
+
  	ret = drm_gem_shmem_vmap_locked(shmem, map);
+	if (ret)
+		drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	else
+		drm_gem_shmem_update_pages_state_locked(shmem);
+
+unlock:
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
@@ -404,9 +799,9 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_vunmap_locked(shmem, map);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
-
-	drm_gem_shmem_update_purgeable_status(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_vunmap);
 
@@ -447,29 +842,140 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
 
  	madv = shmem->madv;
 
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
 
  	return (madv >= 0);
  }
  EXPORT_SYMBOL(drm_gem_shmem_madvise);
 
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+/**
+ * drm_gem_shmem_swap_in_pages_locked() - Moves shmem pages back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves pages back to memory if they were previously evicted
+ * by the memory shrinker.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct sg_table *sgt;
+	int ret;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted) {
+		ret = drm_gem_shmem_acquire_pages_locked(shmem);
+		if (ret)
+			return ret;
+
+		sgt = drm_gem_shmem_get_sg_table(shmem);
+		if (IS_ERR(sgt))
+			return PTR_ERR(sgt);
+
+		ret = dma_map_sgtable(obj->dev->dev, sgt,
+				      DMA_BIDIRECTIONAL, 0);
+		if (ret) {
+			sg_free_table(sgt);
+			kfree(sgt);
+			return ret;
+		}
+
+		shmem->sgt = sgt;
+		shmem->evicted = false;
+		shmem->pages_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+
+		drm_gem_shmem_update_pages_state_locked(shmem);
+	}
+
+	return shmem->pages ? 0 : -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_pages_locked);
+
+/**
+ * drm_gem_shmem_swap_in_locked() - Moves shmem GEM back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves shmem GEM back to memory if it was previously evicted
+ * by the memory shrinker. The GEM is ready to use on success.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted)
+		return obj->funcs->swap_in(obj);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_locked);
+
+static void drm_gem_shmem_unpin_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct drm_device *dev = obj->dev;
 
-	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	if (shmem->evicted)
+		return;
 
  	dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+	drm_gem_shmem_release_pages_locked(shmem);
+	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+
  	sg_free_table(shmem->sgt);
  	kfree(shmem->sgt);
  	shmem->sgt = NULL;
+}
 
-	drm_gem_shmem_put_pages_locked(shmem);
+/**
+ * drm_gem_shmem_evict_locked - Evict shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function unpins shmem pages, allowing them to be swapped out from
+ * memory.
+ */
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
 
-	shmem->madv = -1;
+	lockdep_assert_held(&obj->resv->lock.base);
 
-	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+	WARN_ON(!drm_gem_shmem_is_evictable(shmem));
+	WARN_ON(shmem->madv < 0);
+	WARN_ON(shmem->evicted);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
+
+	shmem->evicted = true;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_EVICTED);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_evict_locked);
+
+/**
+ * drm_gem_shmem_purge_locked - Purge shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function permanently releases shmem pages.
+ */
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	WARN_ON(shmem->madv < 0);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
  	drm_gem_free_mmap_offset(obj);
 
  	/* Our goal here is to return as much of the memory as
@@ -480,6 +986,9 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
  	shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
 
  	invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0, (loff_t)-1);
+
+	shmem->madv = -1;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_PURGED);
  }
  EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
@@ -543,22 +1052,31 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
  	vm_fault_t ret;
  	struct page *page;
  	pgoff_t page_offset;
+	bool pages_inactive;
+	int err;
 
  	/* We don't use vmf->pgoff since that has the fake offset */
  	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
 
  	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (page_offset >= num_pages ||
-	    WARN_ON_ONCE(!shmem->pages) ||
-	    shmem->madv < 0) {
+	pages_inactive = shmem->pages_state < DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	WARN_ON_ONCE(!shmem->pages ^ pages_inactive);
+
+	if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) {
  		ret = VM_FAULT_SIGBUS;
  	} else {
+		err = drm_gem_shmem_swap_in_locked(shmem);
+		if (err) {
+			ret = VM_FAULT_OOM;
+			goto unlock;
+		}
+
  		page = shmem->pages[page_offset];
 
  		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
  	}
-
+unlock:
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
@@ -568,13 +1086,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
  {
  	struct drm_gem_object *obj = vma->vm_private_data;
  	struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
-	int ret;
-
-	WARN_ON(shmem->base.import_attach);
-
-	ret = drm_gem_shmem_get_pages(shmem);
-	WARN_ON_ONCE(ret != 0);
 
+	drm_gem_shmem_get_pages_no_fail(shmem);
  	drm_gem_vm_open(vma);
  }
 
@@ -716,6 +1229,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
 
  	shmem->sgt = sgt;
 
+	drm_gem_shmem_update_pages_state(shmem);
+
  	return sgt;
 
  err_free_sgt:
@@ -762,6 +1277,202 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_prime_import_sg_table);
 
+static struct drm_gem_shmem_shrinker *
+to_drm_shrinker(struct shrinker *shrinker)
+{
+	return container_of(shrinker, struct drm_gem_shmem_shrinker, base);
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_count_objects(struct shrinker *shrinker,
+				     struct shrink_control *sc)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	u64 count = READ_ONCE(gem_shrinker->shrinkable_count);
+
+	if (count >= SHRINK_EMPTY)
+		return SHRINK_EMPTY - 1;
+
+	return count ?: SHRINK_EMPTY;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_run_objects_scan(struct shrinker *shrinker,
+					unsigned long nr_to_scan,
+					bool *lock_contention,
+					bool evict)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	struct drm_gem_shmem_object *shmem;
+	struct list_head still_in_list;
+	struct drm_gem_object *obj;
+	unsigned long freed = 0;
+	struct list_head *lru;
+	size_t page_count;
+
+	INIT_LIST_HEAD(&still_in_list);
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (evict)
+		lru = &gem_shrinker->lru_evictable;
+	else
+		lru = &gem_shrinker->lru_purgeable;
+
+	while (freed < nr_to_scan) {
+		shmem = list_first_entry_or_null(lru, typeof(*shmem), madv_list);
+		if (!shmem)
+			break;
+
+		obj = &shmem->base;
+		page_count = obj->size >> PAGE_SHIFT;
+		list_move_tail(&shmem->madv_list, &still_in_list);
+
+		if (evict && get_nr_swap_pages() < page_count)
+			continue;
+
+		/*
+		 * If it's in the process of being freed, gem_object->free()
+		 * may be blocked on lock waiting to remove it.  So just
+		 * skip it.
+		 */
+		if (!kref_get_unless_zero(&obj->refcount))
+			continue;
+
+		mutex_unlock(&gem_shrinker->lock);
+
+		/* prevent racing with job-submission code paths */
+		if (!dma_resv_trylock(obj->resv)) {
+			*lock_contention |= true;
+			goto shrinker_lock;
+		}
+
+		/* prevent racing with the dma-buf exporting */
+		if (!mutex_trylock(&gem_shrinker->dev->object_name_lock)) {
+			*lock_contention |= true;
+			goto resv_unlock;
+		}
+
+		/* check whether h/w uses this object */
+		if (!dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_WRITE))
+			goto object_name_unlock;
+
+		/* GEM may've become unpurgeable while shrinker was unlocked */
+		if (evict) {
+			if (!drm_gem_shmem_is_evictable(shmem))
+				goto object_name_unlock;
+		} else {
+			if (!drm_gem_shmem_is_purgeable(shmem))
+				goto object_name_unlock;
+		}
+
+		if (evict)
+			freed += obj->funcs->evict(obj);
+		else
+			freed += obj->funcs->purge(obj);
+object_name_unlock:
+		mutex_unlock(&gem_shrinker->dev->object_name_lock);
+resv_unlock:
+		dma_resv_unlock(obj->resv);
+shrinker_lock:
+		drm_gem_object_put(&shmem->base);
+		mutex_lock(&gem_shrinker->lock);
+	}
+
+	list_splice_tail(&still_in_list, lru);
+
+	mutex_unlock(&gem_shrinker->lock);
+
+	return freed;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_scan_objects(struct shrinker *shrinker,
+				    struct shrink_control *sc)
+{
+	unsigned long nr_to_scan = sc->nr_to_scan;
+	bool lock_contention = false;
+	unsigned long freed;
+
+	/* purge as many objects as we can */
+	freed = drm_gem_shmem_shrinker_run_objects_scan(shrinker, nr_to_scan,
+							&lock_contention, false);
+	nr_to_scan -= freed;
+
+	/* evict as many objects as we can */
+	if (freed < nr_to_scan)
+		freed += drm_gem_shmem_shrinker_run_objects_scan(shrinker,
+								 nr_to_scan,
+								 &lock_contention,
+								 true);
+
+	return (!freed && !lock_contention) ? SHRINK_STOP : freed;
+}
+
+/**
+ * drm_gem_shmem_shrinker_register() - Register shmem shrinker
+ * @dev: DRM device
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_shrinker_register(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker;
+	int err;
+
+	if (WARN_ON(dev->shmem_shrinker))
+		return -EBUSY;
+
+	gem_shrinker = kzalloc(sizeof(*gem_shrinker), GFP_KERNEL);
+	if (!gem_shrinker)
+		return -ENOMEM;
+
+	gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects;
+	gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects;
+	gem_shrinker->base.seeks = DEFAULT_SEEKS;
+	gem_shrinker->dev = dev;
+
+	INIT_LIST_HEAD(&gem_shrinker->lru_purgeable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evictable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evicted);
+	INIT_LIST_HEAD(&gem_shrinker->lru_active);
+	mutex_init(&gem_shrinker->lock);
+
+	dev->shmem_shrinker = gem_shrinker;
+
+	err = register_shrinker(&gem_shrinker->base);
+	if (err) {
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+		return err;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_register);
+
+/**
+ * drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker
+ * @dev: DRM device
+ */
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker;
+
+	if (gem_shrinker) {
+		unregister_shrinker(&gem_shrinker->base);
+		WARN_ON(!list_empty(&gem_shrinker->lru_purgeable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evictable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evicted));
+		WARN_ON(!list_empty(&gem_shrinker->lru_active));
+		mutex_destroy(&gem_shrinker->lock);
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_unregister);
+
  MODULE_DESCRIPTION("DRM SHMEM memory-management helpers");
  MODULE_IMPORT_NS(DMA_BUF);
  MODULE_LICENSE("GPL v2");
include/drm/drm_device.h +4 -0
index 9923c7a6885e..929546cad894 100644
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;
  struct drm_vma_offset_manager;
  struct drm_vram_mm;
  struct drm_fb_helper;
+struct drm_gem_shmem_shrinker;
 
  struct inode;
 
@@ -277,6 +278,9 @@ struct drm_device {
  	/** @vram_mm: VRAM MM memory manager */
  	struct drm_vram_mm *vram_mm;
 
+	/** @shmem_shrinker: SHMEM GEM memory shrinker */
+	struct drm_gem_shmem_shrinker *shmem_shrinker;
+
  	/**
  	 * @switch_power_state:
  	 *
include/drm/drm_gem.h +7 -0
index 9d7c61a122dc..390d1ce08ed3 100644
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
  	 * This is optional but necessary for mmap support.
  	 */
  	const struct vm_operations_struct *vm_ops;
+
+	/**
+	 * @purge:
+	 *
+	 * Releases the GEM object's allocated backing storage to the system.
+	 *
+	 * Returns the number of pages that have been freed by purging the GEM object.

include/drm/drm_gem.h struct drm_gem_object_funcs { · line 191

Quoted 10 lines
include/drm/drm_gem.h +10 -0
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*purge)(struct drm_gem_object *obj);
+
+	/**
+	 * @evict:
+	 *
+	 * Unpins the GEM object's allocated backing storage, allowing shmem pages
+	 * to be swapped out.

What's the difference to the existing unpin() callback?

Quoted 8 lines
include/drm/drm_gem.h +8 -0
+	 *
+	 * Returns the number of pages that have been unpinned.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*evict)(struct drm_gem_object *obj);
+
+	/**

include/drm/drm_gem.h struct drm_gem_object_funcs { · line 209

Quoted 10 lines
include/drm/drm_gem.h +10 -0
+	 * @swap_in:
+	 *
+	 * Pins GEM object's allocated backing storage if it was previously evicted,
+	 * moving swapped out pages back to memory.
+	 *
+	 * Returns 0 on success, or -errno on error.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	int (*swap_in)(struct drm_gem_object *obj);

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Quoted 20 lines
include/drm/drm_gem.h +0 -0
  };
 
  /**
include/drm/drm_gem_shmem_helper.h +2 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
  #include <linux/fs.h>
  #include <linux/mm.h>
  #include <linux/mutex.h>
+#include <linux/shrinker.h>
 
  #include <drm/drm_file.h>
  #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
  struct dma_buf_attachment;
  struct drm_mode_create_dumb;
  struct drm_printer;
+struct drm_device;

include/drm/drm_gem_shmem_helper.h line 29

Quoted 10 lines
include/drm/drm_gem_shmem_helper.h +8 -0
  struct sg_table;
 
+enum drm_gem_shmem_pages_state {
+	DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
+	DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+	DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+	DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly. In your patch, they also come with a big zoo of trivial helpers. None of that seems necessary AFAICT.

What's the difference between purge and evict BTW?

Quoted 42 lines
include/drm/drm_gem_shmem_helper.h +26 -1
+
  /**
   * struct drm_gem_shmem_object - GEM object backed by shmem
   */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
  	 * @madv: State for madvise
  	 *
  	 * 0 is active/inuse.
+	 * 1 is not-needed/can-be-purged
  	 * A negative value is the object is purged.
-	 * Positive values are driver specific and not used by the helpers.
  	 */
  	int madv;
 
@@ -91,6 +102,40 @@ struct drm_gem_shmem_object {
  	 * @map_wc: map object write-combined (instead of using shmem defaults).
  	 */
  	bool map_wc;
+
+	/**
+	 * @eviction_disable_count:
+	 *
+	 * The shmem pages are disallowed to be evicted by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int eviction_disable_count;
+
+	/**
+	 * @purging_disable_count:
+	 *
+	 * The shmem pages are disallowed to be purged by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int purging_disable_count;
+
+	/**
+	 * @pages_state: Current state of shmem pages. Used internally by
+	 * memory shrinker.
+	 */
+	enum drm_gem_shmem_pages_state pages_state;
+
+	/**

include/drm/drm_gem_shmem_helper.h struct drm_gem_shmem_object { · line 138

Quoted 10 lines
include/drm/drm_gem_shmem_helper.h +10 -0
+	 * @evicted: True if shmem pages were evicted by the memory shrinker.
+	 * Used internally by memory shrinker.
+	 */
+	bool evicted;
+
+	/**
+	 * @pages_shrinkable: True if shmem pages can be evicted or purged
+	 * by the memory shrinker. Used internally by memory shrinker.
+	 */
+	bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

Best regards
Thomas

Quoted 80 lines
include/drm/drm_gem_shmem_helper.h +54 -4
  };
 
  #define to_drm_gem_shmem_obj(obj) \
@@ -111,15 +156,33 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
 
  int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv);
 
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem);
+
+static inline bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+	return (shmem->madv >= 0) && !shmem->eviction_disable_count &&
+		shmem->base.funcs->evict && shmem->base.funcs->swap_in &&
+		!shmem->vmap_use_count && !shmem->base.dma_buf &&
+		!shmem->base.import_attach && shmem->sgt;
+}
+
  static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem)
  {
-	return (shmem->madv > 0) &&
-		!shmem->vmap_use_count && shmem->sgt &&
-		!shmem->base.dma_buf && !shmem->base.import_attach;
+	return (shmem->madv > 0) && !shmem->purging_disable_count &&
+		!shmem->vmap_use_count && shmem->base.funcs->purge &&
+		!shmem->base.dma_buf && !shmem->base.import_attach &&
+		shmem->sgt;
  }
 
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem);
+
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem);
+
  bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
 
  struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem);
  struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem);
@@ -262,6 +325,38 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct v
  	return drm_gem_shmem_mmap(shmem, vma);
  }
 
+/**
+ * struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs
+ */
+struct drm_gem_shmem_shrinker {
+	/** @base: Shrinker for purging shmem GEM objects */
+	struct shrinker base;
+
+	/** @lock: Protects @lru_* */
+	struct mutex lock;
+
+	/** @lru_purgeable: List of shmem GEM objects available for purging */
+	struct list_head lru_purgeable;
+
+	/** @lru_active: List of active shmem GEM objects */
+	struct list_head lru_active;
+
+	/** @lru_evictable: List of shmem GEM objects that can be evicted */
+	struct list_head lru_evictable;
+
+	/** @lru_evicted: List of evicted shmem GEM objects */
+	struct list_head lru_evicted;
+
+	/** @dev: DRM device that uses this shrinker */
+	struct drm_device *dev;
+
+	/** @shrinkable_count: Count of shmem GEM pages to be purged and evicted */
+	u64 shrinkable_count;
+};
+
+int drm_gem_shmem_shrinker_register(struct drm_device *dev);
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev);
+
  /*
   * Driver ops
   */
Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev
author maintainer
#34 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 26 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 23 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++- include/drm/drm_device.h               |   4 + include/drm/drm_gem.h                  |  35 ++ include/drm/drm_gem_shmem_helper.h     | 105 +++-

  4 files changed, 877 insertions(+), 32 deletions(-)

...

Quoted 18 lines
Quoted 18 lines
include/drm/drm_gem.h +5 -0
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
        * This is optional but necessary for mmap support.
        */
       const struct vm_operations_struct *vm_ops;
+
+    /**
+     * @purge:
+     *
+     * Releases the GEM object's allocated backing storage to the

system.
+     *
+     * Returns the number of pages that have been freed by purging the GEM object.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*purge)(struct drm_gem_object *obj); +

Quoted 8 lines
Quoted 6 lines

+    /**
+     * @evict:
+     *
+     * Unpins the GEM object's allocated backing storage, allowing shmem pages
+     * to be swapped out.

What's the difference to the existing unpin() callback?

Drivers need to do more than just unpinning pages when GEMs are evicted. Unpinning is only a part of the eviction process. I'll improve the doc-comment in v5.

For example, for VirtIO-GPU driver we need to to detach host from the guest's memory before pages are evicted [1].

[1]
https://gitlab.collabora.com/dmitry.osipenko/linux-kernel-rd/-/blob/932eb03198bce3a21353b09ab71e95f1c19b84c2/drivers/gpu/drm/virtio/virtgpu_object.c#L145

In case of Panfrost driver, we will need to remove mappings before pages are evicted.

Quoted 20 lines
Quoted 11 lines

+     *
+     * Returns the number of pages that have been unpinned. +     *
+     * This callback is used by the GEM shrinker. +     */
+    unsigned long (*evict)(struct drm_gem_object *obj); +
+    /**
+     * @swap_in:
+     *
+     * Pins GEM object's allocated backing storage if it was

Quoted 8 lines

previously evicted,
+     * moving swapped out pages back to memory. +     *
+     * Returns 0 on success, or -errno on error. +     *
+     * This callback is used by the GEM shrinker. +     */
+    int (*swap_in)(struct drm_gem_object *obj);

Quoted 2 lines

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Similarly to the unpinning, the pining of pages is only a part of what needs to be done for GPU drivers. Besides of returning pages back to memory, we also need to make them accessible to GPU and this is a driver-specific process. This why we need the additional callbacks.

Quoted 23 lines
Quoted 23 lines

  };
    /**

include/drm/drm_gem_shmem_helper.h +4 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
   #include <linux/fs.h>
   #include <linux/mm.h>
   #include <linux/mutex.h>
+#include <linux/shrinker.h>
     #include <drm/drm_file.h>
   #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
   struct dma_buf_attachment;
   struct drm_mode_create_dumb;
   struct drm_printer;
+struct drm_device;
   struct sg_table;
   +enum drm_gem_shmem_pages_state {
+    DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h line 29

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+    DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+    DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly.

I'll try to revisit this, but I was finding that it's much more difficult to follow and debug code without the explicit states.

Quoted 2 lines

In your patch, they also come with a
big zoo of trivial helpers. None of that seems necessary AFAICT.

There are couple functions which could be squashed, although this may hurt readability of the code a tad. I'll try to take another look at this for v5.

Quoted 1 line

What's the difference between purge and evict BTW?

The evicted pages are moved out from memory to a SWAP partition or file.

The purged pages are destroyed permanently.

Quoted 50 lines
Quoted 50 lines
include/drm/drm_gem_shmem_helper.h +2 -1
+
   /**
    * struct drm_gem_shmem_object - GEM object backed by shmem
    */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
        * @madv: State for madvise
        *
        * 0 is active/inuse.
+     * 1 is not-needed/can-be-purged
        * A negative value is the object is purged.
-     * Positive values are driver specific and not used by the helpers.
        */
       int madv;
   @@ -91,6 +102,40 @@ struct drm_gem_shmem_object {

       * @map_wc: map object write-combined (instead of using shmem defaults).
       */
      bool map_wc;
+
+    /**
+     * @eviction_disable_count:
+     *
+     * The shmem pages are disallowed to be evicted by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int eviction_disable_count;
+
+    /**
+     * @purging_disable_count:
+     *
+     * The shmem pages are disallowed to be purged by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int purging_disable_count;

include/drm/drm_gem_shmem_helper.h +9 -0
+
+    /**
+     * @pages_state: Current state of shmem pages. Used internally by
+     * memory shrinker.
+     */
+    enum drm_gem_shmem_pages_state pages_state;
+
+    /**
+     * @evicted: True if shmem pages were evicted by the memory

shrinker.
+     * Used internally by memory shrinker.
+     */
+    bool evicted;
+

Quoted 8 lines
Quoted 5 lines

+    /**
+     * @pages_shrinkable: True if shmem pages can be evicted or purged +     * by the memory shrinker. Used internally by memory shrinker. +     */
+    bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

When we're transitioning from "evictable" to a "purgeable" state, we must not add pages twice to the "shrinkable_count" variable. Hence this is not a state, but a variable which prevents the double accounting of the pages. Please see drm_gem_shmem_add_pages_to_shrinker() in this patch.

Perhaps something like "pages_accounted_by_shrinker" could be a better name for the variable. I'll revisit this for v5.

#35 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 37 lines · Dmitry Osipenko
Quoted 26 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 23 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++- include/drm/drm_device.h               |   4 + include/drm/drm_gem.h                  |  35 ++ include/drm/drm_gem_shmem_helper.h     | 105 +++-

  4 files changed, 877 insertions(+), 32 deletions(-)

...

Quoted 9 lines
Quoted 9 lines
include/drm/drm_gem.h +5 -0
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
        * This is optional but necessary for mmap support.
        */
       const struct vm_operations_struct *vm_ops;
+
+    /**
+     * @purge:
+     *
+     * Releases the GEM object's allocated backing storage to the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

system.
+     *
+     * Returns the number of pages that have been freed by purging the GEM object.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*purge)(struct drm_gem_object *obj);

Hm I feel like drivers shouldn't need to know the difference here?

Like shmem helpers can track what's purgeable, and for eviction/purging the driver callback should do the same?

The only difference is when we try to re-reserve the backing storage. When the object has been evicted that should suceed, but when the object is purged that will fail.

That's the difference between evict and purge for drivers?

Quoted 14 lines
Quoted 9 lines
Quoted 7 lines

+
+    /**
+     * @evict:
+     *
+     * Unpins the GEM object's allocated backing storage, allowing shmem pages
+     * to be swapped out.

What's the difference to the existing unpin() callback?

Drivers need to do more than just unpinning pages when GEMs are evicted. Unpinning is only a part of the eviction process. I'll improve the doc-comment in v5.

Quoted 8 lines

For example, for VirtIO-GPU driver we need to to detach host from the guest's memory before pages are evicted [1].

[1]
https://gitlab.collabora.com/dmitry.osipenko/linux-kernel-rd/-/blob/932eb03198bce3a21353b09ab71e95f1c19b84c2/drivers/gpu/drm/virtio/virtgpu_object.c#L145

In case of Panfrost driver, we will need to remove mappings before pages are evicted.

It might be good to align this with ttm, otoh that all works quite a bit differently for ttm since ttm supports buffer moves and a lot more fancy stuff.

I'm bringing this up since I have this fancy idea that eventually we could glue shmem helpers into ttm in some cases for managing buffers when they sit in system memory (as opposed to vram).

Quoted 20 lines
Quoted 20 lines
Quoted 19 lines

+     *
+     * Returns the number of pages that have been unpinned. +     *
+     * This callback is used by the GEM shrinker. +     */
+    unsigned long (*evict)(struct drm_gem_object *obj); +
+    /**
+     * @swap_in:
+     *
+     * Pins GEM object's allocated backing storage if it was previously evicted,
+     * moving swapped out pages back to memory.
+     *
+     * Returns 0 on success, or -errno on error.
+     *
+     * This callback is used by the GEM shrinker. +     */
+    int (*swap_in)(struct drm_gem_object *obj);

Quoted 7 lines
Quoted 2 lines

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Similarly to the unpinning, the pining of pages is only a part of what needs to be done for GPU drivers. Besides of returning pages back to memory, we also need to make them accessible to GPU and this is a driver-specific process. This why we need the additional callbacks.

This is a bit much midlayer. The way this works in ttm is you reserve all the objects you need (which makes sure they're physically available again), and then the driver goes through and makes sure the page tables are all set up again.

Once you get towards gpu vm that's really the only approach, since your swap_in has no idea for which vm it needs to restore pagetables (and restoring it for all is a bit meh).

If drivers want to optimize this they can adjust/set any tracking information from their evict callback as needed.

Quoted 33 lines
Quoted 24 lines
Quoted 24 lines

  };
    /**

include/drm/drm_gem_shmem_helper.h +4 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
   #include <linux/fs.h>
   #include <linux/mm.h>
   #include <linux/mutex.h>
+#include <linux/shrinker.h>
     #include <drm/drm_file.h>
   #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
   struct dma_buf_attachment;
   struct drm_mode_create_dumb;
   struct drm_printer;
+struct drm_device;
   struct sg_table;
   +enum drm_gem_shmem_pages_state {
+    DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h line 29

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+    DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+    DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly.

Quoted 2 lines

I'll try to revisit this, but I was finding that it's much more difficult to follow and debug code without the explicit states.

purgeable/purged needs some state, but pinned shouldn't be duplicated, so I concur here a bit.

Quoted 41 lines
Quoted 2 lines

In your patch, they also come with a
big zoo of trivial helpers. None of that seems necessary AFAICT.

There are couple functions which could be squashed, although this may hurt readability of the code a tad. I'll try to take another look at this for v5.

Quoted 1 line

What's the difference between purge and evict BTW?

The evicted pages are moved out from memory to a SWAP partition or file.

The purged pages are destroyed permanently.

Quoted 28 lines
Quoted 28 lines
include/drm/drm_gem_shmem_helper.h +2 -1
+
   /**
    * struct drm_gem_shmem_object - GEM object backed by shmem
    */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
        * @madv: State for madvise
        *
        * 0 is active/inuse.
+     * 1 is not-needed/can-be-purged
        * A negative value is the object is purged.
-     * Positive values are driver specific and not used by the helpers.
        */
       int madv;
   @@ -91,6 +102,40 @@ struct drm_gem_shmem_object {

       * @map_wc: map object write-combined (instead of using shmem defaults).
       */
      bool map_wc;
+
+    /**
+     * @eviction_disable_count:
+     *
+     * The shmem pages are disallowed to be evicted by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int eviction_disable_count;
+

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    /**
+     * @purging_disable_count:
+     *
+     * The shmem pages are disallowed to be purged by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int purging_disable_count;

What are these disable counts for?

The way purgeable works in other drivers is that userspace sets purgeable or not, and it's up to userspace to not make a mess of this.

There's also some interactions, and I guess a bunch of drivers get this wrong in funny ways. Not sure how to best clean this up.

- Once you have a shrinker/dynamic memory management you should _not_ pin pages, except when it's truly permanent like for scanout. Instead drivers should attach dma_fence to the dma_resv to denote in-flight access.

- A pinned buffer object is not allowed to be put into purgeable state, and a bo in purgeable state should not be allowed to be pinned.

- Drivers need to hold dma_resv_lock for long enough in their command submission, i.e. from the point where the reserve the buffers and make sure that mappings exists, to the point where the request is submitted to hw or drm/sched and fences are installed.

But I think a lot of current shmem users just pin as part of execbuf, so this won't work quite so well right out of the box.

Anyway with that design I don't think there should ever be a need to disable shrinking.

Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
include/drm/drm_gem_shmem_helper.h +9 -0
+
+    /**
+     * @pages_state: Current state of shmem pages. Used internally by
+     * memory shrinker.
+     */
+    enum drm_gem_shmem_pages_state pages_state;
+
+    /**
+     * @evicted: True if shmem pages were evicted by the memory

shrinker.
+     * Used internally by memory shrinker.
+     */
+    bool evicted;
+

Quoted 8 lines
Quoted 5 lines

+    /**
+     * @pages_shrinkable: True if shmem pages can be evicted or purged +     * by the memory shrinker. Used internally by memory shrinker. +     */
+    bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

Quoted 7 lines

When we're transitioning from "evictable" to a "purgeable" state, we must not add pages twice to the "shrinkable_count" variable. Hence this is not a state, but a variable which prevents the double accounting of the pages. Please see drm_gem_shmem_add_pages_to_shrinker() in this patch.

Perhaps something like "pages_accounted_by_shrinker" could be a better name for the variable. I'll revisit this for v5.

Hm not sure we need to account this? Usually the shrinker just counts when it's asked to do so, not practively maintain that count. Once you start shrinking burning cpu time is generally not too terrible. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#36 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 21 lines · Daniel Vetter
Quoted 10 lines

...

Quoted 9 lines
Quoted 9 lines
include/drm/drm_gem.h +5 -0
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
        * This is optional but necessary for mmap support.
        */
       const struct vm_operations_struct *vm_ops;
+
+    /**
+     * @purge:
+     *
+     * Releases the GEM object's allocated backing storage to the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

system.
+     *
+     * Returns the number of pages that have been freed by purging the GEM object.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*purge)(struct drm_gem_object *obj);

Hm I feel like drivers shouldn't need to know the difference here?

Quoted 8 lines

Like shmem helpers can track what's purgeable, and for eviction/purging the driver callback should do the same?

The only difference is when we try to re-reserve the backing storage. When the object has been evicted that should suceed, but when the object is purged that will fail.

That's the difference between evict and purge for drivers?

When buffer is purged, we can permanently release the backing storage and the reserved IOV space, re-using the freed space by new BOs.

When buffer is evicted, the BO's IOV should be kept reserved and the re-reservation of the backing storage should succeed.

Quoted 23 lines
Quoted 14 lines
Quoted 9 lines
Quoted 7 lines

+
+    /**
+     * @evict:
+     *
+     * Unpins the GEM object's allocated backing storage, allowing shmem pages
+     * to be swapped out.

What's the difference to the existing unpin() callback?

Drivers need to do more than just unpinning pages when GEMs are evicted. Unpinning is only a part of the eviction process. I'll improve the doc-comment in v5.

Quoted 8 lines

For example, for VirtIO-GPU driver we need to to detach host from the guest's memory before pages are evicted [1].

[1]
https://gitlab.collabora.com/dmitry.osipenko/linux-kernel-rd/-/blob/932eb03198bce3a21353b09ab71e95f1c19b84c2/drivers/gpu/drm/virtio/virtgpu_object.c#L145

In case of Panfrost driver, we will need to remove mappings before pages are evicted.

Quoted 7 lines

It might be good to align this with ttm, otoh that all works quite a bit differently for ttm since ttm supports buffer moves and a lot more fancy stuff.

I'm bringing this up since I have this fancy idea that eventually we could glue shmem helpers into ttm in some cases for managing buffers when they sit in system memory (as opposed to vram).

I'll take a look at ttm for v6.

Quoted 33 lines
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines

+     *
+     * Returns the number of pages that have been unpinned. +     *
+     * This callback is used by the GEM shrinker. +     */
+    unsigned long (*evict)(struct drm_gem_object *obj); +
+    /**
+     * @swap_in:
+     *
+     * Pins GEM object's allocated backing storage if it was previously evicted,
+     * moving swapped out pages back to memory.
+     *
+     * Returns 0 on success, or -errno on error.
+     *
+     * This callback is used by the GEM shrinker. +     */
+    int (*swap_in)(struct drm_gem_object *obj);

Quoted 7 lines
Quoted 2 lines

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Similarly to the unpinning, the pining of pages is only a part of what needs to be done for GPU drivers. Besides of returning pages back to memory, we also need to make them accessible to GPU and this is a driver-specific process. This why we need the additional callbacks.

This is a bit much midlayer. The way this works in ttm is you reserve all the objects you need (which makes sure they're physically available again), and then the driver goes through and makes sure the page tables are all set up again.

Quoted 6 lines

Once you get towards gpu vm that's really the only approach, since your swap_in has no idea for which vm it needs to restore pagetables (and restoring it for all is a bit meh).

If drivers want to optimize this they can adjust/set any tracking information from their evict callback as needed.

In practice, majority of BOs have only one mapping. Only shared BOs usually have extra mappings and shared BOs aren't evictable.

When memory pages are gone, then all the GPU mappings also should be gone. Perhaps it's indeed won't be a bad idea to move out the restoring of h/w VMs from the swap_in() and make drivers to handle the restoring by themselves, so swap_in() will be only about restoring the pages. I'll try to improve it in v6.

Quoted 81 lines
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines

  };
    /**

include/drm/drm_gem_shmem_helper.h +4 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
   #include <linux/fs.h>
   #include <linux/mm.h>
   #include <linux/mutex.h>
+#include <linux/shrinker.h>
     #include <drm/drm_file.h>
   #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
   struct dma_buf_attachment;
   struct drm_mode_create_dumb;
   struct drm_printer;
+struct drm_device;
   struct sg_table;
   +enum drm_gem_shmem_pages_state {
+    DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h line 29

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+    DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+    DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly.

Quoted 2 lines

I'll try to revisit this, but I was finding that it's much more difficult to follow and debug code without the explicit states.

purgeable/purged needs some state, but pinned shouldn't be duplicated, so I concur here a bit.

Quoted 43 lines
Quoted 2 lines

In your patch, they also come with a
big zoo of trivial helpers. None of that seems necessary AFAICT.

There are couple functions which could be squashed, although this may hurt readability of the code a tad. I'll try to take another look at this for v5.

Quoted 1 line

What's the difference between purge and evict BTW?

The evicted pages are moved out from memory to a SWAP partition or file.

The purged pages are destroyed permanently.

Quoted 30 lines
Quoted 30 lines
include/drm/drm_gem_shmem_helper.h +2 -1
+
   /**
    * struct drm_gem_shmem_object - GEM object backed by shmem
    */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
        * @madv: State for madvise
        *
        * 0 is active/inuse.
+     * 1 is not-needed/can-be-purged
        * A negative value is the object is purged.
-     * Positive values are driver specific and not used by the helpers.
        */
       int madv;
   @@ -91,6 +102,40 @@ struct drm_gem_shmem_object {

       * @map_wc: map object write-combined (instead of using shmem defaults).
       */
      bool map_wc;
+
+    /**
+     * @eviction_disable_count:
+     *
+     * The shmem pages are disallowed to be evicted by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int eviction_disable_count;
+
+    /**
+     * @purging_disable_count:

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+     *
+     * The shmem pages are disallowed to be purged by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int purging_disable_count;

What are these disable counts for?

Some of BO types should stay pinned permanently, this applies to both VirtIO and Panfrost drivers that make use of the generic shrinker in this patchset. Hence I made objects unpurgeable and unevictable by default.

Initially the idea of these counts was to allow drivers to explicitly disable purging and eviction, and do it multiple times. If driver disables eviction in two different places in the code, then we need to track the eviction-disable count.

In the v5 of this patchset drivers don't need to explicitly disable shrinking anymore, they only need to enable it. The counts are also used internally by DRM SHMEM core to track the vmappings and pinnings, but perhaps pages_use_count could be used for that instead. I'll revisit it for v6.

Quoted 10 lines

The way purgeable works in other drivers is that userspace sets purgeable or not, and it's up to userspace to not make a mess of this.

There's also some interactions, and I guess a bunch of drivers get this wrong in funny ways. Not sure how to best clean this up.

- Once you have a shrinker/dynamic memory management you should _not_ pin pages, except when it's truly permanent like for scanout. Instead drivers should attach dma_fence to the dma_resv to denote in-flight access.

By default pages are pinned when drm_gem_shmem_get_pages_sgt() is invoked by drivers during of BO creation time.

We could declare that pages_use_count=1 means the pages are allowed to be evicted and purged if shrinker is enabled. Then the further drm_gem_shmem_pin/vmap() calls will bump the pages_use_count, disallowing the eviction and purging, like you're suggesting, and we won't need the explicit counts.

Quoted 10 lines

- A pinned buffer object is not allowed to be put into purgeable state, and a bo in purgeable state should not be allowed to be pinned.

- Drivers need to hold dma_resv_lock for long enough in their command submission, i.e. from the point where the reserve the buffers and make sure that mappings exists, to the point where the request is submitted to hw or drm/sched and fences are installed.

But I think a lot of current shmem users just pin as part of execbuf, so this won't work quite so well right out of the box.

The current shmem users assume that BO is pinned permanently once it has been created.

Quoted 2 lines

Anyway with that design I don't think there should ever be a need to disable shrinking.

To me what you described mostly matches to what I did in the v5.

Quoted 28 lines
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
include/drm/drm_gem_shmem_helper.h +9 -0
+
+    /**
+     * @pages_state: Current state of shmem pages. Used internally by
+     * memory shrinker.
+     */
+    enum drm_gem_shmem_pages_state pages_state;
+
+    /**
+     * @evicted: True if shmem pages were evicted by the memory

shrinker.
+     * Used internally by memory shrinker.
+     */
+    bool evicted;
+

Quoted 8 lines
Quoted 5 lines

+    /**
+     * @pages_shrinkable: True if shmem pages can be evicted or purged +     * by the memory shrinker. Used internally by memory shrinker. +     */
+    bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

When we're transitioning from "evictable" to a "purgeable" state, we must not add pages twice to the "shrinkable_count" variable. Hence this is not a state, but a variable which prevents the double accounting of the pages. Please see drm_gem_shmem_add_pages_to_shrinker() in this patch.

Quoted 6 lines
Quoted 2 lines

Perhaps something like "pages_accounted_by_shrinker" could be a better name for the variable. I'll revisit this for v5.

Hm not sure we need to account this? Usually the shrinker just counts when it's asked to do so, not practively maintain that count. Once you start shrinking burning cpu time is generally not too terrible.

We could count pages on demand by walking up the "evictable" list, but then the shrinker's lock needs to be taken by the drm_gem_shmem_shrinker_count_objects() to protect the list.

Previously Rob Clark said that the profiling of freedreno's shrinker showed that it's worthwhile to reduce the locks as much as possible, including the case of counting shrinkable objects.

#37 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 29 lines · Dmitry Osipenko
Quoted 28 lines · Daniel Vetter
Quoted 10 lines

...

Quoted 9 lines
Quoted 9 lines
include/drm/drm_gem.h +5 -0
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
        * This is optional but necessary for mmap support.
        */
       const struct vm_operations_struct *vm_ops;
+
+    /**
+     * @purge:
+     *
+     * Releases the GEM object's allocated backing storage to the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

system.
+     *
+     * Returns the number of pages that have been freed by purging the GEM object.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*purge)(struct drm_gem_object *obj);

Hm I feel like drivers shouldn't need to know the difference here?

Like shmem helpers can track what's purgeable, and for eviction/purging the driver callback should do the same?

The only difference is when we try to re-reserve the backing storage. When the object has been evicted that should suceed, but when the object is purged that will fail.

Quoted 7 lines
Quoted 1 line

That's the difference between evict and purge for drivers?

When buffer is purged, we can permanently release the backing storage and the reserved IOV space, re-using the freed space by new BOs.

When buffer is evicted, the BO's IOV should be kept reserved and the re-reservation of the backing storage should succeed.

Yeah but what's the difference for driver callbacks? In both cases the driver callback needs to tear down gpu mappings and pagetables. The only difference happens after that in the shmem helper: For purge we ditch the shmem object, for evict we keep it. Drivers shouldn't need to care about that difference, hence why the two callbacks?

Quoted 71 lines
Quoted 24 lines
Quoted 15 lines
Quoted 10 lines
Quoted 8 lines

+
+    /**
+     * @evict:
+     *
+     * Unpins the GEM object's allocated backing storage, allowing shmem pages
+     * to be swapped out.

What's the difference to the existing unpin() callback?

Drivers need to do more than just unpinning pages when GEMs are evicted. Unpinning is only a part of the eviction process. I'll improve the doc-comment in v5.

Quoted 8 lines

For example, for VirtIO-GPU driver we need to to detach host from the guest's memory before pages are evicted [1].

[1]
https://gitlab.collabora.com/dmitry.osipenko/linux-kernel-rd/-/blob/932eb03198bce3a21353b09ab71e95f1c19b84c2/drivers/gpu/drm/virtio/virtgpu_object.c#L145

In case of Panfrost driver, we will need to remove mappings before pages are evicted.

Quoted 7 lines

It might be good to align this with ttm, otoh that all works quite a bit differently for ttm since ttm supports buffer moves and a lot more fancy stuff.

I'm bringing this up since I have this fancy idea that eventually we could glue shmem helpers into ttm in some cases for managing buffers when they sit in system memory (as opposed to vram).

I'll take a look at ttm for v6.

Quoted 37 lines
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines

+     *
+     * Returns the number of pages that have been unpinned. +     *
+     * This callback is used by the GEM shrinker. +     */
+    unsigned long (*evict)(struct drm_gem_object *obj); +
+    /**
+     * @swap_in:
+     *
+     * Pins GEM object's allocated backing storage if it was previously evicted,
+     * moving swapped out pages back to memory.
+     *
+     * Returns 0 on success, or -errno on error.
+     *
+     * This callback is used by the GEM shrinker. +     */
+    int (*swap_in)(struct drm_gem_object *obj);

Quoted 7 lines
Quoted 2 lines

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Similarly to the unpinning, the pining of pages is only a part of what needs to be done for GPU drivers. Besides of returning pages back to memory, we also need to make them accessible to GPU and this is a driver-specific process. This why we need the additional callbacks.

This is a bit much midlayer. The way this works in ttm is you reserve all the objects you need (which makes sure they're physically available again), and then the driver goes through and makes sure the page tables are all set up again.

Once you get towards gpu vm that's really the only approach, since your swap_in has no idea for which vm it needs to restore pagetables (and restoring it for all is a bit meh).

Quoted 5 lines
Quoted 2 lines

If drivers want to optimize this they can adjust/set any tracking information from their evict callback as needed.

In practice, majority of BOs have only one mapping. Only shared BOs usually have extra mappings and shared BOs aren't evictable.

That seems like a fairly arbitrary limitations, and e.g. i915 doesn't have this limitation, and also has a shrinker. I don't think it should be built into the design.

Quoted 5 lines

When memory pages are gone, then all the GPU mappings also should be gone. Perhaps it's indeed won't be a bad idea to move out the restoring of h/w VMs from the swap_in() and make drivers to handle the restoring by themselves, so swap_in() will be only about restoring the pages. I'll try to improve it in v6.

Sounds good.
-Daniel

Quoted 187 lines
Quoted 82 lines
Quoted 33 lines
Quoted 24 lines
Quoted 24 lines

  };
    /**

include/drm/drm_gem_shmem_helper.h +4 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
   #include <linux/fs.h>
   #include <linux/mm.h>
   #include <linux/mutex.h>
+#include <linux/shrinker.h>
     #include <drm/drm_file.h>
   #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
   struct dma_buf_attachment;
   struct drm_mode_create_dumb;
   struct drm_printer;
+struct drm_device;
   struct sg_table;
   +enum drm_gem_shmem_pages_state {
+    DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h line 29

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+    DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+    DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly.

Quoted 2 lines

I'll try to revisit this, but I was finding that it's much more difficult to follow and debug code without the explicit states.

purgeable/purged needs some state, but pinned shouldn't be duplicated, so I concur here a bit.

Quoted 43 lines
Quoted 2 lines

In your patch, they also come with a
big zoo of trivial helpers. None of that seems necessary AFAICT.

There are couple functions which could be squashed, although this may hurt readability of the code a tad. I'll try to take another look at this for v5.

Quoted 1 line

What's the difference between purge and evict BTW?

The evicted pages are moved out from memory to a SWAP partition or file.

The purged pages are destroyed permanently.

Quoted 30 lines
Quoted 30 lines
include/drm/drm_gem_shmem_helper.h +2 -1
+
   /**
    * struct drm_gem_shmem_object - GEM object backed by shmem
    */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
        * @madv: State for madvise
        *
        * 0 is active/inuse.
+     * 1 is not-needed/can-be-purged
        * A negative value is the object is purged.
-     * Positive values are driver specific and not used by the helpers.
        */
       int madv;
   @@ -91,6 +102,40 @@ struct drm_gem_shmem_object {

       * @map_wc: map object write-combined (instead of using shmem defaults).
       */
      bool map_wc;
+
+    /**
+     * @eviction_disable_count:
+     *
+     * The shmem pages are disallowed to be evicted by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int eviction_disable_count;
+
+    /**
+     * @purging_disable_count:

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+     *
+     * The shmem pages are disallowed to be purged by the memory shrinker
+     * while count is non-zero. Used internally by memory shrinker. +     */
+    unsigned int purging_disable_count;

What are these disable counts for?

Some of BO types should stay pinned permanently, this applies to both VirtIO and Panfrost drivers that make use of the generic shrinker in this patchset. Hence I made objects unpurgeable and unevictable by default.

Initially the idea of these counts was to allow drivers to explicitly disable purging and eviction, and do it multiple times. If driver disables eviction in two different places in the code, then we need to track the eviction-disable count.

In the v5 of this patchset drivers don't need to explicitly disable shrinking anymore, they only need to enable it. The counts are also used internally by DRM SHMEM core to track the vmappings and pinnings, but perhaps pages_use_count could be used for that instead. I'll revisit it for v6.

Quoted 10 lines

The way purgeable works in other drivers is that userspace sets purgeable or not, and it's up to userspace to not make a mess of this.

There's also some interactions, and I guess a bunch of drivers get this wrong in funny ways. Not sure how to best clean this up.

- Once you have a shrinker/dynamic memory management you should _not_ pin pages, except when it's truly permanent like for scanout. Instead drivers should attach dma_fence to the dma_resv to denote in-flight access.

By default pages are pinned when drm_gem_shmem_get_pages_sgt() is invoked by drivers during of BO creation time.

We could declare that pages_use_count=1 means the pages are allowed to be evicted and purged if shrinker is enabled. Then the further drm_gem_shmem_pin/vmap() calls will bump the pages_use_count, disallowing the eviction and purging, like you're suggesting, and we won't need the explicit counts.

Quoted 10 lines

- A pinned buffer object is not allowed to be put into purgeable state, and a bo in purgeable state should not be allowed to be pinned.

- Drivers need to hold dma_resv_lock for long enough in their command submission, i.e. from the point where the reserve the buffers and make sure that mappings exists, to the point where the request is submitted to hw or drm/sched and fences are installed.

But I think a lot of current shmem users just pin as part of execbuf, so this won't work quite so well right out of the box.

The current shmem users assume that BO is pinned permanently once it has been created.

Quoted 2 lines

Anyway with that design I don't think there should ever be a need to disable shrinking.

To me what you described mostly matches to what I did in the v5.

Quoted 28 lines
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
include/drm/drm_gem_shmem_helper.h +9 -0
+
+    /**
+     * @pages_state: Current state of shmem pages. Used internally by
+     * memory shrinker.
+     */
+    enum drm_gem_shmem_pages_state pages_state;
+
+    /**
+     * @evicted: True if shmem pages were evicted by the memory

shrinker.
+     * Used internally by memory shrinker.
+     */
+    bool evicted;
+

Quoted 8 lines
Quoted 5 lines

+    /**
+     * @pages_shrinkable: True if shmem pages can be evicted or purged +     * by the memory shrinker. Used internally by memory shrinker. +     */
+    bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

When we're transitioning from "evictable" to a "purgeable" state, we must not add pages twice to the "shrinkable_count" variable. Hence this is not a state, but a variable which prevents the double accounting of the pages. Please see drm_gem_shmem_add_pages_to_shrinker() in this patch.

Quoted 6 lines
Quoted 2 lines

Perhaps something like "pages_accounted_by_shrinker" could be a better name for the variable. I'll revisit this for v5.

Hm not sure we need to account this? Usually the shrinker just counts when it's asked to do so, not practively maintain that count. Once you start shrinking burning cpu time is generally not too terrible.

We could count pages on demand by walking up the "evictable" list, but then the shrinker's lock needs to be taken by the drm_gem_shmem_shrinker_count_objects() to protect the list.

Previously Rob Clark said that the profiling of freedreno's shrinker showed that it's worthwhile to reduce the locks as much as possible, including the case of counting shrinkable objects.

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
#38 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

In reply to Dmitry Osipenko
Quoted 299 lines · Dmitry Osipenko
Quoted 21 lines · Daniel Vetter
Quoted 10 lines

...

Quoted 9 lines
Quoted 9 lines
include/drm/drm_gem.h +5 -0
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
       * This is optional but necessary for mmap support.
       */
      const struct vm_operations_struct *vm_ops;
+
+    /**
+     * @purge:
+     *
+     * Releases the GEM object's allocated backing storage to the
Quoted 8 lines
Quoted 8 lines

include/drm/drm_gem.h struct drm_gem_object_funcs { · line 185

Quoted 8 lines

system.

(quoted diff) +2 -0
+     *
+     * Returns the number of pages that have been freed by purging

the GEM object.

include/drm/drm_gem.h +4 -0
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*purge)(struct drm_gem_object *obj);

Hm I feel like drivers shouldn't need to know the difference here?

Quoted 8 lines

Like shmem helpers can track what's purgeable, and for eviction/purging the driver callback should do the same?

The only difference is when we try to re-reserve the backing storage. When the object has been evicted that should suceed, but when the object is purged that will fail.

That's the difference between evict and purge for drivers?

When buffer is purged, we can permanently release the backing storage and the reserved IOV space, re-using the freed space by new BOs.

When buffer is evicted, the BO's IOV should be kept reserved and the re-reservation of the backing storage should succeed.

Quoted 23 lines
Quoted 14 lines
Quoted 9 lines
Quoted 7 lines
include/drm/drm_gem.h +5 -0
+
+    /**
+     * @evict:
+     *
+     * Unpins the GEM object's allocated backing storage, allowing

shmem pages
+ * to be swapped out.

What's the difference to the existing unpin() callback?

Drivers need to do more than just unpinning pages when GEMs are evicted. Unpinning is only a part of the eviction process. I'll improve the doc-comment in v5.

Quoted 8 lines

For example, for VirtIO-GPU driver we need to to detach host from the guest's memory before pages are evicted [1].

[1]
https://gitlab.collabora.com/dmitry.osipenko/linux-kernel-rd/-/blob/932eb03198bce3a21353b09ab71e95f1c19b84c2/drivers/gpu/drm/virtio/virtgpu_object.c#L145

In case of Panfrost driver, we will need to remove mappings before pages are evicted.

Quoted 7 lines

It might be good to align this with ttm, otoh that all works quite a bit differently for ttm since ttm supports buffer moves and a lot more fancy stuff.

I'm bringing this up since I have this fancy idea that eventually we could glue shmem helpers into ttm in some cases for managing buffers when they sit in system memory (as opposed to vram).

I'll take a look at ttm for v6.

Quoted 33 lines
Quoted 20 lines
Quoted 20 lines
Quoted 19 lines
include/drm/drm_gem.h +11 -0
+     *
+     * Returns the number of pages that have been unpinned.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    unsigned long (*evict)(struct drm_gem_object *obj);
+
+    /**
+     * @swap_in:
+     *
+     * Pins GEM object's allocated backing storage if it was

previously evicted,

include/drm/drm_gem.h +7 -0
+     * moving swapped out pages back to memory.
+     *
+     * Returns 0 on success, or -errno on error.
+     *
+     * This callback is used by the GEM shrinker.
+     */
+    int (*swap_in)(struct drm_gem_object *obj);
Quoted 7 lines
Quoted 2 lines

Why do you need swap_in()? This can be done on-demand as part of a pin or vmap operation.

Similarly to the unpinning, the pining of pages is only a part of what needs to be done for GPU drivers. Besides of returning pages back to memory, we also need to make them accessible to GPU and this is a driver-specific process. This why we need the additional callbacks.

This is a bit much midlayer. The way this works in ttm is you reserve all the objects you need (which makes sure they're physically available again), and then the driver goes through and makes sure the page tables are all set up again.

Quoted 6 lines

Once you get towards gpu vm that's really the only approach, since your swap_in has no idea for which vm it needs to restore pagetables (and restoring it for all is a bit meh).

If drivers want to optimize this they can adjust/set any tracking information from their evict callback as needed.

In practice, majority of BOs have only one mapping. Only shared BOs usually have extra mappings and shared BOs aren't evictable.

When memory pages are gone, then all the GPU mappings also should be gone. Perhaps it's indeed won't be a bad idea to move out the restoring of h/w VMs from the swap_in() and make drivers to handle the restoring by themselves, so swap_in() will be only about restoring the pages. I'll try to improve it in v6.

Quoted 81 lines
Quoted 32 lines
Quoted 23 lines
Quoted 23 lines
  };
    /**
include/drm/drm_gem_shmem_helper.h +4 -0
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
  #include <linux/fs.h>
  #include <linux/mm.h>
  #include <linux/mutex.h>
+#include <linux/shrinker.h>
    #include <drm/drm_file.h>
  #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
  struct dma_buf_attachment;
  struct drm_mode_create_dumb;
  struct drm_printer;
+struct drm_device;
  struct sg_table;
  +enum drm_gem_shmem_pages_state {
+    DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h line 29

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+    DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+    DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+    DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};

These states can be detected by looking at the vmap and pin refcounts. No need to store them explicitly.

Quoted 2 lines

I'll try to revisit this, but I was finding that it's much more difficult to follow and debug code without the explicit states.

purgeable/purged needs some state, but pinned shouldn't be duplicated, so I concur here a bit.

Quoted 43 lines
Quoted 2 lines

In your patch, they also come with a
big zoo of trivial helpers. None of that seems necessary AFAICT.

There are couple functions which could be squashed, although this may hurt readability of the code a tad. I'll try to take another look at this for v5.

Quoted 1 line

What's the difference between purge and evict BTW?

The evicted pages are moved out from memory to a SWAP partition or file.

The purged pages are destroyed permanently.

Quoted 30 lines
Quoted 30 lines
include/drm/drm_gem_shmem_helper.h +2 -1
+
  /**
   * struct drm_gem_shmem_object - GEM object backed by shmem
   */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
       * @madv: State for madvise
       *
       * 0 is active/inuse.
+     * 1 is not-needed/can-be-purged
       * A negative value is the object is purged.
-     * Positive values are driver specific and not used by the helpers.
       */
      int madv;
  @@ -91,6 +102,40 @@ struct drm_gem_shmem_object {

* @map_wc: map object write-combined (instead of using shmem
defaults).

       */
      bool map_wc;
include/drm/drm_gem_shmem_helper.h +5 -0
+
+    /**
+     * @eviction_disable_count:
+     *
+     * The shmem pages are disallowed to be evicted by the memory

shrinker

include/drm/drm_gem_shmem_helper.h +6 -0
+     * while count is non-zero. Used internally by memory shrinker.
+     */
+    unsigned int eviction_disable_count;
+
+    /**
+     * @purging_disable_count:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

include/drm/drm_gem_shmem_helper.h struct drm_gem_shmem_object { · line 120

Quoted 6 lines
include/drm/drm_gem_shmem_helper.h +2 -0
+     *
+     * The shmem pages are disallowed to be purged by the memory

shrinker

include/drm/drm_gem_shmem_helper.h +3 -0
+     * while count is non-zero. Used internally by memory shrinker.
+     */
+    unsigned int purging_disable_count;

What are these disable counts for?

Some of BO types should stay pinned permanently, this applies to both VirtIO and Panfrost drivers that make use of the generic shrinker in this patchset. Hence I made objects unpurgeable and unevictable by default.

Initially the idea of these counts was to allow drivers to explicitly disable purging and eviction, and do it multiple times. If driver disables eviction in two different places in the code, then we need to track the eviction-disable count.

In the v5 of this patchset drivers don't need to explicitly disable shrinking anymore, they only need to enable it. The counts are also used internally by DRM SHMEM core to track the vmappings and pinnings, but perhaps pages_use_count could be used for that instead. I'll revisit it for v6.

Quoted 10 lines

The way purgeable works in other drivers is that userspace sets purgeable or not, and it's up to userspace to not make a mess of this.

There's also some interactions, and I guess a bunch of drivers get this wrong in funny ways. Not sure how to best clean this up.

- Once you have a shrinker/dynamic memory management you should _not_ pin pages, except when it's truly permanent like for scanout. Instead drivers should attach dma_fence to the dma_resv to denote in-flight access.

By default pages are pinned when drm_gem_shmem_get_pages_sgt() is invoked by drivers during of BO creation time.

We could declare that pages_use_count=1 means the pages are allowed to be evicted and purged if shrinker is enabled. Then the further drm_gem_shmem_pin/vmap() calls will bump the pages_use_count, disallowing the eviction and purging, like you're suggesting, and we won't need the explicit counts.

Quoted 10 lines

- A pinned buffer object is not allowed to be put into purgeable state, and a bo in purgeable state should not be allowed to be pinned.

- Drivers need to hold dma_resv_lock for long enough in their command submission, i.e. from the point where the reserve the buffers and make sure that mappings exists, to the point where the request is submitted to hw or drm/sched and fences are installed.

But I think a lot of current shmem users just pin as part of execbuf, so this won't work quite so well right out of the box.

The current shmem users assume that BO is pinned permanently once it has been created.

Quoted 2 lines

Anyway with that design I don't think there should ever be a need to disable shrinking.

To me what you described mostly matches to what I did in the v5.

Quoted 28 lines
Quoted 28 lines
Quoted 14 lines
Quoted 14 lines
include/drm/drm_gem_shmem_helper.h +9 -0
+
+    /**
+     * @pages_state: Current state of shmem pages. Used internally by
+     * memory shrinker.
+     */
+    enum drm_gem_shmem_pages_state pages_state;
+
+    /**
+     * @evicted: True if shmem pages were evicted by the memory

shrinker.

include/drm/drm_gem_shmem_helper.h +4 -0
+     * Used internally by memory shrinker.
+     */
+    bool evicted;
+
Quoted 8 lines

include/drm/drm_gem_shmem_helper.h struct drm_gem_shmem_object { · line 138

Quoted 5 lines
include/drm/drm_gem_shmem_helper.h +5 -0
+    /**
+     * @pages_shrinkable: True if shmem pages can be evicted or purged
+     * by the memory shrinker. Used internally by memory shrinker.
+     */
+    bool pages_shrinkable;

As commented before, this state can be foundby looking at existing fields. No need to store it separately.

When we're transitioning from "evictable" to a "purgeable" state, we must not add pages twice to the "shrinkable_count" variable. Hence this is not a state, but a variable which prevents the double accounting of the pages. Please see drm_gem_shmem_add_pages_to_shrinker() in this patch.

Quoted 6 lines
Quoted 2 lines

Perhaps something like "pages_accounted_by_shrinker" could be a better name for the variable. I'll revisit this for v5.

Hm not sure we need to account this? Usually the shrinker just counts when it's asked to do so, not practively maintain that count. Once you start shrinking burning cpu time is generally not too terrible.

Quoted 7 lines

We could count pages on demand by walking up the "evictable" list, but then the shrinker's lock needs to be taken by the drm_gem_shmem_shrinker_count_objects() to protect the list.

Previously Rob Clark said that the profiling of freedreno's shrinker showed that it's worthwhile to reduce the locks as much as possible, including the case of counting shrinkable objects.

Sorry I missed this earlier, but danvet is giving some bad advice here ;-)

You *really* need count_objects() to be lockless and fast, ie. no list iteration. It doesn't have to return the "perfect" value, so it is ok if it is racy / not-atomic / etc. Otherwise you will have bad system performance issues when you start hitting do_shrink_slab() on many threads at once.

BR,
-R

#39 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

In reply to Dmitry Osipenko

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered. Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

Best regards
Thomas

Quoted 1,193 lines
---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++-
  include/drm/drm_device.h               |   4 +
  include/drm/drm_gem.h                  |  35 ++
  include/drm/drm_gem_shmem_helper.h     | 105 +++-
  4 files changed, 877 insertions(+), 32 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +738 -27
index 3ecef571eff3..3838fb8d6f3a 100644
@@ -88,6 +88,13 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
 
  	INIT_LIST_HEAD(&shmem->madv_list);
 
+	/*
+	 * Eviction and purging are disabled by default, shmem user must enable
+	 * them explicitly using drm_gem_shmem_set_evictable/purgeable().
+	 */
+	shmem->eviction_disable_count = 1;
+	shmem->purging_disable_count = 1;
+
  	if (!private) {
  		/*
  		 * Our buffers are kept pinned, so allocating them
@@ -126,6 +133,107 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
 
+static void
+drm_gem_shmem_add_pages_to_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (!shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count + page_count < page_count);
+		gem_shrinker->shrinkable_count += page_count;
+		shmem->pages_shrinkable = true;
+	}
+}
+
+static void
+drm_gem_shmem_remove_pages_from_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count < page_count);
+		gem_shrinker->shrinkable_count -= page_count;
+		shmem->pages_shrinkable = false;
+	}
+}
+
+static void
+drm_gem_shmem_set_pages_state_locked(struct drm_gem_shmem_object *shmem,
+				     enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	lockdep_assert_held(&gem_shrinker->lock);
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (new_state >= DRM_GEM_SHMEM_PAGES_STATE_PINNED) {
+		if (drm_gem_shmem_is_evictable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE;
+
+		if (drm_gem_shmem_is_purgeable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+
+		if (!shmem->pages)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_UNPINNED;
+
+		if (shmem->evicted)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTED;
+	}
+
+	if (shmem->pages_state == new_state)
+		return;
+
+	switch (new_state) {
+	case DRM_GEM_SHMEM_PAGES_STATE_UNPINNED:
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_del_init(&shmem->madv_list);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PINNED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_active);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_purgeable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evictable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evicted);
+		break;
+	}
+
+	shmem->pages_state = new_state;
+}
+
+static void
+drm_gem_shmem_set_pages_state(struct drm_gem_shmem_object *shmem,
+			      enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	if (!gem_shrinker)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+	mutex_unlock(&gem_shrinker->lock);
+}
+
  /**
   * drm_gem_shmem_free - Free resources associated with a shmem GEM object
   * @shmem: shmem GEM object to free
@@ -137,6 +245,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
 
+	/* take out shmem GEM object from the memory shrinker */
+	drm_gem_shmem_madvise(shmem, -1);
+
  	WARN_ON(shmem->vmap_use_count);
 
  	if (obj->import_attach) {
@@ -148,7 +259,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  			sg_free_table(shmem->sgt);
  			kfree(shmem->sgt);
  		}
-		if (shmem->pages)
+		if (shmem->pages_use_count)
  			drm_gem_shmem_put_pages(shmem);
  	}
 
@@ -159,18 +270,226 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
 
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_update_pages_state_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	enum drm_gem_shmem_pages_state new_state;
+
+	if (!gem_shrinker || obj->import_attach)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (!shmem->madv)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	else if (shmem->madv > 0)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+	else
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGED;
+
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+
+	mutex_unlock(&gem_shrinker->lock);
+}
+
+static void drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+	dma_resv_lock(shmem->base.resv, NULL);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static int
+drm_gem_shmem_set_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->eviction_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->evicted) {
+		err = obj->funcs->swap_in(obj);
+		if (err)
+			return err;
+	}
+
+	shmem->eviction_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->purging_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged. Initially purging is
+ * disabled for all GEMs. Each set_pureable() call must have corresponding
+ * set_unpureable() call. If GEM was purged, then -ENOMEM is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable);
+
+static int
+drm_gem_shmem_set_unpurgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	shmem->purging_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_and_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_evictable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_purgeable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_unevictable_locked(shmem);
+	}
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unpurgeable_and_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_unevictable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_purgeable_locked(shmem);
+	}
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable_and_evictable() - Make GEM unpurgeable and
+ * 						 unevictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can't be purged and evicted. Each
+ * set_purgeable_and_evictable() call must have corresponding
+ * unpurgeable_and_unevictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable_and_evictable);
+
+/**
+ * drm_gem_shmem_set_unpurgeable_and_unevictable() - Make GEM purgeable and
+ * 						     evictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged and evicted. Each
+ * unpurgeable_and_unevictable() call must have corresponding
+ * set_purgeable_and_evictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
+	if (ret)
+		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_unpurgeable_and_unevictable);
+
+static int
+drm_gem_shmem_acquire_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct page **pages;
 
-	if (shmem->pages_use_count++ > 0)
+	if (shmem->madv < 0) {
+		WARN_ON(shmem->pages);
+		return -ENOMEM;
+	}
+
+	if (shmem->pages) {
+		WARN_ON(!shmem->evicted);
  		return 0;
+	}
 
  	pages = drm_gem_get_pages(obj);
  	if (IS_ERR(pages)) {
  		DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
-		shmem->pages_use_count = 0;
  		return PTR_ERR(pages);
  	}
 
@@ -189,6 +508,25 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
  	return 0;
  }
 
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->pages_use_count++ > 0)
+		return 0;
+
+	err = drm_gem_shmem_acquire_pages_locked(shmem);
+	if (err) {
+		shmem->pages_use_count = 0;
+		return err;
+	}
+
+	return 0;
+}
+
  /*
   * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -209,21 +547,38 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
  	if (ret)
  		return ret;
  	ret = drm_gem_shmem_get_pages_locked(shmem);
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
  }
  EXPORT_SYMBOL(drm_gem_shmem_get_pages);
 
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_get_pages_no_fail(struct drm_gem_shmem_object *shmem)
  {
-	struct drm_gem_object *obj = &shmem->base;
+	WARN_ON(shmem->base.import_attach);
 
-	if (WARN_ON_ONCE(!shmem->pages_use_count))
-		return;
+	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (--shmem->pages_use_count > 0)
+	if (drm_gem_shmem_get_pages_locked(shmem))
+		shmem->pages_use_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static void
+drm_gem_shmem_release_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	if (!shmem->pages) {
+		WARN_ON(!shmem->evicted && shmem->madv >= 0);
  		return;
+	}
 
  #ifdef CONFIG_X86
  	if (shmem->map_wc)
@@ -236,6 +591,21 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
  	shmem->pages = NULL;
  }
 
+static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (WARN_ON(!shmem->pages_use_count))
+		return;
+
+	if (--shmem->pages_use_count > 0)
+		return;
+
+	drm_gem_shmem_release_pages_locked(shmem);
+}
+
  /*
   * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -246,6 +616,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_put_pages_locked(shmem);
+	drm_gem_shmem_update_pages_state_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
  }
  EXPORT_SYMBOL(drm_gem_shmem_put_pages);
@@ -262,9 +633,21 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);
   */
  int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
  {
+	int err;
+
  	WARN_ON(shmem->base.import_attach);
 
-	return drm_gem_shmem_get_pages(shmem);
+	err = drm_gem_shmem_set_unpurgeable_and_unevictable(shmem);
+	if (err)
+		return err;
+
+	err = drm_gem_shmem_get_pages(shmem);
+	if (err) {
+		drm_gem_shmem_set_purgeable_and_evictable(shmem);
+		return err;
+	}
+
+	return 0;
  }
  EXPORT_SYMBOL(drm_gem_shmem_pin);
 
@@ -280,6 +663,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
  	WARN_ON(shmem->base.import_attach);
 
  	drm_gem_shmem_put_pages(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_unpin);
 
@@ -359,7 +743,18 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
  	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
  	if (ret)
  		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	if (ret)
+		goto unlock;
+
  	ret = drm_gem_shmem_vmap_locked(shmem, map);
+	if (ret)
+		drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	else
+		drm_gem_shmem_update_pages_state_locked(shmem);
+
+unlock:
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
@@ -404,9 +799,9 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_vunmap_locked(shmem, map);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
-
-	drm_gem_shmem_update_purgeable_status(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_vunmap);
 
@@ -447,29 +842,140 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
 
  	madv = shmem->madv;
 
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
 
  	return (madv >= 0);
  }
  EXPORT_SYMBOL(drm_gem_shmem_madvise);
 
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+/**
+ * drm_gem_shmem_swap_in_pages_locked() - Moves shmem pages back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves pages back to memory if they were previously evicted
+ * by the memory shrinker.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct sg_table *sgt;
+	int ret;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted) {
+		ret = drm_gem_shmem_acquire_pages_locked(shmem);
+		if (ret)
+			return ret;
+
+		sgt = drm_gem_shmem_get_sg_table(shmem);
+		if (IS_ERR(sgt))
+			return PTR_ERR(sgt);
+
+		ret = dma_map_sgtable(obj->dev->dev, sgt,
+				      DMA_BIDIRECTIONAL, 0);
+		if (ret) {
+			sg_free_table(sgt);
+			kfree(sgt);
+			return ret;
+		}
+
+		shmem->sgt = sgt;
+		shmem->evicted = false;
+		shmem->pages_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+
+		drm_gem_shmem_update_pages_state_locked(shmem);
+	}
+
+	return shmem->pages ? 0 : -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_pages_locked);
+
+/**
+ * drm_gem_shmem_swap_in_locked() - Moves shmem GEM back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves shmem GEM back to memory if it was previously evicted
+ * by the memory shrinker. The GEM is ready to use on success.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted)
+		return obj->funcs->swap_in(obj);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_locked);
+
+static void drm_gem_shmem_unpin_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct drm_device *dev = obj->dev;
 
-	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	if (shmem->evicted)
+		return;
 
  	dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+	drm_gem_shmem_release_pages_locked(shmem);
+	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+
  	sg_free_table(shmem->sgt);
  	kfree(shmem->sgt);
  	shmem->sgt = NULL;
+}
 
-	drm_gem_shmem_put_pages_locked(shmem);
+/**
+ * drm_gem_shmem_evict_locked - Evict shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function unpins shmem pages, allowing them to be swapped out from
+ * memory.
+ */
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
 
-	shmem->madv = -1;
+	lockdep_assert_held(&obj->resv->lock.base);
 
-	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+	WARN_ON(!drm_gem_shmem_is_evictable(shmem));
+	WARN_ON(shmem->madv < 0);
+	WARN_ON(shmem->evicted);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
+
+	shmem->evicted = true;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_EVICTED);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_evict_locked);
+
+/**
+ * drm_gem_shmem_purge_locked - Purge shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function permanently releases shmem pages.
+ */
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	WARN_ON(shmem->madv < 0);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
  	drm_gem_free_mmap_offset(obj);
 
  	/* Our goal here is to return as much of the memory as
@@ -480,6 +986,9 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
  	shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
 
  	invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0, (loff_t)-1);
+
+	shmem->madv = -1;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_PURGED);
  }
  EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
@@ -543,22 +1052,31 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
  	vm_fault_t ret;
  	struct page *page;
  	pgoff_t page_offset;
+	bool pages_inactive;
+	int err;
 
  	/* We don't use vmf->pgoff since that has the fake offset */
  	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
 
  	dma_resv_lock(shmem->base.resv, NULL);
 
-	if (page_offset >= num_pages ||
-	    WARN_ON_ONCE(!shmem->pages) ||
-	    shmem->madv < 0) {
+	pages_inactive = shmem->pages_state < DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	WARN_ON_ONCE(!shmem->pages ^ pages_inactive);
+
+	if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) {
  		ret = VM_FAULT_SIGBUS;
  	} else {
+		err = drm_gem_shmem_swap_in_locked(shmem);
+		if (err) {
+			ret = VM_FAULT_OOM;
+			goto unlock;
+		}
+
  		page = shmem->pages[page_offset];
 
  		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
  	}
-
+unlock:
  	dma_resv_unlock(shmem->base.resv);
 
  	return ret;
@@ -568,13 +1086,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
  {
  	struct drm_gem_object *obj = vma->vm_private_data;
  	struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
-	int ret;
-
-	WARN_ON(shmem->base.import_attach);
-
-	ret = drm_gem_shmem_get_pages(shmem);
-	WARN_ON_ONCE(ret != 0);
 
+	drm_gem_shmem_get_pages_no_fail(shmem);
  	drm_gem_vm_open(vma);
  }
 
@@ -716,6 +1229,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
 
  	shmem->sgt = sgt;
 
+	drm_gem_shmem_update_pages_state(shmem);
+
  	return sgt;
 
  err_free_sgt:
@@ -762,6 +1277,202 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_prime_import_sg_table);
 
+static struct drm_gem_shmem_shrinker *
+to_drm_shrinker(struct shrinker *shrinker)
+{
+	return container_of(shrinker, struct drm_gem_shmem_shrinker, base);
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_count_objects(struct shrinker *shrinker,
+				     struct shrink_control *sc)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	u64 count = READ_ONCE(gem_shrinker->shrinkable_count);
+
+	if (count >= SHRINK_EMPTY)
+		return SHRINK_EMPTY - 1;
+
+	return count ?: SHRINK_EMPTY;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_run_objects_scan(struct shrinker *shrinker,
+					unsigned long nr_to_scan,
+					bool *lock_contention,
+					bool evict)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	struct drm_gem_shmem_object *shmem;
+	struct list_head still_in_list;
+	struct drm_gem_object *obj;
+	unsigned long freed = 0;
+	struct list_head *lru;
+	size_t page_count;
+
+	INIT_LIST_HEAD(&still_in_list);
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (evict)
+		lru = &gem_shrinker->lru_evictable;
+	else
+		lru = &gem_shrinker->lru_purgeable;
+
+	while (freed < nr_to_scan) {
+		shmem = list_first_entry_or_null(lru, typeof(*shmem), madv_list);
+		if (!shmem)
+			break;
+
+		obj = &shmem->base;
+		page_count = obj->size >> PAGE_SHIFT;
+		list_move_tail(&shmem->madv_list, &still_in_list);
+
+		if (evict && get_nr_swap_pages() < page_count)
+			continue;
+
+		/*
+		 * If it's in the process of being freed, gem_object->free()
+		 * may be blocked on lock waiting to remove it.  So just
+		 * skip it.
+		 */
+		if (!kref_get_unless_zero(&obj->refcount))
+			continue;
+
+		mutex_unlock(&gem_shrinker->lock);
+
+		/* prevent racing with job-submission code paths */
+		if (!dma_resv_trylock(obj->resv)) {
+			*lock_contention |= true;
+			goto shrinker_lock;
+		}
+
+		/* prevent racing with the dma-buf exporting */
+		if (!mutex_trylock(&gem_shrinker->dev->object_name_lock)) {
+			*lock_contention |= true;
+			goto resv_unlock;
+		}
+
+		/* check whether h/w uses this object */
+		if (!dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_WRITE))
+			goto object_name_unlock;
+
+		/* GEM may've become unpurgeable while shrinker was unlocked */
+		if (evict) {
+			if (!drm_gem_shmem_is_evictable(shmem))
+				goto object_name_unlock;
+		} else {
+			if (!drm_gem_shmem_is_purgeable(shmem))
+				goto object_name_unlock;
+		}
+
+		if (evict)
+			freed += obj->funcs->evict(obj);
+		else
+			freed += obj->funcs->purge(obj);
+object_name_unlock:
+		mutex_unlock(&gem_shrinker->dev->object_name_lock);
+resv_unlock:
+		dma_resv_unlock(obj->resv);
+shrinker_lock:
+		drm_gem_object_put(&shmem->base);
+		mutex_lock(&gem_shrinker->lock);
+	}
+
+	list_splice_tail(&still_in_list, lru);
+
+	mutex_unlock(&gem_shrinker->lock);
+
+	return freed;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_scan_objects(struct shrinker *shrinker,
+				    struct shrink_control *sc)
+{
+	unsigned long nr_to_scan = sc->nr_to_scan;
+	bool lock_contention = false;
+	unsigned long freed;
+
+	/* purge as many objects as we can */
+	freed = drm_gem_shmem_shrinker_run_objects_scan(shrinker, nr_to_scan,
+							&lock_contention, false);
+	nr_to_scan -= freed;
+
+	/* evict as many objects as we can */
+	if (freed < nr_to_scan)
+		freed += drm_gem_shmem_shrinker_run_objects_scan(shrinker,
+								 nr_to_scan,
+								 &lock_contention,
+								 true);
+
+	return (!freed && !lock_contention) ? SHRINK_STOP : freed;
+}
+
+/**
+ * drm_gem_shmem_shrinker_register() - Register shmem shrinker
+ * @dev: DRM device
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_shrinker_register(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker;
+	int err;
+
+	if (WARN_ON(dev->shmem_shrinker))
+		return -EBUSY;
+
+	gem_shrinker = kzalloc(sizeof(*gem_shrinker), GFP_KERNEL);
+	if (!gem_shrinker)
+		return -ENOMEM;
+
+	gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects;
+	gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects;
+	gem_shrinker->base.seeks = DEFAULT_SEEKS;
+	gem_shrinker->dev = dev;
+
+	INIT_LIST_HEAD(&gem_shrinker->lru_purgeable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evictable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evicted);
+	INIT_LIST_HEAD(&gem_shrinker->lru_active);
+	mutex_init(&gem_shrinker->lock);
+
+	dev->shmem_shrinker = gem_shrinker;
+
+	err = register_shrinker(&gem_shrinker->base);
+	if (err) {
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+		return err;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_register);
+
+/**
+ * drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker
+ * @dev: DRM device
+ */
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker;
+
+	if (gem_shrinker) {
+		unregister_shrinker(&gem_shrinker->base);
+		WARN_ON(!list_empty(&gem_shrinker->lru_purgeable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evictable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evicted));
+		WARN_ON(!list_empty(&gem_shrinker->lru_active));
+		mutex_destroy(&gem_shrinker->lock);
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_unregister);
+
  MODULE_DESCRIPTION("DRM SHMEM memory-management helpers");
  MODULE_IMPORT_NS(DMA_BUF);
  MODULE_LICENSE("GPL v2");
include/drm/drm_device.h +4 -0
index 9923c7a6885e..929546cad894 100644
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;
  struct drm_vma_offset_manager;
  struct drm_vram_mm;
  struct drm_fb_helper;
+struct drm_gem_shmem_shrinker;
 
  struct inode;
 
@@ -277,6 +278,9 @@ struct drm_device {
  	/** @vram_mm: VRAM MM memory manager */
  	struct drm_vram_mm *vram_mm;
 
+	/** @shmem_shrinker: SHMEM GEM memory shrinker */
+	struct drm_gem_shmem_shrinker *shmem_shrinker;
+
  	/**
  	 * @switch_power_state:
  	 *
include/drm/drm_gem.h +35 -0
index 9d7c61a122dc..390d1ce08ed3 100644
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
  	 * This is optional but necessary for mmap support.
  	 */
  	const struct vm_operations_struct *vm_ops;
+
+	/**
+	 * @purge:
+	 *
+	 * Releases the GEM object's allocated backing storage to the system.
+	 *
+	 * Returns the number of pages that have been freed by purging the GEM object.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*purge)(struct drm_gem_object *obj);
+
+	/**
+	 * @evict:
+	 *
+	 * Unpins the GEM object's allocated backing storage, allowing shmem pages
+	 * to be swapped out.
+	 *
+	 * Returns the number of pages that have been unpinned.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*evict)(struct drm_gem_object *obj);
+
+	/**
+	 * @swap_in:
+	 *
+	 * Pins GEM object's allocated backing storage if it was previously evicted,
+	 * moving swapped out pages back to memory.
+	 *
+	 * Returns 0 on success, or -errno on error.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	int (*swap_in)(struct drm_gem_object *obj);
  };
 
  /**
include/drm/drm_gem_shmem_helper.h +100 -5
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
  #include <linux/fs.h>
  #include <linux/mm.h>
  #include <linux/mutex.h>
+#include <linux/shrinker.h>
 
  #include <drm/drm_file.h>
  #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
  struct dma_buf_attachment;
  struct drm_mode_create_dumb;
  struct drm_printer;
+struct drm_device;
  struct sg_table;
 
+enum drm_gem_shmem_pages_state {
+	DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
+	DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+	DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+	DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};
+
  /**
   * struct drm_gem_shmem_object - GEM object backed by shmem
   */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
  	 * @madv: State for madvise
  	 *
  	 * 0 is active/inuse.
+	 * 1 is not-needed/can-be-purged
  	 * A negative value is the object is purged.
-	 * Positive values are driver specific and not used by the helpers.
  	 */
  	int madv;
 
@@ -91,6 +102,40 @@ struct drm_gem_shmem_object {
  	 * @map_wc: map object write-combined (instead of using shmem defaults).
  	 */
  	bool map_wc;
+
+	/**
+	 * @eviction_disable_count:
+	 *
+	 * The shmem pages are disallowed to be evicted by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int eviction_disable_count;
+
+	/**
+	 * @purging_disable_count:
+	 *
+	 * The shmem pages are disallowed to be purged by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int purging_disable_count;
+
+	/**
+	 * @pages_state: Current state of shmem pages. Used internally by
+	 * memory shrinker.
+	 */
+	enum drm_gem_shmem_pages_state pages_state;
+
+	/**
+	 * @evicted: True if shmem pages were evicted by the memory shrinker.
+	 * Used internally by memory shrinker.
+	 */
+	bool evicted;
+
+	/**
+	 * @pages_shrinkable: True if shmem pages can be evicted or purged
+	 * by the memory shrinker. Used internally by memory shrinker.
+	 */
+	bool pages_shrinkable;
  };
 
  #define to_drm_gem_shmem_obj(obj) \
@@ -111,15 +156,33 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
 
  int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv);
 
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem);
+
+static inline bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+	return (shmem->madv >= 0) && !shmem->eviction_disable_count &&
+		shmem->base.funcs->evict && shmem->base.funcs->swap_in &&
+		!shmem->vmap_use_count && !shmem->base.dma_buf &&
+		!shmem->base.import_attach && shmem->sgt;
+}
+
  static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem)
  {
-	return (shmem->madv > 0) &&
-		!shmem->vmap_use_count && shmem->sgt &&
-		!shmem->base.dma_buf && !shmem->base.import_attach;
+	return (shmem->madv > 0) && !shmem->purging_disable_count &&
+		!shmem->vmap_use_count && shmem->base.funcs->purge &&
+		!shmem->base.dma_buf && !shmem->base.import_attach &&
+		shmem->sgt;
  }
 
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem);
+
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem);
+
  bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
 
  struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem);
  struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem);
@@ -262,6 +325,38 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct v
  	return drm_gem_shmem_mmap(shmem, vma);
  }
 
+/**
+ * struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs
+ */
+struct drm_gem_shmem_shrinker {
+	/** @base: Shrinker for purging shmem GEM objects */
+	struct shrinker base;
+
+	/** @lock: Protects @lru_* */
+	struct mutex lock;
+
+	/** @lru_purgeable: List of shmem GEM objects available for purging */
+	struct list_head lru_purgeable;
+
+	/** @lru_active: List of active shmem GEM objects */
+	struct list_head lru_active;
+
+	/** @lru_evictable: List of shmem GEM objects that can be evicted */
+	struct list_head lru_evictable;
+
+	/** @lru_evicted: List of evicted shmem GEM objects */
+	struct list_head lru_evicted;
+
+	/** @dev: DRM device that uses this shrinker */
+	struct drm_device *dev;
+
+	/** @shrinkable_count: Count of shmem GEM pages to be purged and evicted */
+	u64 shrinkable_count;
+};
+
+int drm_gem_shmem_shrinker_register(struct drm_device *dev);
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev);
+
  /*
   * Driver ops
   */
Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev
#40 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 18 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Quoted 8 lines

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered. Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

We have a few shrinkers already all over, so extracting that does make sense I think. I do agree that there's probably a few more steps than necessary involved right now in all this for the helper<->driver interface.
-Daniel

Quoted 1,146 lines

Best regards
Thomas

Quoted 1,134 lines
---
  drivers/gpu/drm/drm_gem_shmem_helper.c | 765 ++++++++++++++++++++++++-
  include/drm/drm_device.h               |   4 +
  include/drm/drm_gem.h                  |  35 ++
  include/drm/drm_gem_shmem_helper.h     | 105 +++-
  4 files changed, 877 insertions(+), 32 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +738 -27
index 3ecef571eff3..3838fb8d6f3a 100644
@@ -88,6 +88,13 @@ __drm_gem_shmem_create(struct drm_device *dev, size_t size, bool private)
  	INIT_LIST_HEAD(&shmem->madv_list);
+	/*
+	 * Eviction and purging are disabled by default, shmem user must enable
+	 * them explicitly using drm_gem_shmem_set_evictable/purgeable().
+	 */
+	shmem->eviction_disable_count = 1;
+	shmem->purging_disable_count = 1;
+
  	if (!private) {
  		/*
  		 * Our buffers are kept pinned, so allocating them
@@ -126,6 +133,107 @@ struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_create);
+static void
+drm_gem_shmem_add_pages_to_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (!shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count + page_count < page_count);
+		gem_shrinker->shrinkable_count += page_count;
+		shmem->pages_shrinkable = true;
+	}
+}
+
+static void
+drm_gem_shmem_remove_pages_from_shrinker(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	size_t page_count = obj->size >> PAGE_SHIFT;
+
+	if (shmem->pages_shrinkable) {
+		WARN_ON(gem_shrinker->shrinkable_count < page_count);
+		gem_shrinker->shrinkable_count -= page_count;
+		shmem->pages_shrinkable = false;
+	}
+}
+
+static void
+drm_gem_shmem_set_pages_state_locked(struct drm_gem_shmem_object *shmem,
+				     enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	lockdep_assert_held(&gem_shrinker->lock);
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (new_state >= DRM_GEM_SHMEM_PAGES_STATE_PINNED) {
+		if (drm_gem_shmem_is_evictable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE;
+
+		if (drm_gem_shmem_is_purgeable(shmem))
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+
+		if (!shmem->pages)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_UNPINNED;
+
+		if (shmem->evicted)
+			new_state = DRM_GEM_SHMEM_PAGES_STATE_EVICTED;
+	}
+
+	if (shmem->pages_state == new_state)
+		return;
+
+	switch (new_state) {
+	case DRM_GEM_SHMEM_PAGES_STATE_UNPINNED:
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_del_init(&shmem->madv_list);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PINNED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_active);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_purgeable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE:
+		drm_gem_shmem_add_pages_to_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evictable);
+		break;
+
+	case DRM_GEM_SHMEM_PAGES_STATE_EVICTED:
+		drm_gem_shmem_remove_pages_from_shrinker(shmem);
+		list_move_tail(&shmem->madv_list, &gem_shrinker->lru_evicted);
+		break;
+	}
+
+	shmem->pages_state = new_state;
+}
+
+static void
+drm_gem_shmem_set_pages_state(struct drm_gem_shmem_object *shmem,
+			      enum drm_gem_shmem_pages_state new_state)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+
+	if (!gem_shrinker)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+	mutex_unlock(&gem_shrinker->lock);
+}
+
  /**
   * drm_gem_shmem_free - Free resources associated with a shmem GEM object
   * @shmem: shmem GEM object to free
@@ -137,6 +245,9 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
+	/* take out shmem GEM object from the memory shrinker */
+	drm_gem_shmem_madvise(shmem, -1);
+
  	WARN_ON(shmem->vmap_use_count);
  	if (obj->import_attach) {
@@ -148,7 +259,7 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  			sg_free_table(shmem->sgt);
  			kfree(shmem->sgt);
  		}
-		if (shmem->pages)
+		if (shmem->pages_use_count)
  			drm_gem_shmem_put_pages(shmem);
  	}
@@ -159,18 +270,226 @@ void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem)
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_free);
-static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_update_pages_state_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct drm_gem_shmem_shrinker *gem_shrinker = obj->dev->shmem_shrinker;
+	enum drm_gem_shmem_pages_state new_state;
+
+	if (!gem_shrinker || obj->import_attach)
+		return;
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (!shmem->madv)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	else if (shmem->madv > 0)
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE;
+	else
+		new_state = DRM_GEM_SHMEM_PAGES_STATE_PURGED;
+
+	drm_gem_shmem_set_pages_state_locked(shmem, new_state);
+
+	mutex_unlock(&gem_shrinker->lock);
+}
+
+static void drm_gem_shmem_update_pages_state(struct drm_gem_shmem_object *shmem)
+{
+	dma_resv_lock(shmem->base.resv, NULL);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static int
+drm_gem_shmem_set_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->eviction_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->evicted) {
+		err = obj->funcs->swap_in(obj);
+		if (err)
+			return err;
+	}
+
+	shmem->eviction_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret = 0;
+
+	WARN_ON_ONCE(!shmem->purging_disable_count--);
+
+	if (shmem->madv < 0)
+		ret = -ENOMEM;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable() - Make GEM purgeable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged. Initially purging is
+ * disabled for all GEMs. Each set_pureable() call must have corresponding
+ * set_unpureable() call. If GEM was purged, then -ENOMEM is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable);
+
+static int
+drm_gem_shmem_set_unpurgeable_locked(struct drm_gem_shmem_object *shmem)
+{
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	shmem->purging_disable_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	return 0;
+}
+
+static int
+drm_gem_shmem_set_purgeable_and_evictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_evictable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_purgeable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_unevictable_locked(shmem);
+	}
+
+	return ret;
+}
+
+static int
+drm_gem_shmem_set_unpurgeable_and_unevictable_locked(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_locked(shmem);
+	if (!ret) {
+		ret = drm_gem_shmem_set_unevictable_locked(shmem);
+		if (ret)
+			drm_gem_shmem_set_purgeable_locked(shmem);
+	}
+
+	return ret;
+}
+
+/**
+ * drm_gem_shmem_set_purgeable_and_evictable() - Make GEM unpurgeable and
+ * 						 unevictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can't be purged and evicted. Each
+ * set_purgeable_and_evictable() call must have corresponding
+ * unpurgeable_and_unevictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	dma_resv_lock(shmem->base.resv, NULL);
+	ret = drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_purgeable_and_evictable);
+
+/**
+ * drm_gem_shmem_set_unpurgeable_and_unevictable() - Make GEM purgeable and
+ * 						     evictable by memory shrinker
+ * @shmem: shmem GEM object
+ *
+ * Tell memory shrinker that this GEM can be purged and evicted. Each
+ * unpurgeable_and_unevictable() call must have corresponding
+ * set_purgeable_and_evictable() call. If GEM was purged, then -ENOMEM
+ * is returned.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem)
+{
+	int ret;
+
+	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
+	if (ret)
+		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	dma_resv_unlock(shmem->base.resv);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_set_unpurgeable_and_unevictable);
+
+static int
+drm_gem_shmem_acquire_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct page **pages;
-	if (shmem->pages_use_count++ > 0)
+	if (shmem->madv < 0) {
+		WARN_ON(shmem->pages);
+		return -ENOMEM;
+	}
+
+	if (shmem->pages) {
+		WARN_ON(!shmem->evicted);
  		return 0;
+	}
  	pages = drm_gem_get_pages(obj);
  	if (IS_ERR(pages)) {
  		DRM_DEBUG_KMS("Failed to get pages (%ld)\n", PTR_ERR(pages));
-		shmem->pages_use_count = 0;
  		return PTR_ERR(pages);
  	}
@@ -189,6 +508,25 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
  	return 0;
  }
+static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	int err;
+
+	if (shmem->madv < 0)
+		return -ENOMEM;
+
+	if (shmem->pages_use_count++ > 0)
+		return 0;
+
+	err = drm_gem_shmem_acquire_pages_locked(shmem);
+	if (err) {
+		shmem->pages_use_count = 0;
+		return err;
+	}
+
+	return 0;
+}
+
  /*
   * drm_gem_shmem_get_pages - Allocate backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -209,21 +547,38 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
  	if (ret)
  		return ret;
  	ret = drm_gem_shmem_get_pages_locked(shmem);
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
  	return ret;
  }
  EXPORT_SYMBOL(drm_gem_shmem_get_pages);
-static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+static void drm_gem_shmem_get_pages_no_fail(struct drm_gem_shmem_object *shmem)
  {
-	struct drm_gem_object *obj = &shmem->base;
+	WARN_ON(shmem->base.import_attach);
-	if (WARN_ON_ONCE(!shmem->pages_use_count))
-		return;
+	dma_resv_lock(shmem->base.resv, NULL);
-	if (--shmem->pages_use_count > 0)
+	if (drm_gem_shmem_get_pages_locked(shmem))
+		shmem->pages_use_count++;
+
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
+	dma_resv_unlock(shmem->base.resv);
+}
+
+static void
+drm_gem_shmem_release_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	if (!shmem->pages) {
+		WARN_ON(!shmem->evicted && shmem->madv >= 0);
  		return;
+	}
  #ifdef CONFIG_X86
  	if (shmem->map_wc)
@@ -236,6 +591,21 @@ static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
  	shmem->pages = NULL;
  }
+static void drm_gem_shmem_put_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (WARN_ON(!shmem->pages_use_count))
+		return;
+
+	if (--shmem->pages_use_count > 0)
+		return;
+
+	drm_gem_shmem_release_pages_locked(shmem);
+}
+
  /*
   * drm_gem_shmem_put_pages - Decrease use count on the backing pages for a shmem GEM object
   * @shmem: shmem GEM object
@@ -246,6 +616,7 @@ void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem)
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_put_pages_locked(shmem);
+	drm_gem_shmem_update_pages_state_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
  }
  EXPORT_SYMBOL(drm_gem_shmem_put_pages);
@@ -262,9 +633,21 @@ EXPORT_SYMBOL(drm_gem_shmem_put_pages);
   */
  int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem)
  {
+	int err;
+
  	WARN_ON(shmem->base.import_attach);
-	return drm_gem_shmem_get_pages(shmem);
+	err = drm_gem_shmem_set_unpurgeable_and_unevictable(shmem);
+	if (err)
+		return err;
+
+	err = drm_gem_shmem_get_pages(shmem);
+	if (err) {
+		drm_gem_shmem_set_purgeable_and_evictable(shmem);
+		return err;
+	}
+
+	return 0;
  }
  EXPORT_SYMBOL(drm_gem_shmem_pin);
@@ -280,6 +663,7 @@ void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem)
  	WARN_ON(shmem->base.import_attach);
  	drm_gem_shmem_put_pages(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_unpin);
@@ -359,7 +743,18 @@ int drm_gem_shmem_vmap(struct drm_gem_shmem_object *shmem,
  	ret = dma_resv_lock_interruptible(shmem->base.resv, NULL);
  	if (ret)
  		return ret;
+
+	ret = drm_gem_shmem_set_unpurgeable_and_unevictable_locked(shmem);
+	if (ret)
+		goto unlock;
+
  	ret = drm_gem_shmem_vmap_locked(shmem, map);
+	if (ret)
+		drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
+	else
+		drm_gem_shmem_update_pages_state_locked(shmem);
+
+unlock:
  	dma_resv_unlock(shmem->base.resv);
  	return ret;
@@ -404,9 +799,9 @@ void drm_gem_shmem_vunmap(struct drm_gem_shmem_object *shmem,
  {
  	dma_resv_lock(shmem->base.resv, NULL);
  	drm_gem_shmem_vunmap_locked(shmem, map);
+	drm_gem_shmem_update_pages_state_locked(shmem);
+	drm_gem_shmem_set_purgeable_and_evictable_locked(shmem);
  	dma_resv_unlock(shmem->base.resv);
-
-	drm_gem_shmem_update_purgeable_status(shmem);
  }
  EXPORT_SYMBOL(drm_gem_shmem_vunmap);
@@ -447,29 +842,140 @@ int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv)
  	madv = shmem->madv;
+	drm_gem_shmem_update_pages_state_locked(shmem);
+
  	dma_resv_unlock(shmem->base.resv);
  	return (madv >= 0);
  }
  EXPORT_SYMBOL(drm_gem_shmem_madvise);
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+/**
+ * drm_gem_shmem_swap_in_pages_locked() - Moves shmem pages back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves pages back to memory if they were previously evicted
+ * by the memory shrinker.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+	struct sg_table *sgt;
+	int ret;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted) {
+		ret = drm_gem_shmem_acquire_pages_locked(shmem);
+		if (ret)
+			return ret;
+
+		sgt = drm_gem_shmem_get_sg_table(shmem);
+		if (IS_ERR(sgt))
+			return PTR_ERR(sgt);
+
+		ret = dma_map_sgtable(obj->dev->dev, sgt,
+				      DMA_BIDIRECTIONAL, 0);
+		if (ret) {
+			sg_free_table(sgt);
+			kfree(sgt);
+			return ret;
+		}
+
+		shmem->sgt = sgt;
+		shmem->evicted = false;
+		shmem->pages_state = DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+
+		drm_gem_shmem_update_pages_state_locked(shmem);
+	}
+
+	return shmem->pages ? 0 : -ENOMEM;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_pages_locked);
+
+/**
+ * drm_gem_shmem_swap_in_locked() - Moves shmem GEM back to memory
+ * @shmem: shmem GEM object
+ *
+ * This function moves shmem GEM back to memory if it was previously evicted
+ * by the memory shrinker. The GEM is ready to use on success.
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	if (shmem->evicted)
+		return obj->funcs->swap_in(obj);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_swap_in_locked);
+
+static void drm_gem_shmem_unpin_pages_locked(struct drm_gem_shmem_object *shmem)
  {
  	struct drm_gem_object *obj = &shmem->base;
  	struct drm_device *dev = obj->dev;
-	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	if (shmem->evicted)
+		return;
  	dma_unmap_sgtable(dev->dev, shmem->sgt, DMA_BIDIRECTIONAL, 0);
+	drm_gem_shmem_release_pages_locked(shmem);
+	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+
  	sg_free_table(shmem->sgt);
  	kfree(shmem->sgt);
  	shmem->sgt = NULL;
+}
-	drm_gem_shmem_put_pages_locked(shmem);
+/**
+ * drm_gem_shmem_evict_locked - Evict shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function unpins shmem pages, allowing them to be swapped out from
+ * memory.
+ */
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
-	shmem->madv = -1;
+	lockdep_assert_held(&obj->resv->lock.base);
-	drm_vma_node_unmap(&obj->vma_node, dev->anon_inode->i_mapping);
+	WARN_ON(!drm_gem_shmem_is_evictable(shmem));
+	WARN_ON(shmem->madv < 0);
+	WARN_ON(shmem->evicted);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
+
+	shmem->evicted = true;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_EVICTED);
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_evict_locked);
+
+/**
+ * drm_gem_shmem_purge_locked - Purge shmem pages
+ * @shmem: shmem GEM object
+ *
+ * This function permanently releases shmem pages.
+ */
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
+{
+	struct drm_gem_object *obj = &shmem->base;
+
+	lockdep_assert_held(&obj->resv->lock.base);
+
+	WARN_ON(!drm_gem_shmem_is_purgeable(shmem));
+	WARN_ON(shmem->madv < 0);
+
+	drm_gem_shmem_unpin_pages_locked(shmem);
  	drm_gem_free_mmap_offset(obj);
  	/* Our goal here is to return as much of the memory as
@@ -480,6 +986,9 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
  	shmem_truncate_range(file_inode(obj->filp), 0, (loff_t)-1);
  	invalidate_mapping_pages(file_inode(obj->filp)->i_mapping, 0, (loff_t)-1);
+
+	shmem->madv = -1;
+	drm_gem_shmem_set_pages_state(shmem, DRM_GEM_SHMEM_PAGES_STATE_PURGED);
  }
  EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
@@ -543,22 +1052,31 @@ static vm_fault_t drm_gem_shmem_fault(struct vm_fault *vmf)
  	vm_fault_t ret;
  	struct page *page;
  	pgoff_t page_offset;
+	bool pages_inactive;
+	int err;
  	/* We don't use vmf->pgoff since that has the fake offset */
  	page_offset = (vmf->address - vma->vm_start) >> PAGE_SHIFT;
  	dma_resv_lock(shmem->base.resv, NULL);
-	if (page_offset >= num_pages ||
-	    WARN_ON_ONCE(!shmem->pages) ||
-	    shmem->madv < 0) {
+	pages_inactive = shmem->pages_state < DRM_GEM_SHMEM_PAGES_STATE_PINNED;
+	WARN_ON_ONCE(!shmem->pages ^ pages_inactive);
+
+	if (page_offset >= num_pages || (!shmem->pages && !shmem->evicted)) {
  		ret = VM_FAULT_SIGBUS;
  	} else {
+		err = drm_gem_shmem_swap_in_locked(shmem);
+		if (err) {
+			ret = VM_FAULT_OOM;
+			goto unlock;
+		}
+
  		page = shmem->pages[page_offset];
  		ret = vmf_insert_pfn(vma, vmf->address, page_to_pfn(page));
  	}
-
+unlock:
  	dma_resv_unlock(shmem->base.resv);
  	return ret;
@@ -568,13 +1086,8 @@ static void drm_gem_shmem_vm_open(struct vm_area_struct *vma)
  {
  	struct drm_gem_object *obj = vma->vm_private_data;
  	struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
-	int ret;
-
-	WARN_ON(shmem->base.import_attach);
-
-	ret = drm_gem_shmem_get_pages(shmem);
-	WARN_ON_ONCE(ret != 0);
+	drm_gem_shmem_get_pages_no_fail(shmem);
  	drm_gem_vm_open(vma);
  }
@@ -716,6 +1229,8 @@ struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem)
  	shmem->sgt = sgt;
+	drm_gem_shmem_update_pages_state(shmem);
+
  	return sgt;
  err_free_sgt:
@@ -762,6 +1277,202 @@ drm_gem_shmem_prime_import_sg_table(struct drm_device *dev,
  }
  EXPORT_SYMBOL_GPL(drm_gem_shmem_prime_import_sg_table);
+static struct drm_gem_shmem_shrinker *
+to_drm_shrinker(struct shrinker *shrinker)
+{
+	return container_of(shrinker, struct drm_gem_shmem_shrinker, base);
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_count_objects(struct shrinker *shrinker,
+				     struct shrink_control *sc)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	u64 count = READ_ONCE(gem_shrinker->shrinkable_count);
+
+	if (count >= SHRINK_EMPTY)
+		return SHRINK_EMPTY - 1;
+
+	return count ?: SHRINK_EMPTY;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_run_objects_scan(struct shrinker *shrinker,
+					unsigned long nr_to_scan,
+					bool *lock_contention,
+					bool evict)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = to_drm_shrinker(shrinker);
+	struct drm_gem_shmem_object *shmem;
+	struct list_head still_in_list;
+	struct drm_gem_object *obj;
+	unsigned long freed = 0;
+	struct list_head *lru;
+	size_t page_count;
+
+	INIT_LIST_HEAD(&still_in_list);
+
+	mutex_lock(&gem_shrinker->lock);
+
+	if (evict)
+		lru = &gem_shrinker->lru_evictable;
+	else
+		lru = &gem_shrinker->lru_purgeable;
+
+	while (freed < nr_to_scan) {
+		shmem = list_first_entry_or_null(lru, typeof(*shmem), madv_list);
+		if (!shmem)
+			break;
+
+		obj = &shmem->base;
+		page_count = obj->size >> PAGE_SHIFT;
+		list_move_tail(&shmem->madv_list, &still_in_list);
+
+		if (evict && get_nr_swap_pages() < page_count)
+			continue;
+
+		/*
+		 * If it's in the process of being freed, gem_object->free()
+		 * may be blocked on lock waiting to remove it.  So just
+		 * skip it.
+		 */
+		if (!kref_get_unless_zero(&obj->refcount))
+			continue;
+
+		mutex_unlock(&gem_shrinker->lock);
+
+		/* prevent racing with job-submission code paths */
+		if (!dma_resv_trylock(obj->resv)) {
+			*lock_contention |= true;
+			goto shrinker_lock;
+		}
+
+		/* prevent racing with the dma-buf exporting */
+		if (!mutex_trylock(&gem_shrinker->dev->object_name_lock)) {
+			*lock_contention |= true;
+			goto resv_unlock;
+		}
+
+		/* check whether h/w uses this object */
+		if (!dma_resv_test_signaled(obj->resv, DMA_RESV_USAGE_WRITE))
+			goto object_name_unlock;
+
+		/* GEM may've become unpurgeable while shrinker was unlocked */
+		if (evict) {
+			if (!drm_gem_shmem_is_evictable(shmem))
+				goto object_name_unlock;
+		} else {
+			if (!drm_gem_shmem_is_purgeable(shmem))
+				goto object_name_unlock;
+		}
+
+		if (evict)
+			freed += obj->funcs->evict(obj);
+		else
+			freed += obj->funcs->purge(obj);
+object_name_unlock:
+		mutex_unlock(&gem_shrinker->dev->object_name_lock);
+resv_unlock:
+		dma_resv_unlock(obj->resv);
+shrinker_lock:
+		drm_gem_object_put(&shmem->base);
+		mutex_lock(&gem_shrinker->lock);
+	}
+
+	list_splice_tail(&still_in_list, lru);
+
+	mutex_unlock(&gem_shrinker->lock);
+
+	return freed;
+}
+
+static unsigned long
+drm_gem_shmem_shrinker_scan_objects(struct shrinker *shrinker,
+				    struct shrink_control *sc)
+{
+	unsigned long nr_to_scan = sc->nr_to_scan;
+	bool lock_contention = false;
+	unsigned long freed;
+
+	/* purge as many objects as we can */
+	freed = drm_gem_shmem_shrinker_run_objects_scan(shrinker, nr_to_scan,
+							&lock_contention, false);
+	nr_to_scan -= freed;
+
+	/* evict as many objects as we can */
+	if (freed < nr_to_scan)
+		freed += drm_gem_shmem_shrinker_run_objects_scan(shrinker,
+								 nr_to_scan,
+								 &lock_contention,
+								 true);
+
+	return (!freed && !lock_contention) ? SHRINK_STOP : freed;
+}
+
+/**
+ * drm_gem_shmem_shrinker_register() - Register shmem shrinker
+ * @dev: DRM device
+ *
+ * Returns:
+ * 0 on success or a negative error code on failure.
+ */
+int drm_gem_shmem_shrinker_register(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker;
+	int err;
+
+	if (WARN_ON(dev->shmem_shrinker))
+		return -EBUSY;
+
+	gem_shrinker = kzalloc(sizeof(*gem_shrinker), GFP_KERNEL);
+	if (!gem_shrinker)
+		return -ENOMEM;
+
+	gem_shrinker->base.count_objects = drm_gem_shmem_shrinker_count_objects;
+	gem_shrinker->base.scan_objects = drm_gem_shmem_shrinker_scan_objects;
+	gem_shrinker->base.seeks = DEFAULT_SEEKS;
+	gem_shrinker->dev = dev;
+
+	INIT_LIST_HEAD(&gem_shrinker->lru_purgeable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evictable);
+	INIT_LIST_HEAD(&gem_shrinker->lru_evicted);
+	INIT_LIST_HEAD(&gem_shrinker->lru_active);
+	mutex_init(&gem_shrinker->lock);
+
+	dev->shmem_shrinker = gem_shrinker;
+
+	err = register_shrinker(&gem_shrinker->base);
+	if (err) {
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+		return err;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_register);
+
+/**
+ * drm_gem_shmem_shrinker_unregister() - Unregister shmem shrinker
+ * @dev: DRM device
+ */
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev)
+{
+	struct drm_gem_shmem_shrinker *gem_shrinker = dev->shmem_shrinker;
+
+	if (gem_shrinker) {
+		unregister_shrinker(&gem_shrinker->base);
+		WARN_ON(!list_empty(&gem_shrinker->lru_purgeable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evictable));
+		WARN_ON(!list_empty(&gem_shrinker->lru_evicted));
+		WARN_ON(!list_empty(&gem_shrinker->lru_active));
+		mutex_destroy(&gem_shrinker->lock);
+		dev->shmem_shrinker = NULL;
+		kfree(gem_shrinker);
+	}
+}
+EXPORT_SYMBOL_GPL(drm_gem_shmem_shrinker_unregister);
+
  MODULE_DESCRIPTION("DRM SHMEM memory-management helpers");
  MODULE_IMPORT_NS(DMA_BUF);
  MODULE_LICENSE("GPL v2");
include/drm/drm_device.h +4 -0
index 9923c7a6885e..929546cad894 100644
@@ -16,6 +16,7 @@ struct drm_vblank_crtc;
  struct drm_vma_offset_manager;
  struct drm_vram_mm;
  struct drm_fb_helper;
+struct drm_gem_shmem_shrinker;
  struct inode;
@@ -277,6 +278,9 @@ struct drm_device {
  	/** @vram_mm: VRAM MM memory manager */
  	struct drm_vram_mm *vram_mm;
+	/** @shmem_shrinker: SHMEM GEM memory shrinker */
+	struct drm_gem_shmem_shrinker *shmem_shrinker;
+
  	/**
  	 * @switch_power_state:
  	 *
include/drm/drm_gem.h +35 -0
index 9d7c61a122dc..390d1ce08ed3 100644
@@ -172,6 +172,41 @@ struct drm_gem_object_funcs {
  	 * This is optional but necessary for mmap support.
  	 */
  	const struct vm_operations_struct *vm_ops;
+
+	/**
+	 * @purge:
+	 *
+	 * Releases the GEM object's allocated backing storage to the system.
+	 *
+	 * Returns the number of pages that have been freed by purging the GEM object.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*purge)(struct drm_gem_object *obj);
+
+	/**
+	 * @evict:
+	 *
+	 * Unpins the GEM object's allocated backing storage, allowing shmem pages
+	 * to be swapped out.
+	 *
+	 * Returns the number of pages that have been unpinned.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	unsigned long (*evict)(struct drm_gem_object *obj);
+
+	/**
+	 * @swap_in:
+	 *
+	 * Pins GEM object's allocated backing storage if it was previously evicted,
+	 * moving swapped out pages back to memory.
+	 *
+	 * Returns 0 on success, or -errno on error.
+	 *
+	 * This callback is used by the GEM shrinker.
+	 */
+	int (*swap_in)(struct drm_gem_object *obj);
  };
  /**
include/drm/drm_gem_shmem_helper.h +100 -5
index 70889533962a..a65557b446e6 100644
@@ -6,6 +6,7 @@
  #include <linux/fs.h>
  #include <linux/mm.h>
  #include <linux/mutex.h>
+#include <linux/shrinker.h>
  #include <drm/drm_file.h>
  #include <drm/drm_gem.h>
@@ -15,8 +16,18 @@
  struct dma_buf_attachment;
  struct drm_mode_create_dumb;
  struct drm_printer;
+struct drm_device;
  struct sg_table;
+enum drm_gem_shmem_pages_state {
+	DRM_GEM_SHMEM_PAGES_STATE_PURGED = -2,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTED = -1,
+	DRM_GEM_SHMEM_PAGES_STATE_UNPINNED = 0,
+	DRM_GEM_SHMEM_PAGES_STATE_PINNED = 1,
+	DRM_GEM_SHMEM_PAGES_STATE_EVICTABLE = 2,
+	DRM_GEM_SHMEM_PAGES_STATE_PURGEABLE = 3,
+};
+
  /**
   * struct drm_gem_shmem_object - GEM object backed by shmem
   */
@@ -43,8 +54,8 @@ struct drm_gem_shmem_object {
  	 * @madv: State for madvise
  	 *
  	 * 0 is active/inuse.
+	 * 1 is not-needed/can-be-purged
  	 * A negative value is the object is purged.
-	 * Positive values are driver specific and not used by the helpers.
  	 */
  	int madv;
@@ -91,6 +102,40 @@ struct drm_gem_shmem_object {
  	 * @map_wc: map object write-combined (instead of using shmem defaults).
  	 */
  	bool map_wc;
+
+	/**
+	 * @eviction_disable_count:
+	 *
+	 * The shmem pages are disallowed to be evicted by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int eviction_disable_count;
+
+	/**
+	 * @purging_disable_count:
+	 *
+	 * The shmem pages are disallowed to be purged by the memory shrinker
+	 * while count is non-zero. Used internally by memory shrinker.
+	 */
+	unsigned int purging_disable_count;
+
+	/**
+	 * @pages_state: Current state of shmem pages. Used internally by
+	 * memory shrinker.
+	 */
+	enum drm_gem_shmem_pages_state pages_state;
+
+	/**
+	 * @evicted: True if shmem pages were evicted by the memory shrinker.
+	 * Used internally by memory shrinker.
+	 */
+	bool evicted;
+
+	/**
+	 * @pages_shrinkable: True if shmem pages can be evicted or purged
+	 * by the memory shrinker. Used internally by memory shrinker.
+	 */
+	bool pages_shrinkable;
  };
  #define to_drm_gem_shmem_obj(obj) \
@@ -111,15 +156,33 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct
  int drm_gem_shmem_madvise(struct drm_gem_shmem_object *shmem, int madv);
+int drm_gem_shmem_set_purgeable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_purgeable_and_evictable(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_set_unpurgeable_and_unevictable(struct drm_gem_shmem_object *shmem);
+
+static inline bool drm_gem_shmem_is_evictable(struct drm_gem_shmem_object *shmem)
+{
+	return (shmem->madv >= 0) && !shmem->eviction_disable_count &&
+		shmem->base.funcs->evict && shmem->base.funcs->swap_in &&
+		!shmem->vmap_use_count && !shmem->base.dma_buf &&
+		!shmem->base.import_attach && shmem->sgt;
+}
+
  static inline bool drm_gem_shmem_is_purgeable(struct drm_gem_shmem_object *shmem)
  {
-	return (shmem->madv > 0) &&
-		!shmem->vmap_use_count && shmem->sgt &&
-		!shmem->base.dma_buf && !shmem->base.import_attach;
+	return (shmem->madv > 0) && !shmem->purging_disable_count &&
+		!shmem->vmap_use_count && shmem->base.funcs->purge &&
+		!shmem->base.dma_buf && !shmem->base.import_attach &&
+		shmem->sgt;
  }
-void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem);
+int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem);
+
+void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem);
+
  bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
+void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
  struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem);
  struct sg_table *drm_gem_shmem_get_pages_sgt(struct drm_gem_shmem_object *shmem);
@@ -262,6 +325,38 @@ static inline int drm_gem_shmem_object_mmap(struct drm_gem_object *obj, struct v
  	return drm_gem_shmem_mmap(shmem, vma);
  }
+/**
+ * struct drm_gem_shmem_shrinker - Generic memory shrinker for shmem GEMs
+ */
+struct drm_gem_shmem_shrinker {
+	/** @base: Shrinker for purging shmem GEM objects */
+	struct shrinker base;
+
+	/** @lock: Protects @lru_* */
+	struct mutex lock;
+
+	/** @lru_purgeable: List of shmem GEM objects available for purging */
+	struct list_head lru_purgeable;
+
+	/** @lru_active: List of active shmem GEM objects */
+	struct list_head lru_active;
+
+	/** @lru_evictable: List of shmem GEM objects that can be evicted */
+	struct list_head lru_evictable;
+
+	/** @lru_evicted: List of evicted shmem GEM objects */
+	struct list_head lru_evicted;
+
+	/** @dev: DRM device that uses this shrinker */
+	struct drm_device *dev;
+
+	/** @shrinkable_count: Count of shmem GEM pages to be purged and evicted */
+	u64 shrinkable_count;
+};
+
+int drm_gem_shmem_shrinker_register(struct drm_device *dev);
+void drm_gem_shmem_shrinker_unregister(struct drm_device *dev);
+
  /*
   * Driver ops
   */
Signature · 6 lines
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev
Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#41 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

In reply to Thomas Zimmermann
Quoted 18 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Quoted 5 lines

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered.  Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

Daniel mostly suggesting to make interface more flexible for future drivers, so we won't need to re-do it later on. My version of the interface is based on what drivers need today.

Why do you think it's a problem to turn shmem helper into the simple generic memory manager? I don't see how it's better to have drivers duplicating the exactly same efforts and making different mistakes.

The shmem shrinker implementation is mostly based on the freedreno's shrinker and it's very easy to enable generic shrinker for VirtIO and Panfrost drivers. I think in the future freedreno and other drivers could switch to use drm shmem instead of open coding the memory management.

Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

#42 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 29 lines · Dmitry Osipenko
Quoted 18 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Quoted 5 lines

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered.  Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

Daniel mostly suggesting to make interface more flexible for future drivers, so we won't need to re-do it later on. My version of the interface is based on what drivers need today.

Quoted 8 lines

Why do you think it's a problem to turn shmem helper into the simple generic memory manager? I don't see how it's better to have drivers duplicating the exactly same efforts and making different mistakes.

The shmem shrinker implementation is mostly based on the freedreno's shrinker and it's very easy to enable generic shrinker for VirtIO and Panfrost drivers. I think in the future freedreno and other drivers could switch to use drm shmem instead of open coding the memory management.

Yeah I think we have enough shrinkers all over drm to actually design something solid here.

There's also the i915 shrinker and some kinda shrinker in ttm too. So we are definitely past the "have 3 examples to make sure you design something solid" rule of thumb.

I also have a bit an idea that we could try to glue the shmem shrinker into ttm, at least at a very high level that's something that would make some sense.

Quoted 9 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers? I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#43 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 42 lines · Daniel Vetter
Quoted 29 lines · Dmitry Osipenko
Quoted 18 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Quoted 5 lines

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered.  Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

Daniel mostly suggesting to make interface more flexible for future drivers, so we won't need to re-do it later on. My version of the interface is based on what drivers need today.

Quoted 8 lines

Why do you think it's a problem to turn shmem helper into the simple generic memory manager? I don't see how it's better to have drivers duplicating the exactly same efforts and making different mistakes.

The shmem shrinker implementation is mostly based on the freedreno's shrinker and it's very easy to enable generic shrinker for VirtIO and Panfrost drivers. I think in the future freedreno and other drivers could switch to use drm shmem instead of open coding the memory management.

Yeah I think we have enough shrinkers all over drm to actually design something solid here.

Quoted 7 lines

There's also the i915 shrinker and some kinda shrinker in ttm too. So we are definitely past the "have 3 examples to make sure you design something solid" rule of thumb.

I also have a bit an idea that we could try to glue the shmem shrinker into ttm, at least at a very high level that's something that would make some sense.

Before gluing the shmem shrinker into ttm, the drivers should be switched to ttm? Or do you mean something else by the gluing?

Perhaps it should be possible to have a common drm-shrinker helper that will do the basic-common things like tracking the eviction size and check whether BO is exported or locked, but we shouldn't consider doing this for now. For the starter more reasonable should be to create a common shrinker base for drivers that use drm-shmem, IMO.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

Quoted 7 lines
Quoted 4 lines

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

The drm-shmem only handles shmem pages, not the mappings of these pages.

Signature · 2 lines
Best regards,
Dmitry
#44 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 47 lines · Dmitry Osipenko
Quoted 46 lines · Daniel Vetter
Quoted 29 lines · Dmitry Osipenko
Quoted 18 lines · Thomas Zimmermann

Hi

Am 18.04.22 um 00:37 schrieb Dmitry Osipenko:

Quoted 8 lines

Introduce a common DRM SHMEM shrinker. It allows to reduce code duplication among DRM drivers that implement theirs own shrinkers. This is initial version of the shrinker that covers basic needs of GPU drivers, both purging and eviction of shmem objects are supported.

This patch is based on a couple ideas borrowed from Rob's Clark MSM shrinker and Thomas' Zimmermann variant of SHMEM shrinker.

Quoted 6 lines

In order to start using DRM SHMEM shrinker drivers should:

1. Implement new purge(), evict() + swap_in() GEM callbacks.
2. Register shrinker using drm_gem_shmem_shrinker_register(drm_device).
3. Use drm_gem_shmem_set_purgeable_and_evictable(shmem) and alike API functions to activate shrinking of GEMs.

Quoted 5 lines

Honestly speaking, after reading the patch and the discussion here I really don't like where all tis is going. The interfaces and implementation are overengineered.  Descisions about evicting and purging should be done by the memory manager. For the most part, it's none of the driver's business.

Daniel mostly suggesting to make interface more flexible for future drivers, so we won't need to re-do it later on. My version of the interface is based on what drivers need today.

Quoted 8 lines

Why do you think it's a problem to turn shmem helper into the simple generic memory manager? I don't see how it's better to have drivers duplicating the exactly same efforts and making different mistakes.

The shmem shrinker implementation is mostly based on the freedreno's shrinker and it's very easy to enable generic shrinker for VirtIO and Panfrost drivers. I think in the future freedreno and other drivers could switch to use drm shmem instead of open coding the memory management.

Yeah I think we have enough shrinkers all over drm to actually design something solid here.

There's also the i915 shrinker and some kinda shrinker in ttm too. So we are definitely past the "have 3 examples to make sure you design something solid" rule of thumb.

Quoted 6 lines
Quoted 3 lines

I also have a bit an idea that we could try to glue the shmem shrinker into ttm, at least at a very high level that's something that would make some sense.

Before gluing the shmem shrinker into ttm, the drivers should be switched to ttm? Or do you mean something else by the gluing?

No, drivers which don't need ttm shouldn't be forced to use it.

Quoted 5 lines

Perhaps it should be possible to have a common drm-shrinker helper that will do the basic-common things like tracking the eviction size and check whether BO is exported or locked, but we shouldn't consider doing this for now. For the starter more reasonable should be to create a common shrinker base for drivers that use drm-shmem, IMO.

Yeah that might be the more practical approach. But really this was just an aside, absolutely no need to worry about this for now. I just wanted to point out that there really is a lot of use for this.

Quoted 16 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

Quoted 7 lines
Quoted 4 lines

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 10 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 1 line

The drm-shmem only handles shmem pages, not the mappings of these pages.

Yeah that's why you need an evict callback into the driver. That part is clear.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#45 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 17 lines · Daniel Vetter
Quoted 17 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Quoted 5 lines
Quoted 3 lines

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Signature · 2 lines
Best regards,
Dmitry
#46 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 21 lines · Dmitry Osipenko
Quoted 20 lines · Daniel Vetter
Quoted 20 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

Quoted 6 lines
Quoted 2 lines

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.
-Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#47 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 25 lines · Daniel Vetter
Quoted 24 lines · Dmitry Osipenko
Quoted 22 lines · Daniel Vetter
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 8 lines
Quoted 3 lines

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Okay, good that we're on the same page now.

Quoted 44 lines
Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Quoted 4 lines

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.

Shouldn't be very doable or at least worth the efforts. It's userspace that manages data uploading, kernel only provides transport for the virtio-gpu commands.

Drivers are free to use the same function for both purge() and evict() callbacks if they want. Getting rid of the purge() callback creates more problems than solves, IMO.

Signature · 2 lines
Best regards,
Dmitry
#48 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 87 lines · Dmitry Osipenko
Quoted 25 lines · Daniel Vetter
Quoted 24 lines · Dmitry Osipenko
Quoted 22 lines · Daniel Vetter
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 8 lines
Quoted 3 lines

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Okay, good that we're on the same page now.

Quoted 44 lines
Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Quoted 4 lines

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.

Quoted 7 lines

Shouldn't be very doable or at least worth the efforts. It's userspace that manages data uploading, kernel only provides transport for the virtio-gpu commands.

Drivers are free to use the same function for both purge() and evict() callbacks if they want. Getting rid of the purge() callback creates more problems than solves, IMO.

Hm this still sounds pretty funny and defeats the point of purgeable/evictable buffers a bit I think. But also I guess we'd pushed this bikeshed to the max, so I think if you make ->purge optional and just call ->evict if that's not present, and document it all in the kerneldoc, then I think that's good.

I just don't think that encouraging drivers to distinguish between evict/purge is a good idea for almost all of them. -Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#49 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 97 lines · Daniel Vetter
Quoted 87 lines · Dmitry Osipenko
Quoted 25 lines · Daniel Vetter
Quoted 24 lines · Dmitry Osipenko
Quoted 22 lines · Daniel Vetter
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 8 lines
Quoted 3 lines

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Okay, good that we're on the same page now.

Quoted 44 lines
Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Quoted 4 lines

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.

Quoted 7 lines

Shouldn't be very doable or at least worth the efforts. It's userspace that manages data uploading, kernel only provides transport for the virtio-gpu commands.

Drivers are free to use the same function for both purge() and evict() callbacks if they want. Getting rid of the purge() callback creates more problems than solves, IMO.

Quoted 5 lines

Hm this still sounds pretty funny and defeats the point of purgeable/evictable buffers a bit I think. But also I guess we'd pushed this bikeshed to the max, so I think if you make ->purge optional and just call ->evict if that's not present, and document it all in the kerneldoc, then I think that's good.

This is a good enough compromise to me.

Quoted 2 lines

I just don't think that encouraging drivers to distinguish between evict/purge is a good idea for almost all of them.

Intel's shrinker checks the "madvise" status of BOs and then decides what to do based on it. Perhaps we could move the decision-making about purging to drivers and then it will be single evict() callback, but will drivers really ever need to be responsible for this decision-making or this will be an unnecessary boilerplate code in the drivers? I'll think more about this.

Thank you all for taking time to look at this patchset. I'm preparing the new version.

Signature · 2 lines
Best regards,
Dmitry
#50 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 109 lines · Dmitry Osipenko
Quoted 97 lines · Daniel Vetter
Quoted 87 lines · Dmitry Osipenko
Quoted 25 lines · Daniel Vetter
Quoted 24 lines · Dmitry Osipenko
Quoted 22 lines · Daniel Vetter
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 8 lines
Quoted 3 lines

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Okay, good that we're on the same page now.

Quoted 44 lines
Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Quoted 4 lines

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.

Quoted 7 lines

Shouldn't be very doable or at least worth the efforts. It's userspace that manages data uploading, kernel only provides transport for the virtio-gpu commands.

Drivers are free to use the same function for both purge() and evict() callbacks if they want. Getting rid of the purge() callback creates more problems than solves, IMO.

Quoted 5 lines

Hm this still sounds pretty funny and defeats the point of purgeable/evictable buffers a bit I think. But also I guess we'd pushed this bikeshed to the max, so I think if you make ->purge optional and just call ->evict if that's not present, and document it all in the kerneldoc, then I think that's good.

This is a good enough compromise to me.

Quoted 2 lines

I just don't think that encouraging drivers to distinguish between evict/purge is a good idea for almost all of them.

Quoted 6 lines

Intel's shrinker checks the "madvise" status of BOs and then decides what to do based on it. Perhaps we could move the decision-making about purging to drivers and then it will be single evict() callback, but will drivers really ever need to be responsible for this decision-making or this will be an unnecessary boilerplate code in the drivers? I'll think more about this.

tbh I wouldn't worry about details, you've convinced me that some differentiation between evict and purged makes sense. And yeah maybe drivers should have a helper to check that instead of explicit argument, but that's a bikeshed color choice which should be fairly easy to adjust later on still.

Quoted 2 lines

Thank you all for taking time to look at this patchset. I'm preparing the new version.

Cheers, Daniel

Signature · 3 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
author maintainer
#51 lore raw

Re: [PATCH v4 11/15] drm/shmem-helper: Add generic memory shrinker

Quoted 117 lines · Daniel Vetter
Quoted 109 lines · Dmitry Osipenko
Quoted 97 lines · Daniel Vetter
Quoted 87 lines · Dmitry Osipenko
Quoted 25 lines · Daniel Vetter
Quoted 24 lines · Dmitry Osipenko
Quoted 22 lines · Daniel Vetter
Quoted 13 lines
Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

I'd like to ask you to reduce the scope of the patchset and build the shrinker only for virtio-gpu. I know that I first suggested to build upon shmem helpers, but it seems that it's easier to do that in a later patchset.

The first version of the VirtIO shrinker didn't support memory eviction. Memory eviction support requires page fault handler to be aware of the evicted pages, what should we do about it? The page fault handling is a part of memory management, hence to me drm-shmem is already kinda a MM.

Hm I still don't get that part, why does that also not go through the shmem helpers?

The drm_gem_shmem_vm_ops includes the page faults handling, it's a helper by itself that is used by DRM drivers.

Quoted 7 lines

I could try to move all the shrinker logic to the VirtIO and re-invent virtio_gem_shmem_vm_ops, but what is the point of doing this for each driver if we could have it once and for all in the common drm-shmem code?

Maybe I should try to factor out all the shrinker logic from drm-shmem into a new drm-shmem-shrinker that could be shared by drivers? Will you be okay with this option?

I think we're talking past each another a bit. I'm only bringing up the purge vs eviction topic we discussed in the other subthread again.

Quoted 8 lines
Quoted 3 lines

Thomas asked to move the whole shrinker code to the VirtIO driver and I's saying that this is not a great idea to me, or am I misunderstanding the Thomas' suggestion? Thomas?

I think it was just me creating a confusion here.

fwiw I do also think that shrinker in shmem helpers makes sense, just in case that was also lost in confusion.

Okay, good that we're on the same page now.

Quoted 44 lines
Quoted 37 lines
Quoted 31 lines
Quoted 9 lines
Quoted 2 lines

I'm still confused why drivers need to know the difference between evition and purging. Or maybe I'm confused again.

Example:

If userspace uses IOV addresses, then these addresses must be kept reserved while buffer is evicted.

If BO is purged, then we don't need to retain the IOV space allocated for the purged BO.

Yeah but is that actually needed by anyone? If userspace fails to allocate another bo because of lack of gpu address space then it's very easy to handle that:

1. Make a rule that "out of gpu address space" gives you a special errno code like ENOSPC

2. If userspace gets that it walks the list of all buffers it marked as purgeable and nukes them (whether they have been evicted or not). Then it retries the bo allocation.

Alternatively you can do step 2 also directly from the bo alloc ioctl in step 1. Either way you clean up va space, and actually a lot more (you potentially nuke all buffers marked as purgeable, not just the ones that have been purged already) and only when va cleanup is actually needed

Trying to solve this problem at eviction time otoh means:
- we have this difference between eviction and purging
- it's still not complete, you still need to glue step 2 above into your driver somehow, and once step 2 above is glued in doing additional cleanup in the purge function is just duplicated logic

Quoted 5 lines

So at least in my opinion this isn't the justification we need. And we should definitely not just add that complication "in case, for the future", if we don't have a real need right now. Adding it later on is easy, removing it later on because it just gets in the way and confuses is much harder.

Quoted 6 lines

The IOVA space is only one example.

In case of the VirtIO driver, we may have two memory allocation for a BO. One is the shmem allcation in guest and the other is in host's vram. If we will only release the guest's memory on purge, then the vram will remain allocated until BO is destroyed, which unnecessarily sub-optimal.

Quoted 4 lines

Hm but why don't you just nuke the memory on the host side too when you evict? Allowing the guest memory to be swapped out while keeping the host memory allocation alive also doesn't make a lot of sense for me. Both can be recreated (I guess at least?) on swap-in.

Quoted 7 lines

Shouldn't be very doable or at least worth the efforts. It's userspace that manages data uploading, kernel only provides transport for the virtio-gpu commands.

Drivers are free to use the same function for both purge() and evict() callbacks if they want. Getting rid of the purge() callback creates more problems than solves, IMO.

Quoted 5 lines

Hm this still sounds pretty funny and defeats the point of purgeable/evictable buffers a bit I think. But also I guess we'd pushed this bikeshed to the max, so I think if you make ->purge optional and just call ->evict if that's not present, and document it all in the kerneldoc, then I think that's good.

This is a good enough compromise to me.

Quoted 2 lines

I just don't think that encouraging drivers to distinguish between evict/purge is a good idea for almost all of them.

Quoted 6 lines

Intel's shrinker checks the "madvise" status of BOs and then decides what to do based on it. Perhaps we could move the decision-making about purging to drivers and then it will be single evict() callback, but will drivers really ever need to be responsible for this decision-making or this will be an unnecessary boilerplate code in the drivers? I'll think more about this.

Quoted 5 lines

tbh I wouldn't worry about details, you've convinced me that some differentiation between evict and purged makes sense. And yeah maybe drivers should have a helper to check that instead of explicit argument, but that's a bikeshed color choice which should be fairly easy to adjust later on still.

I already reworked patches like you suggested to use single evict() cb and etc. But those are minor things, the lockings are more important. I'm now having more h/w on my hands and yesterday found that Lima driver was getting deadlock using the new dma-buf locking convention that I'm working on, so I'm now re-testing all thoroughly and will send out v6 once will be confident in it.

Signature · 2 lines
Best regards,
Dmitry
author maintainer
#52 lore raw

PATCH v4 12/15 drm/virtio: Support memory shrinking

In reply to Dmitry Osipenko

Support generic DRM SHMEM memory shrinker and add new madvise IOCTL to the VirtIO-GPU driver. Userspace (BO cache manager of Mesa driver) will mark BOs as "don't need" using the new IOCTL to let shrinker purge the marked BOs on OOM, the shrinker will also evict unpurgeable shmem BOs from memory if guest supports SWAP. Altogether this allows to prevent OOM kills of guest applications that use VirGL by lowering memory pressure.

---
 drivers/gpu/drm/virtio/virtgpu_drv.h    |  15 ++-
 drivers/gpu/drm/virtio/virtgpu_gem.c    |  46 ++++++++
 drivers/gpu/drm/virtio/virtgpu_ioctl.c  |  37 +++++++
 drivers/gpu/drm/virtio/virtgpu_kms.c    |   9 ++
 drivers/gpu/drm/virtio/virtgpu_object.c | 139 +++++++++++++++++++-----
 drivers/gpu/drm/virtio/virtgpu_plane.c  |  17 ++-
 drivers/gpu/drm/virtio/virtgpu_vq.c     |  40 +++++++
 include/uapi/drm/virtgpu_drm.h          |  14 +++
 8 files changed, 288 insertions(+), 29 deletions(-)
drivers/gpu/drm/virtio/virtgpu_drv.h +14 -1
index b2d93cb12ebf..c8918a271e1c 100644
@@ -274,7 +274,7 @@ struct virtio_gpu_fpriv {
 };
 
 /* virtgpu_ioctl.c */
-#define DRM_VIRTIO_NUM_IOCTLS 12
+#define DRM_VIRTIO_NUM_IOCTLS 13
 extern struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS];
 void virtio_gpu_create_context(struct drm_device *dev, struct drm_file *file);
 
@@ -310,6 +310,10 @@ void virtio_gpu_array_put_free(struct virtio_gpu_object_array *objs);
 void virtio_gpu_array_put_free_delayed(struct virtio_gpu_device *vgdev,
 				       struct virtio_gpu_object_array *objs);
 void virtio_gpu_array_put_free_work(struct work_struct *work);
+int virtio_gpu_array_prepare(struct virtio_gpu_device *vgdev,
+			     struct virtio_gpu_object_array *objs);
+int virtio_gpu_gem_host_mem_release(struct virtio_gpu_object *bo);
+bool virtio_gpu_gem_madvise(struct virtio_gpu_object *obj, int madv);
 
 /* virtgpu_vq.c */
 int virtio_gpu_alloc_vbufs(struct virtio_gpu_device *vgdev);
@@ -321,6 +325,8 @@ void virtio_gpu_cmd_create_resource(struct virtio_gpu_device *vgdev,
 				    struct virtio_gpu_fence *fence);
 void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
 				   struct virtio_gpu_object *bo);
+int virtio_gpu_cmd_release_resource(struct virtio_gpu_device *vgdev,
+				    struct virtio_gpu_object *bo);
 void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev,
 					uint64_t offset,
 					uint32_t width, uint32_t height,
@@ -341,6 +347,9 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,
 			      struct virtio_gpu_object *obj,
 			      struct virtio_gpu_mem_entry *ents,
 			      unsigned int nents);
+void virtio_gpu_object_detach(struct virtio_gpu_device *vgdev,
+			      struct virtio_gpu_object *obj,
+			      struct virtio_gpu_fence *fence);
 int virtio_gpu_attach_status_page(struct virtio_gpu_device *vgdev);
 int virtio_gpu_detach_status_page(struct virtio_gpu_device *vgdev);
 void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
@@ -483,4 +492,8 @@ void virtio_gpu_vram_unmap_dma_buf(struct device *dev,
 				   struct sg_table *sgt,
 				   enum dma_data_direction dir);
 
+/* virtgpu_gem_shrinker.c */
+int virtio_gpu_gem_shrinker_init(struct virtio_gpu_device *vgdev);
+void virtio_gpu_gem_shrinker_fini(struct virtio_gpu_device *vgdev);
+
 #endif
drivers/gpu/drm/virtio/virtgpu_gem.c +46 -0

49 lines not on this page · raw message

drivers/gpu/drm/virtio/virtgpu_ioctl.c +37 -0
index f8d83358d2a0..55ee9bd2098e 100644
@@ -217,6 +217,10 @@ static int virtio_gpu_execbuffer_ioctl(struct drm_device *dev, void *data,
 		ret = virtio_gpu_array_lock_resv(buflist);
 		if (ret)
 			goto out_memdup;
+
+		ret = virtio_gpu_array_prepare(vgdev, buflist);
+		if (ret)
+			goto out_unresv;
 	}
 
 	out_fence = virtio_gpu_fence_alloc(vgdev, fence_ctx, ring_idx);
@@ -423,6 +427,10 @@ static int virtio_gpu_transfer_from_host_ioctl(struct drm_device *dev,
 	if (ret != 0)
 		goto err_put_free;
 
+	ret = virtio_gpu_array_prepare(vgdev, objs);
+	if (ret)
+		goto err_unlock;
+
 	fence = virtio_gpu_fence_alloc(vgdev, vgdev->fence_drv.context, 0);
 	if (!fence) {
 		ret = -ENOMEM;
@@ -482,6 +490,10 @@ static int virtio_gpu_transfer_to_host_ioctl(struct drm_device *dev, void *data,
 		if (ret != 0)
 			goto err_put_free;
 
+		ret = virtio_gpu_array_prepare(vgdev, objs);
+		if (ret)
+			goto err_unlock;
+
 		ret = -ENOMEM;
 		fence = virtio_gpu_fence_alloc(vgdev, vgdev->fence_drv.context,
 					       0);
@@ -836,6 +848,28 @@ static int virtio_gpu_context_init_ioctl(struct drm_device *dev,
 	return ret;
 }
 
+static int virtio_gpu_madvise_ioctl(struct drm_device *dev,
+				    void *data,
+				    struct drm_file *file)
+{
+	struct drm_virtgpu_madvise *args = data;
+	struct virtio_gpu_object *bo;
+	struct drm_gem_object *obj;
+
+	if (args->madv > VIRTGPU_MADV_DONTNEED)
+		return -EOPNOTSUPP;
+
+	obj = drm_gem_object_lookup(file, args->bo_handle);
+	if (!obj)
+		return -ENOENT;
+
+	bo = gem_to_virtio_gpu_obj(obj);
+	args->retained = virtio_gpu_gem_madvise(bo, args->madv);
+	drm_gem_object_put(obj);
+
+	return 0;
+}
+
 struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS] = {
 	DRM_IOCTL_DEF_DRV(VIRTGPU_MAP, virtio_gpu_map_ioctl,
 			  DRM_RENDER_ALLOW),
@@ -875,4 +909,7 @@ struct drm_ioctl_desc virtio_gpu_ioctls[DRM_VIRTIO_NUM_IOCTLS] = {
 
 	DRM_IOCTL_DEF_DRV(VIRTGPU_CONTEXT_INIT, virtio_gpu_context_init_ioctl,
 			  DRM_RENDER_ALLOW),
+
+	DRM_IOCTL_DEF_DRV(VIRTGPU_MADVISE, virtio_gpu_madvise_ioctl,
+			  DRM_RENDER_ALLOW),
 };
drivers/gpu/drm/virtio/virtgpu_kms.c +9 -0
index 0d1e3eb61bee..1175999acea1 100644
@@ -238,6 +238,12 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 		goto err_scanouts;
 	}
 
+	ret = drm_gem_shmem_shrinker_register(dev);
+	if (ret) {
+		DRM_ERROR("shrinker init failed\n");
+		goto err_modeset;
+	}
+
 	virtio_device_ready(vgdev->vdev);
 
 	if (num_capsets)
@@ -250,6 +256,8 @@ int virtio_gpu_init(struct virtio_device *vdev, struct drm_device *dev)
 			   5 * HZ);
 	return 0;
 
+err_modeset:
+	virtio_gpu_modeset_fini(vgdev);
 err_scanouts:
 	virtio_gpu_free_vbufs(vgdev);
 err_vbufs:
@@ -289,6 +297,7 @@ void virtio_gpu_release(struct drm_device *dev)
 	if (!vgdev)
 		return;
 
+	drm_gem_shmem_shrinker_unregister(dev);
 	virtio_gpu_modeset_fini(vgdev);
 	virtio_gpu_free_vbufs(vgdev);
 	virtio_gpu_cleanup_cap_cache(vgdev);
drivers/gpu/drm/virtio/virtgpu_object.c +113 -26

168 lines not on this page · raw message

drivers/gpu/drm/virtio/virtgpu_plane.c +15 -2
index 7148f3813d8b..2db6166a0307 100644
@@ -246,20 +246,28 @@ static int virtio_gpu_plane_prepare_fb(struct drm_plane *plane,
 	struct virtio_gpu_device *vgdev = dev->dev_private;
 	struct virtio_gpu_framebuffer *vgfb;
 	struct virtio_gpu_object *bo;
+	int err;
 
 	if (!new_state->fb)
 		return 0;
 
 	vgfb = to_virtio_gpu_framebuffer(new_state->fb);
 	bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
-	if (!bo || (plane->type == DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob))
+
+	err = drm_gem_shmem_set_unpurgeable_and_unevictable(&bo->base);
+	if (err)
+		return err;
+
+	if (plane->type == DRM_PLANE_TYPE_PRIMARY && !bo->guest_blob)
 		return 0;
 
 	if (bo->dumb && (plane->state->fb != new_state->fb)) {
 		vgfb->fence = virtio_gpu_fence_alloc(vgdev, vgdev->fence_drv.context,
 						     0);
-		if (!vgfb->fence)
+		if (!vgfb->fence) {
+			drm_gem_shmem_set_purgeable_and_evictable(&bo->base);
 			return -ENOMEM;
+		}
 	}
 
 	return 0;
@@ -269,15 +277,20 @@ static void virtio_gpu_plane_cleanup_fb(struct drm_plane *plane,
 					struct drm_plane_state *state)
 {
 	struct virtio_gpu_framebuffer *vgfb;
+	struct virtio_gpu_object *bo;
 
 	if (!state->fb)
 		return;
 
 	vgfb = to_virtio_gpu_framebuffer(state->fb);
+	bo = gem_to_virtio_gpu_obj(vgfb->base.obj[0]);
+
 	if (vgfb->fence) {
 		dma_fence_put(&vgfb->fence->f);
 		vgfb->fence = NULL;
 	}
+
+	drm_gem_shmem_set_purgeable_and_evictable(&bo->base);
 }
 
 static void virtio_gpu_cursor_plane_update(struct drm_plane *plane,
drivers/gpu/drm/virtio/virtgpu_vq.c +40 -0
index 06566e44307d..2a04dad1ae89 100644
@@ -536,6 +536,21 @@ void virtio_gpu_cmd_unref_resource(struct virtio_gpu_device *vgdev,
 		virtio_gpu_cleanup_object(bo);
 }
 
+int virtio_gpu_cmd_release_resource(struct virtio_gpu_device *vgdev,
+				    struct virtio_gpu_object *bo)
+{
+	struct virtio_gpu_resource_unref *cmd_p;
+	struct virtio_gpu_vbuffer *vbuf;
+
+	cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
+	memset(cmd_p, 0, sizeof(*cmd_p));
+
+	cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_UNREF);
+	cmd_p->resource_id = cpu_to_le32(bo->hw_res_handle);
+
+	return virtio_gpu_queue_ctrl_buffer(vgdev, vbuf);
+}
+
 void virtio_gpu_cmd_set_scanout(struct virtio_gpu_device *vgdev,
 				uint32_t scanout_id, uint32_t resource_id,
 				uint32_t width, uint32_t height,
@@ -636,6 +651,23 @@ virtio_gpu_cmd_resource_attach_backing(struct virtio_gpu_device *vgdev,
 	virtio_gpu_queue_fenced_ctrl_buffer(vgdev, vbuf, fence);
 }
 
+static void
+virtio_gpu_cmd_resource_detach_backing(struct virtio_gpu_device *vgdev,
+				       u32 resource_id,
+				       struct virtio_gpu_fence *fence)
+{
+	struct virtio_gpu_resource_attach_backing *cmd_p;
+	struct virtio_gpu_vbuffer *vbuf;
+
+	cmd_p = virtio_gpu_alloc_cmd(vgdev, &vbuf, sizeof(*cmd_p));
+	memset(cmd_p, 0, sizeof(*cmd_p));
+
+	cmd_p->hdr.type = cpu_to_le32(VIRTIO_GPU_CMD_RESOURCE_DETACH_BACKING);
+	cmd_p->resource_id = cpu_to_le32(resource_id);
+
+	virtio_gpu_queue_fenced_ctrl_buffer(vgdev, vbuf, fence);
+}
+
 static void virtio_gpu_cmd_get_display_info_cb(struct virtio_gpu_device *vgdev,
 					       struct virtio_gpu_vbuffer *vbuf)
 {
@@ -1099,6 +1131,14 @@ void virtio_gpu_object_attach(struct virtio_gpu_device *vgdev,
 					       ents, nents, NULL);
 }
 
+void virtio_gpu_object_detach(struct virtio_gpu_device *vgdev,
+			      struct virtio_gpu_object *obj,
+			      struct virtio_gpu_fence *fence)
+{
+	virtio_gpu_cmd_resource_detach_backing(vgdev, obj->hw_res_handle,
+					       fence);
+}
+
 void virtio_gpu_cursor_ping(struct virtio_gpu_device *vgdev,
 			    struct virtio_gpu_output *output)
 {
include/uapi/drm/virtgpu_drm.h +14 -0
index 0512fde5e697..12197d8e9759 100644
@@ -48,6 +48,7 @@ extern "C" {
 #define DRM_VIRTGPU_GET_CAPS  0x09
 #define DRM_VIRTGPU_RESOURCE_CREATE_BLOB 0x0a
 #define DRM_VIRTGPU_CONTEXT_INIT 0x0b
+#define DRM_VIRTGPU_MADVISE 0x0c
 
 #define VIRTGPU_EXECBUF_FENCE_FD_IN	0x01
 #define VIRTGPU_EXECBUF_FENCE_FD_OUT	0x02
@@ -196,6 +197,15 @@ struct drm_virtgpu_context_init {
 	__u64 ctx_set_params;
 };
 
+#define VIRTGPU_MADV_WILLNEED 0
+#define VIRTGPU_MADV_DONTNEED 1
+struct drm_virtgpu_madvise {
+	__u32 bo_handle;
+	__u32 retained; /* out, non-zero if BO can be used */
+	__u32 madv;
+	__u32 pad;
+};
+
 /*
  * Event code that's given when VIRTGPU_CONTEXT_PARAM_POLL_RINGS_MASK is in
  * effect.  The event size is sizeof(drm_event), since there is no additional
@@ -246,6 +256,10 @@ struct drm_virtgpu_context_init {
 	DRM_IOWR(DRM_COMMAND_BASE + DRM_VIRTGPU_CONTEXT_INIT,		\
 		struct drm_virtgpu_context_init)
 
+#define DRM_IOCTL_VIRTGPU_MADVISE \
+	DRM_IOWR(DRM_COMMAND_BASE + DRM_VIRTGPU_MADVISE, \
+		 struct drm_virtgpu_madvise)
+
 #if defined(__cplusplus)
 }
 #endif
Signature · 1 line
2.35.1
author maintainer
#53 lore raw

PATCH v4 13/15 drm/panfrost: Switch to generic memory shrinker

In reply to Dmitry Osipenko

Replace Panfrost's memory shrinker with a generic DRM SHMEM memory shrinker.

---
 drivers/gpu/drm/panfrost/Makefile             |   1 -
 drivers/gpu/drm/panfrost/panfrost_device.h    |   4 -
 drivers/gpu/drm/panfrost/panfrost_drv.c       |  19 +--
 drivers/gpu/drm/panfrost/panfrost_gem.c       |  30 +++--
 drivers/gpu/drm/panfrost/panfrost_gem.h       |   9 --
 .../gpu/drm/panfrost/panfrost_gem_shrinker.c  | 122 ------------------
 drivers/gpu/drm/panfrost/panfrost_job.c       |  18 ++-
 7 files changed, 39 insertions(+), 164 deletions(-)
 delete mode 100644 drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
drivers/gpu/drm/panfrost/Makefile +0 -1
index b71935862417..ecf0864cb515 100644
@@ -5,7 +5,6 @@ panfrost-y := \
 	panfrost_device.o \
 	panfrost_devfreq.o \
 	panfrost_gem.o \
-	panfrost_gem_shrinker.o \
 	panfrost_gpu.o \
 	panfrost_job.o \
 	panfrost_mmu.o \
drivers/gpu/drm/panfrost/panfrost_device.h +0 -4
index 8b25278f34c8..fe04b21fc044 100644
@@ -115,10 +115,6 @@ struct panfrost_device {
 		atomic_t pending;
 	} reset;
 
-	struct mutex shrinker_lock;
-	struct list_head shrinker_list;
-	struct shrinker shrinker;
-
 	struct panfrost_devfreq pfdevfreq;
 };
 
drivers/gpu/drm/panfrost/panfrost_drv.c +2 -17
index 7fcbc2a5b6cd..57a93555813f 100644
@@ -160,7 +160,6 @@ panfrost_lookup_bos(struct drm_device *dev,
 			break;
 		}
 
-		atomic_inc(&bo->gpu_usecount);
 		job->mappings[i] = mapping;
 	}
 
@@ -391,7 +390,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,
 {
 	struct panfrost_file_priv *priv = file_priv->driver_priv;
 	struct drm_panfrost_madvise *args = data;
-	struct panfrost_device *pfdev = dev->dev_private;
 	struct drm_gem_object *gem_obj;
 	struct panfrost_gem_object *bo;
 	int ret = 0;
@@ -404,7 +402,6 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,
 
 	bo = to_panfrost_bo(gem_obj);
 
-	mutex_lock(&pfdev->shrinker_lock);
 	mutex_lock(&bo->mappings.lock);
 	if (args->madv == PANFROST_MADV_DONTNEED) {
 		struct panfrost_gem_mapping *first;
@@ -430,17 +427,8 @@ static int panfrost_ioctl_madvise(struct drm_device *dev, void *data,
 
 	args->retained = drm_gem_shmem_madvise(&bo->base, args->madv);
 
-	if (args->retained) {
-		if (args->madv == PANFROST_MADV_DONTNEED)
-			list_add_tail(&bo->base.madv_list,
-				      &pfdev->shrinker_list);
-		else if (args->madv == PANFROST_MADV_WILLNEED)
-			list_del_init(&bo->base.madv_list);
-	}
-
 out_unlock_mappings:
 	mutex_unlock(&bo->mappings.lock);
-	mutex_unlock(&pfdev->shrinker_lock);
 
 	drm_gem_object_put(gem_obj);
 	return ret;
@@ -571,9 +559,6 @@ static int panfrost_probe(struct platform_device *pdev)
 	ddev->dev_private = pfdev;
 	pfdev->ddev = ddev;
 
-	mutex_init(&pfdev->shrinker_lock);
-	INIT_LIST_HEAD(&pfdev->shrinker_list);
-
 	err = panfrost_device_init(pfdev);
 	if (err) {
 		if (err != -EPROBE_DEFER)
@@ -595,7 +580,7 @@ static int panfrost_probe(struct platform_device *pdev)
 	if (err < 0)
 		goto err_out1;
 
-	panfrost_gem_shrinker_init(ddev);
+	drm_gem_shmem_shrinker_register(ddev);
 
 	return 0;
 
@@ -613,8 +598,8 @@ static int panfrost_remove(struct platform_device *pdev)
 	struct panfrost_device *pfdev = platform_get_drvdata(pdev);
 	struct drm_device *ddev = pfdev->ddev;
 
+	drm_gem_shmem_shrinker_unregister(ddev);
 	drm_dev_unregister(ddev);
-	panfrost_gem_shrinker_cleanup(ddev);
 
 	pm_runtime_get_sync(pfdev->dev);
 	pm_runtime_disable(pfdev->dev);
drivers/gpu/drm/panfrost/panfrost_gem.c +20 -10
index 293e799e2fe8..b4a7ea7c8f00 100644
@@ -19,16 +19,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)
 	struct panfrost_gem_object *bo = to_panfrost_bo(obj);
 	struct panfrost_device *pfdev = obj->dev->dev_private;
 
-	/*
-	 * Make sure the BO is no longer inserted in the shrinker list before
-	 * taking care of the destruction itself. If we don't do that we have a
-	 * race condition between this function and what's done in
-	 * panfrost_gem_shrinker_scan().
-	 */
-	mutex_lock(&pfdev->shrinker_lock);
-	list_del_init(&bo->base.madv_list);
-	mutex_unlock(&pfdev->shrinker_lock);
-
 	/*
 	 * If we still have mappings attached to the BO, there's a problem in
 	 * our refcounting.
@@ -195,6 +185,22 @@ static int panfrost_gem_pin(struct drm_gem_object *obj)
 	return drm_gem_shmem_pin(&bo->base);
 }
 
+static unsigned long panfrost_gem_purge(struct drm_gem_object *obj)
+{
+	struct drm_gem_shmem_object *shmem = to_drm_gem_shmem_obj(obj);
+	struct panfrost_gem_object *bo = to_panfrost_bo(obj);
+
+	if (!mutex_trylock(&bo->mappings.lock))
+		return 0;
+
+	panfrost_gem_teardown_mappings_locked(bo);
+	drm_gem_shmem_purge_locked(shmem);
+
+	mutex_unlock(&bo->mappings.lock);
+
+	return obj->size >> PAGE_SHIFT;
+}
+
 static const struct drm_gem_object_funcs panfrost_gem_funcs = {
 	.free = panfrost_gem_free_object,
 	.open = panfrost_gem_open,
@@ -207,6 +213,7 @@ static const struct drm_gem_object_funcs panfrost_gem_funcs = {
 	.vunmap = drm_gem_shmem_object_vunmap,
 	.mmap = drm_gem_shmem_object_mmap,
 	.vm_ops = &drm_gem_shmem_vm_ops,
+	.purge = panfrost_gem_purge,
 };
 
 /**
@@ -266,6 +273,9 @@ panfrost_gem_create_with_handle(struct drm_file *file_priv,
 	if (ret)
 		return ERR_PTR(ret);
 
+	if (!bo->is_heap)
+		drm_gem_shmem_set_purgeable(shmem);
+
 	return bo;
 }
 
drivers/gpu/drm/panfrost/panfrost_gem.h +0 -9
index 8088d5fd8480..09da064f1c07 100644
@@ -30,12 +30,6 @@ struct panfrost_gem_object {
 		struct mutex lock;
 	} mappings;
 
-	/*
-	 * Count the number of jobs referencing this BO so we don't let the
-	 * shrinker reclaim this object prematurely.
-	 */
-	atomic_t gpu_usecount;
-
 	bool noexec		:1;
 	bool is_heap		:1;
 };
@@ -84,7 +78,4 @@ panfrost_gem_mapping_get(struct panfrost_gem_object *bo,
 void panfrost_gem_mapping_put(struct panfrost_gem_mapping *mapping);
 void panfrost_gem_teardown_mappings_locked(struct panfrost_gem_object *bo);
 
-void panfrost_gem_shrinker_init(struct drm_device *dev);
-void panfrost_gem_shrinker_cleanup(struct drm_device *dev);
-
 #endif /* __PANFROST_GEM_H__ */
drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c deleted +0 -122

122 lines not on this page · raw message

drivers/gpu/drm/panfrost/panfrost_job.c +17 -1
index fda5871aebe3..bcf496b837ce 100644
@@ -271,6 +271,19 @@ static void panfrost_attach_object_fences(struct drm_gem_object **bos,
 		dma_resv_add_fence(bos[i]->resv, fence, DMA_RESV_USAGE_WRITE);
 }
 
+static int panfrost_objects_prepare(struct drm_gem_object **bos, int bo_count)
+{
+	struct panfrost_gem_object *bo;
+	int ret = 0;
+
+	while (!ret && bo_count--) {
+		bo = to_panfrost_bo(bos[bo_count]);
+		ret = bo->base.madv ? -ENOMEM : 0;
+	}
+
+	return ret;
+}
+
 int panfrost_job_push(struct panfrost_job *job)
 {
 	struct panfrost_device *pfdev = job->pfdev;
@@ -282,6 +295,10 @@ int panfrost_job_push(struct panfrost_job *job)
 	if (ret)
 		return ret;
 
+	ret = panfrost_objects_prepare(job->bos, job->bo_count);
+	if (ret)
+		goto unlock;
+
 	mutex_lock(&pfdev->sched_lock);
 	drm_sched_job_arm(&job->base);
 
@@ -323,7 +340,6 @@ static void panfrost_job_cleanup(struct kref *ref)
 			if (!job->mappings[i])
 				break;
 
-			atomic_dec(&job->mappings[i]->obj->gpu_usecount);
 			panfrost_gem_mapping_put(job->mappings[i]);
 		}
 		kvfree(job->mappings);
Signature · 1 line
2.35.1
author maintainer
#54 lore raw

PATCH v4 14/15 drm/shmem-helper: Make drm_gem_shmem_get_pages() private

In reply to Dmitry Osipenko

VirtIO-GPU driver was the only user of drm_gem_shmem_get_pages() and it now uses drm_gem_shmem_get_pages_sgt(). Make the get_pages() private to drm_gem_shmem_helper.

---
 drivers/gpu/drm/drm_gem_shmem_helper.c | 3 +--
 include/drm/drm_gem_shmem_helper.h     | 1 -
 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +1 -2
index 3838fb8d6f3a..6d83d11211fc 100644
@@ -537,7 +537,7 @@ static int drm_gem_shmem_get_pages_locked(struct drm_gem_shmem_object *shmem)
  * Returns:
  * 0 on success or a negative error code on failure.
  */
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
+static int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
 {
 	int ret;
 
@@ -554,7 +554,6 @@ int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem)
 
 	return ret;
 }
-EXPORT_SYMBOL(drm_gem_shmem_get_pages);
 
 static void drm_gem_shmem_get_pages_no_fail(struct drm_gem_shmem_object *shmem)
 {
include/drm/drm_gem_shmem_helper.h +0 -1
index a65557b446e6..84316c281292 100644
@@ -144,7 +144,6 @@ struct drm_gem_shmem_object {
 struct drm_gem_shmem_object *drm_gem_shmem_create(struct drm_device *dev, size_t size);
 void drm_gem_shmem_free(struct drm_gem_shmem_object *shmem);
 
-int drm_gem_shmem_get_pages(struct drm_gem_shmem_object *shmem);
 void drm_gem_shmem_put_pages(struct drm_gem_shmem_object *shmem);
 int drm_gem_shmem_pin(struct drm_gem_shmem_object *shmem);
 void drm_gem_shmem_unpin(struct drm_gem_shmem_object *shmem);
Signature · 1 line
2.35.1
author maintainer
#55 lore raw

PATCH v4 15/15 drm/shmem-helper: Remove drm_gem_shmem_purge()

In reply to Dmitry Osipenko

The drm_gem_shmem_purge() was added back in 2019 and never had a user since then. GEM's purging is now managed by the generic shrinker and only the "locked" variant of the function is wanted. Remove the obsoleted function.

---
 drivers/gpu/drm/drm_gem_shmem_helper.c | 11 -----------
 include/drm/drm_gem_shmem_helper.h     |  2 --
 2 files changed, 13 deletions(-)
drivers/gpu/drm/drm_gem_shmem_helper.c +0 -11
index 6d83d11211fc..3695fdb24ac8 100644
@@ -991,17 +991,6 @@ void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem)
 }
 EXPORT_SYMBOL(drm_gem_shmem_purge_locked);
 
-bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem)
-{
-	if (!dma_resv_trylock(shmem->base.resv))
-		return false;
-	drm_gem_shmem_purge_locked(shmem);
-	dma_resv_unlock(shmem->base.resv);
-
-	return true;
-}
-EXPORT_SYMBOL(drm_gem_shmem_purge);
-
 /**
  * drm_gem_shmem_dumb_create - Create a dumb shmem buffer object
  * @file: DRM file structure to create the dumb buffer for
include/drm/drm_gem_shmem_helper.h +0 -2
index 84316c281292..91e97aa829a7 100644
@@ -179,8 +179,6 @@ int drm_gem_shmem_swap_in_pages_locked(struct drm_gem_shmem_object *shmem);
 int drm_gem_shmem_swap_in_locked(struct drm_gem_shmem_object *shmem);
 
 void drm_gem_shmem_evict_locked(struct drm_gem_shmem_object *shmem);
-
-bool drm_gem_shmem_purge(struct drm_gem_shmem_object *shmem);
 void drm_gem_shmem_purge_locked(struct drm_gem_shmem_object *shmem);
 
 struct sg_table *drm_gem_shmem_get_sg_table(struct drm_gem_shmem_object *shmem);
Signature · 1 line
2.35.1