Skip to contentSkip to conversation

drm/amd/display: Pin native scanout to VRAM on large-carveout APUs

RFC PATCH In review Matthew Schwartz 5 participants 9 messages Jun 16 – Jun 25

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 drm/amd/display: Pin native scanout to VRAM on large-carveout APUs

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] The GTT fallback logic fails to handle `-EINVAL` for shared framebuffers and `-ENOSPC` when VRAM…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] The GTT fallback logic fails to handle -EINVAL for shared framebuffers and -ENOSPC when VRAM is full, breaking atomic commits.
- [High] Unconditional pinning to VRAM causes severe CPU thrashing and dmesg spam on every page flip under memory pressure. --

commit 56a58731a1850c7efcd10ad3db0b21ff08dd0a48
Author: Matthew Schwartz <[email protected]>
Subject: drm/amd/display: Pin native scanout to VRAM on large-carveout APUs

This patch modifies the framebuffer preparation logic to pin native scanout buffers in VRAM for APUs with large carveouts. If the VRAM allocation fails, it falls back to GTT to ensure the swapchain remains in a single memory domain.

Quoted 19 lines
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +8 -3
index 23a9faa2ea89f..8a5d03185c479 100644
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane, · line 979

Quoted 10 lines
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +6 -0
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);

[Severity: High]
Does this unconditional VRAM pinning attempt cause severe CPU thrashing and dmesg spam on every page flip under memory pressure?

Since prepare_fb runs on every page flip for unpinned swapchain buffers, a full VRAM might cause amdgpu_bo_pin() to perform an expensive memory eviction search and log a hard error via dev_err() 60 times a second.

Additionally, if a buffer was previously placed in GTT via the fallback and unpinned, the next page flip will attempt to migrate it back to VRAM, causing an eviction storm.

drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane, · line 980

Quoted 1 line
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +1 -0
+	if (r == -ENOMEM && pin_vram_only) {

[Severity: High]
Will this strict check for -ENOMEM fail to handle -EINVAL and -ENOSPC, causing the VRAM-to-GTT fallback to be skipped?

When VRAM is full, TTM's allocation often returns -ENOSPC instead of -ENOMEM.

If a framebuffer is shared across planes (e.g., cloned displays) and the first plane falls back to GTT, the buffer is pinned in GTT with pin_count = 1. When the second plane calls amdgpu_bo_pin() with AMDGPU_GEM_DOMAIN_VRAM, it returns -EINVAL.

Skipping the fallback in these cases would break the atomic commit.

Quoted 12 lines
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +10 -0
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#2 lore raw
In reply to Matthew Schwartz
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

That's intentional behavior.

Quoted 7 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

The display guys need to take a closer look at that, but it sounds like what we used to have before and that caused problems.

We somehow need to change the DC stuff to allow switching between VRAM and GTT frame buffers to fully fix this.

Regards,
Christian.

Quoted 95 lines

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
maintainer
#3 lore raw
Quoted 9 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Quoted 4 lines

That's intentional behavior.

Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Harry

Quoted 109 lines
Quoted 5 lines

AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

The display guys need to take a closer look at that, but it sounds like what we used to have before and that caused problems.

We somehow need to change the DC stuff to allow switching between VRAM and GTT frame buffers to fully fix this.

Regards,
Christian.

Quoted 95 lines

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
maintainer
#4 lore raw
Quoted 10 lines · Harry Wentland
Quoted 9 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Quoted 8 lines
Quoted 4 lines

That's intentional behavior.

Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Yes, exactly that.

My educated guess is that the display server then falls back to using a copy instead of a flip and that helps saving memory somehow (e.g. less scanout buffers alocated concurrently).

Would it somehow be possible to get DC to dynamically switch between VRAM and GTT?

Regards,
Christian

Quoted 113 lines

Harry

Quoted 109 lines
Quoted 5 lines

AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

The display guys need to take a closer look at that, but it sounds like what we used to have before and that caused problems.

We somehow need to change the DC stuff to allow switching between VRAM and GTT frame buffers to fully fix this.

Regards,
Christian.

Quoted 95 lines

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
maintainer
#5 lore raw
Quoted 20 lines · Christian König
Quoted 10 lines · Harry Wentland
Quoted 9 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Quoted 8 lines
Quoted 4 lines

That's intentional behavior.

Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Quoted 6 lines

Yes, exactly that.

My educated guess is that the display server then falls back to using a copy instead of a flip and that helps saving memory somehow (e.g. less scanout buffers alocated concurrently).

Would it somehow be possible to get DC to dynamically switch between VRAM and GTT?

DCN can't switch between mapped and unmapped memory. I'm not a memory management expert but wouldn't GTT be in GART (mapped) and VRAM in the (unmapped) FB aperture?

From DCHUB HW doc:
"No change from mapped to unmapped or unmapped to mapped is allowed for immediate f lip"

If so, we can't async flip between them.

Harry

Quoted 117 lines

Regards,
Christian

Quoted 113 lines

Harry

Quoted 109 lines
Quoted 5 lines

AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

The display guys need to take a closer look at that, but it sounds like what we used to have before and that caused problems.

We somehow need to change the DC stuff to allow switching between VRAM and GTT frame buffers to fully fix this.

Regards,
Christian.

Quoted 95 lines

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
#6 lore raw
Quoted 31 lines · Harry Wentland
Quoted 20 lines · Christian König
Quoted 10 lines · Harry Wentland
Quoted 9 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Quoted 8 lines
Quoted 4 lines

That's intentional behavior.

Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Quoted 5 lines

Yes, exactly that.

My educated guess is that the display server then falls back to using a copy instead of a flip and that helps saving memory somehow (e.g. less scanout buffers alocated concurrently).

Would it somehow be possible to get DC to dynamically switch between VRAM and GTT?

DCN can't switch between mapped and unmapped memory. I'm not a memory management expert but wouldn't GTT be in GART (mapped) and VRAM in the (unmapped) FB aperture?

Quoted 5 lines

From DCHUB HW doc:
"No change from mapped to unmapped or unmapped to mapped is allowed for immediate f lip"

If so, we can't async flip between them.

A possible alternative solution then might be explicitly pinning the BO to either VRAM or GTT for an async commit, wherever the current scanout BO is located.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast
maintainer
#7 lore raw
In reply to Harry Wentland

Adding a few more people on CC.

Quoted 23 lines · Harry Wentland
Quoted 22 lines · Christian König
Quoted 10 lines · Harry Wentland
Quoted 9 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

Quoted 8 lines
Quoted 4 lines

That's intentional behavior.

Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Yes, exactly that.

Quoted 8 lines
Quoted 4 lines

My educated guess is that the display server then falls back to using a copy instead of a flip and that helps saving memory somehow (e.g. less scanout buffers alocated concurrently).

Would it somehow be possible to get DC to dynamically switch between VRAM and GTT?

DCN can't switch between mapped and unmapped memory. I'm not a memory management expert but wouldn't GTT be in GART (mapped) and VRAM in the (unmapped) FB aperture?

Well at the moment yes, but that is changeable.

It's correct that for scanout we currently access GTT buffers "mapped" through the GART while VRAM buffers are accessed "unmapped" through the FB aperture.

As far as I know starting with I think Vega or Navi 1x (or maybe Navi 2x/3x you probably know that better than me) the DCN block became capable of scanning out through the GART as well, which is a prerequisite that we can allocate framebuffers on APUs in GTT in the first place.

So what we could do in theory is to allocate a window in the GART address space, map the VRAM buffer into it and then start scanning out from that address instead.

We already do that for a couple of workarounds on older HW and to copy TMZ buffers etc... So most of the infrastructure necessary is now already there.

This would also reduce the pressure on the VRAM manager backend because we don't need to allocate those buffers contigiously any more. Leading to less pin failed with -12 errors when memory becomes tight.

The only problem I see is that I don't know if this doesn't have any negative effect on the DCN power consumption or something like that? That's where I need input from the display team.

Additional to that we need to find somebody who implements/tests that stuff, but maybe Timur or somebody else from Valve could take that since I think.

Regards,
Christian.

Quoted 127 lines

From DCHUB HW doc:
"No change from mapped to unmapped or unmapped to mapped is allowed for immediate f lip"

If so, we can't async flip between them.

Harry

Quoted 117 lines

Regards,
Christian

Quoted 113 lines

Harry

Quoted 109 lines
Quoted 5 lines

AMDGPU_SG_THRESHOLD, so small-carveout parts keep their existing VRAM|GTT placement, and fall back to GTT when the buffer does not fit in VRAM, so the flip still succeeds and the swapchain stays in one domain. Imported buffers may only be pinnable in GTT, so leave those on the default domains.

The display guys need to take a closer look at that, but it sounds like what we used to have before and that caused problems.

We somehow need to change the DC stuff to allow switching between VRAM and GTT frame buffers to fully fix this.

Regards,
Christian.

Quoted 95 lines

---
Hi,

This came up while testing my kernel patch to fix mem_type detection for
async flips here: https://lore.kernel.org/amd-gfx/[email protected]/

I found a new issue where splitting a swapchain between VRAM and GTT causes a noticeable stutter in gameplay if gamescope is using direct scanout and tearing is enabled while a game is already running.

Once a swapchain is split across the VRAM carveout and GTT, the scanout buffer's mem_type changes from one flip to the next, so amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Under direct scanout with tearing that rejection recurs every time the displayed buffer crosses domains, which is what surfaces as the choppiness.

With this patch, I can enable tearing on top of an already-disabled frame limit mid-game and no longer reproduce the choppiness.

amdgpu_gem_info confirms the swapchain converges to a single domain instead of splitting across VRAM and GTT.

Before:

0x00000f81:      3981312 byte GTT exported as ino:275 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88248 signalled
0x00000f82:      3981312 byte GTT exported as ino:276 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC     write fence:drm_sched gfx_0.0.0 seq 88224 signalled
0x00000f83:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:277 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 88236 signalled

After:

0x00000f82:      3981312 byte VRAM VISIBLE pin count 1 exported as ino:548 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC        write fence:drm_sched gfx_0.0.0 seq 822258 signalled
0x00000f83:      3981312 byte VRAM VISIBLE exported as ino:549 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822255 signalled
0x00000f84:      3981312 byte VRAM VISIBLE exported as ino:550 NO_CPU_ACCESS CPU_GTT_USWC VRAM_CLEARED VRAM_CONTIGUOUS EXPLICIT_SYNC    write fence:drm_sched gfx_0.0.0 seq 822261 signalled

Does this seem like the correct approach to take for fixing the observed issue? I wanted to start with an RFC to make sure I didn't overlook anything obvious or miss any better methods of fixing this.

Thanks,
Matt

---
 .../amd/display/amdgpu_dm/amdgpu_dm_plane.c   | 29 +++++++++++++++++--
 1 file changed, 26 insertions(+), 3 deletions(-)
drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +26 -3
index 23a9faa2ea89..b99f938e58ec 100644
@@ -932,6 +932,7 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	struct amdgpu_bo *rbo;
 	struct dm_plane_state *dm_plane_state_new, *dm_plane_state_old;
 	uint32_t domain;
+	bool pin_vram_only;
 	int r;
 
 	if (!new_state->fb) {
@@ -958,13 +959,35 @@ static int amdgpu_dm_plane_helper_prepare_fb(struct drm_plane *plane,
 	if (r)
 		goto error_unlock;
 
-	if (plane->type != DRM_PLANE_TYPE_CURSOR)
-		domain = amdgpu_display_supported_domains(adev, rbo->flags);
-	else
+	/*
+	 * Pin native scanout in VRAM on APUs so a swapchain stays in one
+	 * memory domain. A VRAM/GTT split changes its mem_type between flips
+	 * and amdgpu_dm_crtc_mem_type_changed() rejects the async flip. Skip
+	 * small carveouts that may not fit, and imported buffers.
+	 */
+	pin_vram_only = plane->type != DRM_PLANE_TYPE_CURSOR &&
+			(adev->flags & AMD_IS_APU) &&
+			!rbo->tbo.base.import_attach &&
+			adev->gmc.real_vram_size > AMDGPU_SG_THRESHOLD;
+
+	if (plane->type == DRM_PLANE_TYPE_CURSOR || pin_vram_only)
 		domain = AMDGPU_GEM_DOMAIN_VRAM;
+	else
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
 
 	rbo->flags |= AMDGPU_GEM_CREATE_VRAM_CONTIGUOUS;
 	r = amdgpu_bo_pin(rbo, domain);
+	if (r == -ENOMEM && pin_vram_only) {
+		/*
+		 * VRAM could not fit the buffer. Fall back to GTT where
+		 * allowed so the swapchain stays in one domain.
+		 */
+		domain = amdgpu_display_supported_domains(adev, rbo->flags);
+		if (domain & AMDGPU_GEM_DOMAIN_GTT) {
+			domain = AMDGPU_GEM_DOMAIN_GTT;
+			r = amdgpu_bo_pin(rbo, domain);
+		}
+	}
 	if (unlikely(r != 0)) {
 		if (r != -ERESTARTSYS)
 			DRM_ERROR("Failed to pin framebuffer with error %d\n", r);
#8 lore raw
In reply to Christian König
Quoted 13 lines · Christian König
Quoted 12 lines · Harry Wentland
Quoted 11 lines · Christian König
Quoted 7 lines · Matthew Schwartz

Native scanout buffers on APUs are pinned with the VRAM|GTT domain, so under VRAM carveout pressure a swapchain can end up split across VRAM and GTT. The scanout buffer's memory type then changes from one flip to the next, and amdgpu_dm_crtc_mem_type_changed() rejects an async page flip across the change. The result is repeated async page flip failures, observed as choppy updates under carveout pressure, until the buffers reconverge to a single domain.

That's intentional behavior.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

Pin native scanout buffers in VRAM only so the swapchain stays in one memory domain. Restrict this to APUs whose carveout is larger than

Above you mention that under VRAM pressure a swapchain can end up split across VRAM and GTT. Wouldn't restricting the swapchain to VRAM now mean that in those cases you fail to allocate the swapchain entirely?

Yes, exactly that.

This doesn't affect swapchain allocation, does it? It only affects whether or not an atomic commit succeeds or fails.

Quoted 1 line

My educated guess is that the display server then falls back to using a copy instead of a flip and that helps saving memory somehow (e.g. less scanout buffers alocated concurrently).

If an atomic commit fails (e.g. because the buffer can't be pinned to VRAM), I'd expect a Wayland compositor to first fall back from direct scanout of a client buffer to compositing, or if it's for a composited buffer, from an async commit to a non-async one.

Xorg can further fall back to copying from the new buffer to the current scanout buffer, not sure any Wayland compositor does that though. Mutter doesn't, I suspect it would freeze if a non-async commit to a composited buffer fails.

Signature · 2 lines
Earthling Michel Dänzer       \        GNOME / Xwayland / Mesa developer
https://redhat.com             \               Libre software enthusiast