drm/vram-helper: Reduce memory fragmentation
Subsystems (2)
Thomas Zimmermann
maintainercover drm/vram-helper: Reduce memory fragmentation
VRAM memory can easily fragment, which leads to OOM errors on devices with little VRAM available (i.e., all of them). This patchset addresses the problem by placing BOs near the bottom and top ends of the VRAM and keeping available areas near the middle.
Thomas Zimmermann (2):
drm/vram-helper: Don't put new BOs into VRAM drm/vram-helper: Alternate between bottom-up and top-down placement
drivers/gpu/drm/drm_gem_vram_helper.c | 30 +++++++++++++++++++++------ include/drm/drm_gem_vram_helper.h | 10 +++++---- 2 files changed, 30 insertions(+), 10 deletions(-)
--
2.26.0
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerPATCH 1/2 drm/vram-helper: Don't put new BOs into VRAM
Most drivers that use VRAM helpers have only a few MiB of framebuffer memory available. To reduce fragmentation, new BOs are now put into system memory by default. Only pin operations are allowed to move BOs into VRAM.
--- drivers/gpu/drm/drm_gem_vram_helper.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/drm_gem_vram_helper.c +1 -1
@@ -197,7 +197,7 @@ static int drm_gem_vram_init(struct drm_device *dev, acc_size =ttm_bo_dma_acc_size (bdev, size,sizeof (*gbo)); gbo->bo.bdev = bdev; -drm_gem_vram_placement (gbo,TTM_PL_FLAG_VRAM |TTM_PL_FLAG_SYSTEM ); +drm_gem_vram_placement (gbo,TTM_PL_FLAG_SYSTEM ); ret = ttm_bo_init(bdev, &gbo->bo, size, ttm_bo_type_device, &gbo->placement, pg_align, false, acc_size,
Signature · 6 lines
2.26.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerPATCH 2/2 drm/vram-helper: Alternate between bottom-up and top-down placement
In reply to Thomas ZimmermannWith limited VRAM available, fragmentation can lead to OOM errors. Alternating between bottom-up and top-down placement keeps BOs near the ends of the VRAM and the available pages consecutively near the middle.
A real-world example with 16 MiB of VRAM is shown below.
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm
0x0000000000000000-0x000000000000057f: 1407: free 0x000000000000057f-0x0000000000000b5b: 1500: used 0x0000000000000b5b-0x0000000000000ff0: 1173: free
The first free area was the location of the fbdev framebuffer. The used area is Weston's current framebuffer of 1500 pages. Weston now cannot do a pageflip to another 1500 page-wide framebuffer, even though enough pages are available. The patch resolves this problem to
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm
0x0000000000000000-0x00000000000005dc: 1500: used 0x00000000000005dc-0x0000000000000a14: 1080: free 0x0000000000000a14-0x0000000000000ff0: 1500: used
with both of Weston's framebuffers located near the ends of the VRAM memory.
--- drivers/gpu/drm/drm_gem_vram_helper.c | 28 ++++++++++++++++++++++----- include/drm/drm_gem_vram_helper.h | 10 ++++++---- 2 files changed, 29 insertions(+), 9 deletions(-)
drivers/gpu/drm/drm_gem_vram_helper.c +23 -5
@@ -146,15 +146,33 @@ static void drm_gem_vram_placement(struct drm_gem_vram_object *gbo,unsigned int i;unsigned int c =0 ;u32 invariant_flags = pl_flag &TTM_PL_FLAG_TOPDOWN ; +struct drm_device *dev = gbo->bo.base.dev; +struct drm_vram_mm *vmm = dev->vram_mm; gbo->placement.placement = gbo->placements; gbo->placement.busy_placement = gbo->placements; -if (pl_flag &TTM_PL_FLAG_VRAM ) - gbo->placements[c++].flags =TTM_PL_FLAG_WC | -TTM_PL_FLAG_UNCACHED | -TTM_PL_FLAG_VRAM | - invariant_flags; +if (pl_flag &TTM_PL_FLAG_VRAM ) { +/* +* We usually have at most 2 pinned BOs during pageflips, +* plus cursor BOs. Even with a significant number of free +' pages, always placing bottom-up can lead to fragmentation +* and OOM situations. So if there's no explicit request for +* top-down placement, we alternatingly place BOs bottom-up +* and top-down. The placement strategy should help to keep +* free VRAM pages available near the middle of the VRAM. +*/ + gbo->placements[c].flags =TTM_PL_FLAG_WC | +TTM_PL_FLAG_UNCACHED | +TTM_PL_FLAG_VRAM | + invariant_flags; +if (!(invariant_flags &TTM_PL_FLAG_TOPDOWN )) { +if (vmm->place_topdown) + gbo->placements[c].flags |=TTM_PL_FLAG_TOPDOWN ; + vmm->place_topdown = !vmm->place_topdown; + } + ++c; + }if (pl_flag &TTM_PL_FLAG_SYSTEM ) gbo->placements[c++].flags = TTM_PL_MASK_CACHING |
include/drm/drm_gem_vram_helper.h +6 -4
@@ -165,10 +165,10 @@ void drm_gem_vram_simple_display_pipe_cleanup_fb(/** * struct drm_vram_mm - An instance of VRAM MM -* @vram_base: Base address of the managed video memory -* @vram_size: Size of the managed video memory in bytes -* @bdev: The TTM BO device. -* @funcs: TTM BO functions +* @vram_base: Base address of the managed video memory +* @vram_size: Size of the managed video memory in bytes +* @bdev: The TTM BO device +* @place_topdown: Flags the next BO to be placed at the VRAM's high end * * The fields &struct drm_vram_mm.vram_base and * &struct drm_vram_mm.vrm_size are managed by VRAM MM, but are
@@ -180,6 +180,8 @@ struct drm_vram_mm {size_t vram_size;struct ttm_bo_device bdev; + +bool place_topdown; };/**
Signature · 6 lines
2.26.0 _______________________________________________ dri-devel mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Quoted 16 lines · Thomas Zimmermann
With limited VRAM available, fragmentation can lead to OOM errors. Alternating between bottom-up and top-down placement keeps BOs near the ends of the VRAM and the available pages consecutively near the middle.
A real-world example with 16 MiB of VRAM is shown below.
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm0x0000000000000000-0x000000000000057f: 1407: free 0x000000000000057f-0x0000000000000b5b: 1500: used 0x0000000000000b5b-0x0000000000000ff0: 1173: free
The first free area was the location of the fbdev framebuffer. The used area is Weston's current framebuffer of 1500 pages. Weston now cannot do a pageflip to another 1500 page-wide framebuffer, even though enough pages are available. The patch resolves this problem to
Quoted 7 lines
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm0x0000000000000000-0x00000000000005dc: 1500: used 0x00000000000005dc-0x0000000000000a14: 1080: free 0x0000000000000a14-0x0000000000000ff0: 1500: used
with both of Weston's framebuffers located near the ends of the VRAM memory.
I don't think it is that simple.
First: How will that interact with cursor bo allocations? IIRC the strategy for them is to allocate top-down, for similar reasons (avoid small cursor bo allocs fragment vram memory).
Second: I think ttm will move bo's from vram to system only on memory pressure. So you can still end up with fragmented memory. To make the scheme with one fb @ top and one @ bottom work reliable you have to be more aggressive on pushing out framebuffers.
Third: I'd suggest make topdown allocations depending on current state instead of simply alternating, i.e. if there is a pinned framebuffer @ offset 0, then go for top-down.
I also think using this scheme should be optional. In the simplest case we can allow drivers opt-in. Or we try do to something clever
automatically: using the strategy only for framebuffers larger than 1/4 or 1/3 of total vram memory (which makes alloc failures due to fragmentation much more likely).
cheers,
Gerd
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerRe: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi Gerd,
thanks for the feedback.
Am 23.04.20 um 13:18 schrieb Gerd Hoffmann:
Quoted 22 lines · Thomas Zimmermann
Quoted 21 lines
With limited VRAM available, fragmentation can lead to OOM errors. Alternating between bottom-up and top-down placement keeps BOs near the ends of the VRAM and the available pages consecutively near the middle.
A real-world example with 16 MiB of VRAM is shown below.
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm0x0000000000000000-0x000000000000057f: 1407: free 0x000000000000057f-0x0000000000000b5b: 1500: used 0x0000000000000b5b-0x0000000000000ff0: 1173: free
The first free area was the location of the fbdev framebuffer. The used area is Weston's current framebuffer of 1500 pages. Weston now cannot do a pageflip to another 1500 page-wide framebuffer, even though enough pages are available. The patch resolves this problem to
Quoted 1 line
cat
/sys/kernel/debug/dri/0/vram-mm0x0000000000000000-0x00000000000005dc: 1500: used 0x00000000000005dc-0x0000000000000a14: 1080: free 0x0000000000000a14-0x0000000000000ff0: 1500: used
Quoted 8 lines
Quoted 2 lines
with both of Weston's framebuffers located near the ends of the VRAM memory.
I don't think it is that simple.
First: How will that interact with cursor bo allocations? IIRC the strategy for them is to allocate top-down, for similar reasons (avoid small cursor bo allocs fragment vram memory).
In ast, 2 cursor BOs are allocated during driver initialization and kept permanently at the vram's top end. I don't know about other drivers.
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
Quoted 5 lines
Second: I think ttm will move bo's from vram to system only on memory pressure. So you can still end up with fragmented memory. To make the scheme with one fb @ top and one @ bottom work reliable you have to be more aggressive on pushing out framebuffers.
I'm the process of converting mgag200 to atomic modesetting. The given example is what I observed. I'm not claiming that the placement scheme is perfect, but it is required to get mgag200 working with atomic modesetting's pageflip logic. So we're solving a real problem here.
The bug comes from Weston's allocation strategy. Looking at the debug output:
Quoted 1 line
Quoted 1 line
0x0000000000000000-0x000000000000057f: 1407: free
This was fbdev's framebuffer with 1600x900@32bpp
Quoted 1 line
Quoted 1 line
0x000000000000057f-0x0000000000000b5b: 1500: used
This is Weston's framebuffer also with 1600x900@32bpp. But Weston allocates an additional, unused 60 scanlines. That is to render with tiles of 64x64px, I suppose. fbdev doesn't do that, hence Weston's second framebuffer doesn't fit into the free location of the fbdev framebuffer.
The other drivers with a small amount of vram are also prone to this problem. They simply have not yet encountered such a setup.
Quoted 4 lines
Third: I'd suggest make topdown allocations depending on current state instead of simply alternating, i.e. if there is a pinned framebuffer @ offset 0, then go for top-down.
That's what the current patch does. If the last pin was at the bottom, the next goes to the top. And then the other way around. Without alternating between both end of vram, the problem would occur again when fragmentation happens near the top end.
Quoted 6 lines
I also think using this scheme should be optional. In the simplest case we can allow drivers opt-in. Or we try do to something clever
automatically: using the strategy only for framebuffers larger than 1/4 or 1/3 of total vram memory (which makes alloc failures due to fragmentation much more likely).
I'd like to not change behavior automatically, but we can surely make this optional.
Looking again at the vram helpers, this functionality could be implemented in drm_gem_vram_plane_helper_prepare_fb(). Drivers with other placement strategies could implement their own helper for prepare_fb.
Best regards
Thomas
Quoted 4 lines
cheers,
Gerd
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Quoted 8 lines
Quoted 5 lines
I don't think it is that simple.
First: How will that interact with cursor bo allocations? IIRC the strategy for them is to allocate top-down, for similar reasons (avoid small cursor bo allocs fragment vram memory).
In ast, 2 cursor BOs are allocated during driver initialization and kept permanently at the vram's top end. I don't know about other drivers.
One-time allocation at init time shouldn't be a problem.
Quoted 3 lines
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
The problem isn't the size. The problem is dynamically allocated cursor BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.
Quoted 9 lines
Quoted 4 lines
Second: I think ttm will move bo's from vram to system only on memory pressure. So you can still end up with fragmented memory. To make the scheme with one fb @ top and one @ bottom work reliable you have to be more aggressive on pushing out framebuffers.
I'm the process of converting mgag200 to atomic modesetting. The given example is what I observed. I'm not claiming that the placement scheme is perfect, but it is required to get mgag200 working with atomic modesetting's pageflip logic. So we're solving a real problem here.
I don't doubt this is a real problem.
Quoted 7 lines
The bug comes from Weston's allocation strategy. Looking at the debug output:
Quoted 1 line
Quoted 1 line
0x0000000000000000-0x000000000000057f: 1407: free
This was fbdev's framebuffer with 1600x900@32bpp
Quoted 7 lines
Quoted 1 line
Quoted 1 line
0x000000000000057f-0x0000000000000b5b: 1500: used
This is Weston's framebuffer also with 1600x900@32bpp. But Weston allocates an additional, unused 60 scanlines. That is to render with tiles of 64x64px, I suppose. fbdev doesn't do that, hence Weston's second framebuffer doesn't fit into the free location of the fbdev framebuffer.
Sure. Assume there is just enough vram to fit in fbdev and two weston framebuffers. fbdev is allocated from bottom, first weston fb from top, second weston fb from bottom again. fbdev is not pushed out (no memory pressure yet) so the second weston fb ends up in the middle of vram fragmenting things. And now you are again in the situation where you might have enough free vram for an allocation but can't use it due to fragmention (probably harder to trigger in practice though).
That's why I would suggest to explicitly move out unpinned framebuffers (aka fbdev) before pinning a new one (second weston fb) instead of depending on ttm moving things out on OOM, to make sure you never allocate something in the middle of vram.
Quoted 8 lines
Quoted 3 lines
Third: I'd suggest make topdown allocations depending on current state instead of simply alternating, i.e. if there is a pinned framebuffer @ offset 0, then go for top-down.
That's what the current patch does. If the last pin was at the bottom, the next goes to the top. And then the other way around. Without alternating between both end of vram, the problem would occur again when fragmentation happens near the top end.
I'd feel better when checking the state of my current pins to figure whenever I should alloc top-bottom or not, for robustness reasons.
Quoted 3 lines
Looking again at the vram helpers, this functionality could be implemented in
drm_gem_vram_plane_helper_prepare_fb(). Drivers with other placement strategies could implement their own helper forprepare_fb.
vram helpers could also simply offer two prepare_fb variants.
cheers,
Gerd
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi guys,
one thing you don't seem to have considered yet is that not setting the top-down flag also won't get you the bottom-up approach, but rather the best fit.
Additional to that the top-down allocations have quite a bit more CPU overhead since they don't walk the size tree, but the address tree in search for a fitting hole. Nirmoy is currently working on improving this.
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 7 lines
Quoted 3 lines
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
The problem isn't the size. The problem is dynamically allocated cursor BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.
Well maybe instead of a ping/pong approach between top and bottom use a size based approach to place small BOs at the top and only the larger ones at the bottom (or the other way around).
Cheers,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerRe: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi Christian
Am 23.04.20 um 17:04 schrieb Christian König:
Quoted 5 lines
Hi guys,
one thing you don't seem to have considered yet is that not setting the top-down flag also won't get you the bottom-up approach, but rather the best fit.
Kind of unrelated: Would it pick the smallest free area that can hold a BO? IOW, would a small cursor BO fill up a small free area even if there's a much larger free area somewhere?
Quoted 4 lines
Additional to that the top-down allocations have quite a bit more CPU overhead since they don't walk the size tree, but the address tree in search for a fitting hole. Nirmoy is currently working on improving this.
I'm not worried about that. Compositors alternate between only two framebuffer instances. Once their BOs have been placed, they remain in vram.
Quoted 5 lines
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 3 lines
Quoted 3 lines
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
Quoted 8 lines
Quoted 4 lines
The problem isn't the size. The problem is dynamically allocated cursor BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.
Well maybe instead of a ping/pong approach between top and bottom use a size based approach to place small BOs at the top and only the larger ones at the bottom (or the other way around).
That's what the current code does and it results in the OOM. Basically, there's a large BO in the middle of vram and the pageflip's destination BO does not fit before or after the large one.
Best regards
Thomas
Quoted 7 lines
Cheers,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Am 24.04.20 um 08:27 schrieb Thomas Zimmermann:
Quoted 11 lines
Hi Christian
Am 23.04.20 um 17:04 schrieb Christian König:
Quoted 5 lines
Hi guys,
one thing you don't seem to have considered yet is that not setting the top-down flag also won't get you the bottom-up approach, but rather the best fit.
Kind of unrelated: Would it pick the smallest free area that can hold a BO? IOW, would a small cursor BO fill up a small free area even if there's a much larger free area somewhere?
Yes, exactly that.
The bottom-up method searches for the lowest hole which can fit the requirements.
The top-down method searches for the highest hole which can fit the requirements.
Both of those walk the holes by the address index tree, but there is also the best method which looks at the holes by their size and tries to find the best fit.
The best method usually only needs a single tree walk instead of a linear search.
Quoted 8 lines
Quoted 8 lines
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 7 lines
Quoted 3 lines
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
The problem isn't the size. The problem is dynamically allocated cursor BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.
Quoted 6 lines
Quoted 3 lines
Well maybe instead of a ping/pong approach between top and bottom use a size based approach to place small BOs at the top and only the larger ones at the bottom (or the other way around).
That's what the current code does and it results in the OOM. Basically, there's a large BO in the middle of vram and the pageflip's destination BO does not fit before or after the large one.
Tough problem. No idea how to otherwise fix this without making the large BO moveable somehow.
Christian.
Quoted 10 lines
Best regards
ThomasQuoted 6 lines
Cheers,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerRe: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi
Am 24.04.20 um 08:56 schrieb Christian König:
Quoted 6 lines
Am 24.04.20 um 08:27 schrieb Thomas Zimmermann:
Quoted 5 lines
Hi Christian
Am 23.04.20 um 17:04 schrieb Christian König:
Quoted 2 lines
Hi guys,
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
one thing you don't seem to have considered yet is that not setting the top-down flag also won't get you the bottom-up approach, but rather the best fit.
Kind of unrelated: Would it pick the smallest free area that can hold a BO? IOW, would a small cursor BO fill up a small free area even if there's a much larger free area somewhere?
Yes, exactly that.
I see.
Quoted 3 lines
The bottom-up method searches for the lowest hole which can fit the requirements.
But there's no way to explictely select bottom-up? What is TTM_PL_FLAG_CONTIGUOUS doing?
Best regards
Thomas
Quoted 49 lines
The top-down method searches for the highest hole which can fit the requirements.
Both of those walk the holes by the address index tree, but there is also the best method which looks at the holes by their size and tries to find the best fit.
The best method usually only needs a single tree walk instead of a linear search.
Quoted 11 lines
Quoted 11 lines
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 10 lines
Quoted 5 lines
But cursor BOs are small, so they don't make much of a difference. What
is needed is space for 2 primary framebuffers during pageflips, with one
of them pinned. The other framebuffer can be located anywhere.The problem isn't the size. The problem is dynamically allocated cursor
BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.Quoted 6 lines
Quoted 3 lines
Well maybe instead of a ping/pong approach between top and bottom use a size based approach to place small BOs at the top and only the larger ones at the bottom (or the other way around).
That's what the current code does and it results in the OOM. Basically, there's a large BO in the middle of vram and the pageflip's destination BO does not fit before or after the large one.
Tough problem. No idea how to otherwise fix this without making the large BO moveable somehow.
Christian.
Quoted 10 lines
Best regards
ThomasQuoted 6 lines
Cheers,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Am 24.04.20 um 09:03 schrieb Thomas Zimmermann:
Quoted 15 lines
Hi
Am 24.04.20 um 08:56 schrieb Christian König:
Quoted 12 lines
Am 24.04.20 um 08:27 schrieb Thomas Zimmermann:
Quoted 11 lines
Hi Christian
Am 23.04.20 um 17:04 schrieb Christian König:
Quoted 5 lines
Hi guys,
one thing you don't seem to have considered yet is that not setting the top-down flag also won't get you the bottom-up approach, but rather the best fit.
Kind of unrelated: Would it pick the smallest free area that can hold a BO? IOW, would a small cursor BO fill up a small free area even if there's a much larger free area somewhere?
Quoted 7 lines
Quoted 1 line
Yes, exactly that.
I see.
Quoted 2 lines
The bottom-up method searches for the lowest hole which can fit the requirements.
But there's no way to explictely select bottom-up? What is
TTM_PL_FLAG_CONTIGUOUSdoing?
Making sure that we allocate contiguous for amdgpu. Added that because I couldn't fix a few problems otherwise.
Should probably be a driver private placement flag instead, but didn't had time to clean that up.
Regards,
Christian.
Quoted 50 lines
Best regards
ThomasQuoted 45 lines
The top-down method searches for the highest hole which can fit the requirements.
Both of those walk the holes by the address index tree, but there is also the best method which looks at the holes by their size and tries to find the best fit.
The best method usually only needs a single tree walk instead of a linear search.
Quoted 11 lines
Quoted 11 lines
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 10 lines
Quoted 5 lines
But cursor BOs are small, so they don't make much of a difference. What
is needed is space for 2 primary framebuffers during pageflips, with one
of them pinned. The other framebuffer can be located anywhere.The problem isn't the size. The problem is dynamically allocated cursor
BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.Quoted 6 lines
Quoted 3 lines
Well maybe instead of a ping/pong approach between top and bottom use a size based approach to place small BOs at the top and only the larger ones at the bottom (or the other way around).
That's what the current code does and it results in the OOM. Basically, there's a large BO in the middle of vram and the pageflip's destination BO does not fit before or after the large one.
Tough problem. No idea how to otherwise fix this without making the large BO moveable somehow.
Christian.
Quoted 9 lines
Best regards
ThomasQuoted 6 lines
Cheers,
Christian.
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerRe: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
In reply to Gerd HoffmannHi Gerd
Am 23.04.20 um 15:57 schrieb Gerd Hoffmann:
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
I don't think it is that simple.
First: How will that interact with cursor bo allocations? IIRC the strategy for them is to allocate top-down, for similar reasons (avoid small cursor bo allocs fragment vram memory).
In ast, 2 cursor BOs are allocated during driver initialization and kept permanently at the vram's top end. I don't know about other drivers.
One-time allocation at init time shouldn't be a problem.
Quoted 8 lines
Quoted 3 lines
But cursor BOs are small, so they don't make much of a difference. What is needed is space for 2 primary framebuffers during pageflips, with one of them pinned. The other framebuffer can be located anywhere.
The problem isn't the size. The problem is dynamically allocated cursor BOs can also fragment vram, especially if top-bottom allocation is also used for large framebuffers so cursor BOs can end up somewhere in the middle of vram.
But the problem is the size. Cursor BOs are unrelated. Of the vram-helper-based drivers, only ast and mgag200 use cursors, and both perma-pin them to the end of vram.
Quoted 28 lines
Quoted 10 lines
Quoted 5 lines
Second: I think ttm will move bo's from vram to system only on memory pressure. So you can still end up with fragmented memory. To make the scheme with one fb @ top and one @ bottom work reliable you have to be more aggressive on pushing out framebuffers.
I'm the process of converting mgag200 to atomic modesetting. The given example is what I observed. I'm not claiming that the placement scheme is perfect, but it is required to get mgag200 working with atomic modesetting's pageflip logic. So we're solving a real problem here.
I don't doubt this is a real problem.
Quoted 7 lines
The bug comes from Weston's allocation strategy. Looking at the debug output:
Quoted 1 line
Quoted 1 line
0x0000000000000000-0x000000000000057f: 1407: free
This was fbdev's framebuffer with 1600x900@32bpp
Quoted 7 lines
Quoted 1 line
Quoted 1 line
0x000000000000057f-0x0000000000000b5b: 1500: used
This is Weston's framebuffer also with 1600x900@32bpp. But Weston allocates an additional, unused 60 scanlines. That is to render with tiles of 64x64px, I suppose. fbdev doesn't do that, hence Weston's second framebuffer doesn't fit into the free location of the fbdev framebuffer.
Quoted 7 lines
Sure. Assume there is just enough vram to fit in fbdev and two weston framebuffers. fbdev is allocated from bottom, first weston fb from top, second weston fb from bottom again. fbdev is not pushed out (no memory pressure yet) so the second weston fb ends up in the middle of vram fragmenting things. And now you are again in the situation where you might have enough free vram for an allocation but can't use it due to fragmention (probably harder to trigger in practice though).
Not quite. Framebuffer BOs of the current or smaller size will fit into vram. It's only a problem if the size of the new framebuffer BO increases. And that is exactly what currently happens with mgag200.
That aside, it's not a fair point, as you constructed an example that no memory manager can resolve.
Quoted 5 lines
That's why I would suggest to explicitly move out unpinned framebuffers (aka fbdev) before pinning a new one (second weston fb) instead of depending on ttm moving things out on OOM, to make sure you never allocate something in the middle of vram.
We cannot do that. Evicting BOs from vram involves an unmap operation. We did that in an earlier version of the code and received reports about performance regressions and CPU cycles in TLB update.
So we added the lazy-unmap feature, where BOs are only unmapped and evicted when necessary. I think it was even you who suggested this idea. :)
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Third: I'd suggest make topdown allocations depending on current state instead of simply alternating, i.e. if there is a pinned framebuffer @ offset 0, then go for top-down.
Quoted 7 lines
Quoted 4 lines
That's what the current patch does. If the last pin was at the bottom, the next goes to the top. And then the other way around. Without alternating between both end of vram, the problem would occur again when fragmentation happens near the top end.
I'd feel better when checking the state of my current pins to figure whenever I should alloc top-bottom or not, for robustness reasons.
I don't understand why this is more robust. For example, if you pin a larger number of BOs, and than evict every other BO, there will always be free areas among the remaining BOs. All that changes is the location of those areas.
For a strategy, one cannot look at the BO size alone. In my initial example, BOs are ~6 MiB and vram is 16 MiB. So any strategy ala 'choose top-bottom for BOs >1/3 of vram' selects top-bottom for all framebuffer BOs. This would result in the same OOM, but from top to bottom.
At some point one has to choose to switch to top-down, and then back again at one of the next BOs. So the current patch effectively splits vram into a lower half and an upper half and puts BOs in alternating halves.
Best regards
Thomas
Quoted 15 lines
Quoted 4 lines
Looking again at the vram helpers, this functionality could be implemented in
drm_gem_vram_plane_helper_prepare_fb(). Drivers with other placement strategies could implement their own helper forprepare_fb.vram helpers could also simply offer two
prepare_fbvariants.cheers,
Gerd_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi,
Quoted 3 lines
At some point one has to choose to switch to top-down, and then back again at one of the next BOs. So the current patch effectively splits vram into a lower half and an upper half and puts BOs in alternating halves.
Hmm, so maybe just make the split explicit instead of playing tricks with top_bottom? Register the lower vram half as TT_VRAM, register the upper half as TT_PRIV. That'll make sure you never have a big allocation in the middle blocking things. Or maybe add a allocation strategy flag to ttm which effectively does the same.
take care,
Gerd
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Thomas Zimmermann
maintainerRe: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi Gerd
Am 24.04.20 um 11:38 schrieb Gerd Hoffmann:
Quoted 11 lines
Hi,
Quoted 3 lines
At some point one has to choose to switch to top-down, and then back again at one of the next BOs. So the current patch effectively splits vram into a lower half and an upper half and puts BOs in alternating halves.
Hmm, so maybe just make the split explicit instead of playing tricks with
top_bottom? Register the lower vram half asTT_VRAM, register the upper half asTT_PRIV. That'll make sure you never have a big allocation in the middle blocking things. Or maybe add a allocation strategy flag to ttm which effectively does the same.
It's not that easy. Current cursors n ast are statically allocated. As soon as you add dynamic cursors into the mix, you'd get OOMs. If the framebuffer BO goes into VRAM and the cursor BO goes into PRIV, pinning the next framebuffer BO during a pageflip would send it to VRAM. But VRAM is still filled with the first framebuffer BO. So the memory manager needs some flexibility with the placement. Handling all that is already TTM's job.
I'm preparing v2 of this patch set. It'll get static cursors out of the way and make the feature opt-in.
Best regards
Thomas
Quoted 4 lines
take care,
Gerd
Signature · 6 lines
Thomas Zimmermann Graphics Driver Developer SUSE Software Solutions Germany GmbH Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Hi,
Quoted 2 lines
It's not that easy. Current cursors n ast are statically allocated. As soon as you add dynamic cursors into the mix, you'd get OOMs.
Well, with the split you can simply allocate dynamic cursors with top-bottom to keep them out of the way. It's not 100% perfect, the area where the cursors are allocated from has a bit less free vram, so the split effectively isn't 50/50 but more like 49/51. But hey, still alot better than what we have today.
With two static cursors you can allocate one from TT_VRAM and one from TT_PRIV so the remaining free vram is the same in both regions.
Quoted 3 lines
If the framebuffer BO goes into VRAM and the cursor BO goes into PRIV, pinning the next framebuffer BO during a pageflip would send it to VRAM.
Oh, seems my idea outline was a bit to short. The plan is *not* to alternate between VRAM and PRIV on allocations. The plan is to add both PRIV and VRAM to the placement array when pinning the framebuffer.
ttm can simply place the framebuffers as it pleases and where it wants. Due to the split it can't do a big blocking allocation in the middle of vram though.
If you are going to pageflip you should have one framebuffer already pinned (the one currently displayed). If that happens to live TT_VRAM ttm should be able to make room in TT_PRIV to pin the new framebuffer you want pageflip to, and visa-versa.
ttm will only evict unpinned BOs if needed, when running with smaller framebuffers ttm will happily keep more than two framebuffers in vram. All fully automatic without vram helpers having to pay much attention to the allocation strategy.
Quoted 2 lines
I'm preparing v2 of this patch set. It'll get static cursors out of the way and make the feature opt-in.
I think with the split model there is no need to make that optional.
cheers,
Gerd
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel
Re: [PATCH 2/2] drm/vram-helper: Alternate between bottom-up and top-down placement
Am 29.04.20 um 09:25 schrieb Gerd Hoffmann:
Quoted 16 lines
Hi,Quoted 2 lines
It's not that easy. Current cursors n ast are statically allocated. As soon as you add dynamic cursors into the mix, you'd get OOMs.
Well, with the split you can simply allocate dynamic cursors with top-bottom to keep them out of the way. It's not 100% perfect, the area where the cursors are allocated from has a bit less free vram, so the split effectively isn't 50/50 but more like 49/51. But hey, still alot better than what we have today.
With two static cursors you can allocate one from
TT_VRAMand one fromTT_PRIVso the remaining free vram is the same in both regions.Quoted 3 lines
If the framebuffer BO goes into VRAM and the cursor BO goes into PRIV, pinning the next framebuffer BO during a pageflip would send it to VRAM.
Quoted 7 lines
Oh, seems my idea outline was a bit to short. The plan is *not* to alternate between VRAM and PRIV on allocations. The plan is to add both PRIV and VRAM to the placement array when pinning the framebuffer.
ttm can simply place the framebuffers as it pleases and where it wants. Due to the split it can't do a big blocking allocation in the middle of vram though.
There is an easier and most likely less CPU intense way of doing this.
All you need to do is to make allocations which are below halve of your address space allocate from the beginning of a hole and if they are above halve of the address space from the end of a hole.
This way you should always keep the largest hole in the middle.
This can be done with another flag to drm_mm and would most likely would be a nice addition to other drivers as well.
Regards,
Christian.
Quoted 11 lines
If you are going to pageflip you should have one framebuffer already pinned (the one currently displayed). If that happens to live
TT_VRAMttm should be able to make room inTT_PRIVto pin the new framebuffer you want pageflip to, and visa-versa.ttm will only evict unpinned BOs if needed, when running with smaller framebuffers ttm will happily keep more than two framebuffers in vram. All fully automatic without vram helpers having to pay much attention to the allocation strategy.
Quoted 7 lines
Quoted 2 lines
I'm preparing v2 of this patch set. It'll get static cursors out of the way and make the feature opt-in.
I think with the split model there is no need to make that optional.
cheers,
Gerd
_______________________________________________
dri-devel mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/dri-devel