reducing direct map fragmentation
Hi all,
There are use-cases that need to remove pages from the direct map or at least map them at PTE level. These use-cases include vfree, module loading, ftrace, kprobe, BPF, secretmem and generally any caller of set_memory/set_direct_map APIs.
Remapping pages at PTE level causes split of the PUD and PMD sized mappings in the direct map which leads to performance degradation.
To reduce the performance hit caused by the fragmentation of the direct map, it makes sense to group and/or cache the base pages removed from the direct map so that the most of base pages created during a split of a large page will be consumed by users requiring PTE level mappings.
Last year the proposal to use a new migrate type for such cache received strong pushback and the suggested alternative was to try to use slab instead.
I've been thinking about it (yeah, it took me a while) and I believe slab is not appropriate because use cases require at least page size allocations and some would really benefit from higher order allocations, and in the most cases the code that allocates memory excluded from the direct map needs the struct page/folio.
For example, caching allocations of text in 2M pages would benefit from reduced iTLB pressure and doing kmalloc() from vmalloc() will be way more intrusive than using some variant of __alloc_pages().
Secretmem and potentially PKS protected page tables also need struct page/folio.
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the free() path, the pages that are mapped at PTE level will be put into that cache.
The cache is internally implemented as a buddy allocator so it can satisfy high order allocations, and there will be a shrinker to release free pages from that cache to the page allocator.
I hope to have a first prototype posted Really Soon.
Signature · 2 lines
Sincerely yours, Mike.
Quoted 1 line · Mike Rapoport
Hi all,
Hi Mike, I'm interested in this topic and hope to discuss this with you at LSF/MM/BPF.
Quoted 5 lines
There are use-cases that need to remove pages from the direct map or at least map them at PTE level. These use-cases include vfree, module loading, ftrace, kprobe, BPF, secretmem and generally any caller of set_memory/set_direct_map APIs.
Quoted 7 lines
Remapping pages at PTE level causes split of the PUD and PMD sized mappings in the direct map which leads to performance degradation.
To reduce the performance hit caused by the fragmentation of the direct map, it makes sense to group and/or cache the base pages removed from the direct map so that the most of base pages created during a split of a large page will be consumed by users requiring PTE level mappings.
How much performance difference did you see in your test when direct map was fragmented, or is there a way to check this difference?
Quoted 14 lines
Last year the proposal to use a new migrate type for such cache received strong pushback and the suggested alternative was to try to use slab instead.
I've been thinking about it (yeah, it took me a while) and I believe slab is not appropriate because use cases require at least page size allocations and some would really benefit from higher order allocations, and in the most cases the code that allocates memory excluded from the direct map needs the struct page/folio.
For example, caching allocations of text in 2M pages would benefit from reduced iTLB pressure and doing
kmalloc()fromvmalloc()will be way more intrusive than using some variant of__alloc_pages().
Quoted 7 lines
Secretmem and potentially PKS protected page tables also need struct page/folio.
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.
I would like to discuss not only having cache layer of pages but also how direct map could be merged correctly and efficiently.
I vaguely recall that Aaron Lu sent RFC series about this and Kirill A. Shutemov's feedback was to batch merge operations. [1]
Also a CPA API called by the cache layer that could merge fragmented mappings would work for merging 4K pages to 2M [2], but won't work for merging 2M mappings to 1G mappings.
At that time I didn't follow more discussions (e.g. execmem_alloc()) Maybe I'm missing some points.
[1] https://lore.kernel.org/linux-mm/20220809100408.rm6ofiewtty6rvcl@box
[2] https://lore.kernel.org/linux-mm/[email protected]
Quoted 5 lines
The cache is internally implemented as a buddy allocator so it can satisfy high order allocations, and there will be a shrinker to release free pages from that cache to the page allocator.
I hope to have a first prototype posted Really Soon.
Looking forward to that!
Wonder how it would be shaped.
Quoted 4 lines
Signature · 3 lines
-- Sincerely yours, Mike.
Quoted 6 lines · Hyeonggon Yoo
Quoted 1 line · Mike Rapoport
Hi all,
Hi Mike, I'm interested in this topic and hope to discuss this with you at LSF/MM/BPF.
Quoted 7 lines
Quoted 4 lines
To reduce the performance hit caused by the fragmentation of the direct map, it makes sense to group and/or cache the base pages removed from the direct map so that the most of base pages created during a split of a large page will be consumed by users requiring PTE level mappings.
How much performance difference did you see in your test when direct map was fragmented, or is there a way to check this difference?
I did some benchmarks a while ago with the entire direct map forced to 2M or 4k pages. The results I had are here:
https://docs.google.com/spreadsheets/d/1tdD-cu8e93vnfGsTFxZ5YdaEfs2E1GELlvWNOGkJV2U/edit?usp=sharing
Intel folks did more comprehensive testing and their results are here:
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 8 lines
Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.I would like to discuss not only having cache layer of pages but also how direct map could be merged correctly and efficiently.
Quoted 6 lines
I vaguely recall that Aaron Lu sent RFC series about this and Kirill A. Shutemov's feedback was to batch merge operations. [1]
Also a CPA API called by the cache layer that could merge fragmented mappings would work for merging 4K pages to 2M [2], but won't work for merging 2M mappings to 1G mappings.
One possible way is to make CPA scan all PMDs in 1G page after merging a 2M page. Not sure how efficient would it be though.
Quoted 6 lines
At that time I didn't follow more discussions (e.g.
execmem_alloc()) Maybe I'm missing some points.[1] https://lore.kernel.org/linux-mm/20220809100408.rm6ofiewtty6rvcl@box
Signature · 2 lines
Sincerely yours, Mike.
Quoted 15 lines · Mike Rapoport
Quoted 6 lines · Hyeonggon Yoo
Quoted 1 line · Mike Rapoport
Hi all,
Hi Mike, I'm interested in this topic and hope to discuss this with you at LSF/MM/BPF.
Quoted 7 lines
Quoted 4 lines
To reduce the performance hit caused by the fragmentation of the direct map, it makes sense to group and/or cache the base pages removed from the direct map so that the most of base pages created during a split of a large page will be consumed by users requiring PTE level mappings.
How much performance difference did you see in your test when direct map was fragmented, or is there a way to check this difference?
Quoted 8 lines
I did some benchmarks a while ago with the entire direct map forced to 2M or 4k pages. The results I had are here:
https://docs.google.com/spreadsheets/d/1tdD-cu8e93vnfGsTFxZ5YdaEfs2E1GELlvWNOGkJV2U/edit?usp=sharing
Intel folks did more comprehensive testing and their results are here:
Thanks!
Hmm it might not be best choice to unconditionally
merge 2M mappings to 1G a mapping. (maybe should be controlled via a boot parameter or something)
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.I would like to discuss not only having cache layer of pages but also how direct map could be merged correctly and efficiently.
I vaguely recall that Aaron Lu sent RFC series about this and Kirill A. Shutemov's feedback was to batch merge operations. [1]
Quoted 6 lines
Quoted 3 lines
Also a CPA API called by the cache layer that could merge fragmented mappings would work for merging 4K pages to 2M [2], but won't work for merging 2M mappings to 1G mappings.
One possible way is to make CPA scan all PMDs in 1G page after merging a 2M page. Not sure how efficient would it be though.
That seems to be similar to what Kirill A. Shutemov has been tried. He may have opinions about that?
[3] https://lore.kernel.org/lkml/[email protected]
Quoted 10 lines
Quoted 6 lines
At that time I didn't follow more discussions (e.g.
execmem_alloc()) Maybe I'm missing some points.[1] https://lore.kernel.org/linux-mm/20220809100408.rm6ofiewtty6rvcl@box
Signature · 2 lines
Sincerely yours, Mike.
Quoted 18 lines · Hyeonggon Yoo
Quoted 17 lines · Mike Rapoport
Quoted 14 lines · Hyeonggon Yoo
Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.I would like to discuss not only having cache layer of pages but also how direct map could be merged correctly and efficiently.
I vaguely recall that Aaron Lu sent RFC series about this and Kirill A. Shutemov's feedback was to batch merge operations. [1]
Also a CPA API called by the cache layer that could merge fragmented mappings would work for merging 4K pages to 2M [2], but won't work for merging 2M mappings to 1G mappings.
Quoted 7 lines
Quoted 2 lines
One possible way is to make CPA scan all PMDs in 1G page after merging a 2M page. Not sure how efficient would it be though.
That seems to be similar to what Kirill A. Shutemov has been tried. He may have opinions about that?
Kirill's patch attempted to restore 1G page for each cpa_flush(), so it scanned a lot of page tables without a guarantee that collapsing small mappings to a large page is possible.
If we call a function that will collapse a 2M when we know for sure that the collapse is possible, it will be more efficient.
Signature · 2 lines
Sincerely yours, Mike.
Hi,
On Wed 01-02-23 20:06:37, Mike Rapoport wrote:
[...]
Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.
Are there stil open questions which would benefit from a discussion at LSFMM this year?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 11 lines · Michal Hocko
Hi,
On Wed 01-02-23 20:06:37, Mike Rapoport wrote:
[...]Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.Are there stil open questions which would benefit from a discussion at LSFMM this year?
Yes, I believe.
I was trying to get some numbers to see what would be the benefit of __GFP_UNMAPPED and I couldn't find a benchmark that will produce results with good signal-to-noise ratio.
So while it seems that there's a general agreement on how to implement caching of 2M pages, there is still no evidence that it will be universally useful.
It would be interesting to discuss the reasons for inconclusive results, and more importantly, what should be the general direction for dealing with the direct map fragmentation.
As it seems now, packing code allocations into 2M pages would be an improvement, while data allocations that fragment the direct map do not impact much the overall system performance.
I'll bring the mmtest results I have to begin the discussion.
Quoted 3 lines
Signature · 2 lines
Michal Hocko SUSE Labs
Signature · 2 lines
Sincerely yours, Mike.
Quoted 27 lines · Mike Rapoport
Quoted 11 lines · Michal Hocko
Hi,
On Wed 01-02-23 20:06:37, Mike Rapoport wrote:
[...]Quoted 4 lines
My current proposal is to have a cache of 2M pages close to the page allocator and use a GFP flag to make allocation request use that cache. On the
free()path, the pages that are mapped at PTE level will be put into that cache.Are there stil open questions which would benefit from a discussion at LSFMM this year?
Yes, I believe.
I was trying to get some numbers to see what would be the benefit of
__GFP_UNMAPPEDand I couldn't find a benchmark that will produce results with good signal-to-noise ratio.So while it seems that there's a general agreement on how to implement caching of 2M pages, there is still no evidence that it will be universally useful.
It would be interesting to discuss the reasons for inconclusive results, and more importantly, what should be the general direction for dealing with the direct map fragmentation.
Quoted 5 lines
As it seems now, packing code allocations into 2M pages would be an improvement, while data allocations that fragment the direct map do not impact much the overall system performance.
I'll bring the mmtest results I have to begin the discussion.
Makes sense. Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs