Generating physically contiguous memory after page allocation
Subsystems (30)
cover Generating physically contiguous memory after page allocation
From: Zi Yan <[email protected]>
Hi all,
This patchset produces physically contiguous memory by moving in-use pages without allocating any new pages. It targets two scenarios that complements khugepaged use cases: 1) avoiding page reclaim and memory compaction when the system is under memory pressure because this patchset does not allocate any new pages, 2) generating pages larger than 2^MAX_ORDER without changing the buddy allocator.
To demonstrate its use, I add very basic 1GB THP support and enable promoting 512 2MB THPs to a 1GB THP in my patchset. Promoting 512 4KB pages to a 2MB THP is also implemented.
The patches are on top of v5.0-rc5. They are posted as part of my upcoming LSF/MM proposal.
Motivation
----
The goal of this patchset is to provide alternative way of generating physically contiguous memory and making it available as arbitrary sized large pages. This patchset generates physically contiguous memory/arbitrary size pages after pages are allocated by moving virtually-contiguous pages to become physically contiguous at any size, thus it does not require changes to memory allocators. On the other hand, it works only for moveable pages, so it also faces the same fragmentation issues as memory compaction, i.e., if non-moveable pages spread across the entire memory, this patchset can only generate contiguity between any two non-moveable pages.
Large pages and physically contiguous memory are important to devices, such as GPUs, FPGAs, NICs and RDMA controllers, because they can often achieve better performance when operating on large pages. The same can be said of CPU performance, of course, but there is an important difference: GPUs and high-throughput devices often take a more severe performance hit, in the event of a TLB miss and subsequent page table walks, as compared to a CPU. The effect is sufficiently large that such devices *really* want a highly reliable way to allocate large pages to minimize the number of potential TLB misses and the time spent on the induced page table walks.
Vendors (like Oracle, Mellanox, IBM, NVIDIA) are interested in generating physically contiguous memory beyond THP sizes and looking for solutions [1],[2],[3]. This patchset provides an alternative approach, compared to allocating physically contiguous memory at page allocation time, to generating physically contiguous memory after pages are allocated. This approach can avoid page reclaim and memory compaction, which happen during the process of page allocation, but still produces comparable physically contiguous memory.
In terms of THPs, it helps, but we are interested in even larger contiguous ranges (or page size support) to further reduce the address translation overheads. With this patchset, we can generate pages larger than PMD-level THPs without requiring MAX_ORDER changes in the buddy allocators.
Patch structure
----
The patchset I developed to generate physically contiguous memory/arbitrary sized pages merely moves pages around. There are three components in this patchset:
1) a new page migration mechanism, called exchange pages, that exchanges the content of two in-use pages instead of performing two back-to-back page migration. It saves on overheads and avoids page reclaim and memory compaction in the page allocation path, although it is not strictly required if enough free memory is available in the system.
2) a new mechanism that utilizes both page migration and exchange pages to produce physically contiguous memory/arbitrary sized pages without allocating any new pages, unlike what khugepaged does. It works on per-VMA basis, creating physically contiguous memory out of each VMA, which is virtually contiguous. A simple range tree is used to ensure no two VMAs are overlapping with each other in the physical address space.
3) a use case of the new physically contiguous memory producing mechanism that generates 1GB THPs by migrating and exchanging pages and promoting 512 contiguous 2MB THPs to a 1GB THP, although even larger physically contiguous memory ranges can be generated. The 1GB THP implement is very basic, which can handle 1GB THP faults when buddy allocator is modified to allocate 1GB pages, support 1GB THP split to 2MB THP and in-place promotion from 2MB THP to 1GB THP, and PMD/PTE-mapped 1GB THP. These are not fully tested.
[1] https://lwn.net/Articles/736170/
[2] https://lwn.net/Articles/753167/
[3] https://blogs.nvidia.com/blog/2018/06/08/worlds-fastest-exascale-ai-supercomputer-summit/
Zi Yan (31):
mm: migrate: Add exchange_pages to exchange two lists of pages.
mm: migrate: Add THP exchange support.
mm: migrate: Add tmpfs exchange support.
mm: add mem_defrag functionality.
mem_defrag: split a THP if either src or dst is THP only.
mm: Make MAX_ORDER configurable in Kconfig for buddy allocator.
mm: deallocate pages with order > MAX_ORDER.
mm: add pagechain container for storing multiple pages.
mm: thp: 1GB anonymous page implementation.
mm: proc: add 1GB THP kpageflag.
mm: debug: print compound page order in dump_page().
mm: stats: Separate PMD THP and PUD THP stats.
mm: thp: 1GB THP copy on write implementation.
mm: thp: handling 1GB THP reference bit.
mm: thp: add 1GB THP split_huge_pud_page() function.
mm: thp: check compound_mapcount of PMD-mapped PUD THPs at free time.
mm: thp: split properly PMD-mapped PUD THP to PTE-mapped PUD THP.
mm: page_vma_walk: teach it about PMD-mapped PUD THP.
mm: thp: 1GB THP support in try_to_unmap().
mm: thp: split 1GB THPs at page reclaim.
mm: thp: 1GB zero page shrinker.
mm: thp: 1GB THP follow_p*d_page() support.
mm: support 1GB THP pagemap support.
sysctl: add an option to only print the head page virtual address.
mm: thp: add a knob to enable/disable 1GB THPs.
mm: thp: promote PTE-mapped THP to PMD-mapped THP.
mm: thp: promote PMD-mapped PUD pages to PUD-mapped PUD pages.
mm: vmstats: add page promotion stats.
mm: madvise: add madvise options to split PMD and PUD THPs.
mm: mem_defrag: thp: PMD THP and PUD THP in-place promotion support.
sysctl: toggle to promote PUD-mapped 1GB THP or not.
arch/x86/Kconfig | 15 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + arch/x86/include/asm/pgalloc.h | 69 + arch/x86/include/asm/pgtable.h | 20 + arch/x86/include/asm/sparsemem.h | 4 +- arch/x86/mm/pgtable.c | 38 + drivers/base/node.c | 3 + fs/exec.c | 4 + fs/proc/meminfo.c | 2 + fs/proc/page.c | 2 + fs/proc/task_mmu.c | 47 +- include/asm-generic/pgtable.h | 110 + include/linux/huge_mm.h | 78 +- include/linux/khugepaged.h | 1 + include/linux/ksm.h | 5 + include/linux/mem_defrag.h | 60 + include/linux/memcontrol.h | 5 + include/linux/mm.h | 34 + include/linux/mm_types.h | 5 + include/linux/mmu_notifier.h | 13 + include/linux/mmzone.h | 1 + include/linux/page-flags.h | 79 +- include/linux/pagechain.h | 73 + include/linux/rmap.h | 10 +- include/linux/sched/coredump.h | 4 + include/linux/swap.h | 2 + include/linux/syscalls.h | 3 + include/linux/vm_event_item.h | 33 + include/uapi/asm-generic/mman-common.h | 15 + include/uapi/linux/kernel-page-flags.h | 2 + kernel/events/uprobes.c | 4 +- kernel/fork.c | 14 + kernel/sysctl.c | 101 +- mm/Makefile | 2 + mm/compaction.c | 17 +- mm/debug.c | 8 +- mm/exchange.c | 878 +++++++ mm/filemap.c | 8 + mm/gup.c | 60 +- mm/huge_memory.c | 3360 ++++++++++++++++++++---- mm/hugetlb.c | 4 +- mm/internal.h | 46 + mm/khugepaged.c | 7 +- mm/ksm.c | 39 +- mm/madvise.c | 121 + mm/mem_defrag.c | 1941 ++++++++++++++ mm/memcontrol.c | 13 + mm/memory.c | 55 +- mm/migrate.c | 14 +- mm/mmap.c | 29 + mm/page_alloc.c | 108 +- mm/page_vma_mapped.c | 129 +- mm/pgtable-generic.c | 78 +- mm/rmap.c | 283 +- mm/swap.c | 38 + mm/swap_slots.c | 2 + mm/swapfile.c | 4 +- mm/userfaultfd.c | 2 +- mm/util.c | 7 + mm/vmscan.c | 55 +- mm/vmstat.c | 32 + 61 files changed, 7452 insertions(+), 745 deletions(-) create mode 100644 include/linux/mem_defrag.h create mode 100644 include/linux/pagechain.h create mode 100644 mm/exchange.c create mode 100644 mm/mem_defrag.c
--
2.20.1
RFC PATCH 1/31 mm: migrate: Add exchange_pages to exchange two lists of pages.
From: Zi Yan <[email protected]>
In stead of using two migrate_pages(), a single exchange_pages() would be sufficient and without allocating new pages.
--- include/linux/ksm.h | 5 + mm/Makefile | 1 + mm/exchange.c | 846 ++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 6 + mm/ksm.c | 35 ++ mm/migrate.c | 4 +- 6 files changed, 895 insertions(+), 2 deletions(-) create mode 100644 mm/exchange.c
include/linux/ksm.h +5 -0
@@ -53,6 +53,7 @@ struct page *ksm_might_need_to_copy(struct page *page,void rmap_walk_ksm (struct page *page,struct rmap_walk_control *rwc);void ksm_migrate_page (struct page *newpage,struct page *oldpage); +void ksm_exchange_page (struct page *to_page,struct page *from_page);#else /* !CONFIG_KSM */
@@ -86,6 +87,10 @@ static inline void rmap_walk_ksm(struct page *page,static inline void ksm_migrate_page (struct page *newpage,struct page *oldpage) { } +static inline void ksm_exchange_page (struct page *to_page, +struct page *from_page) +{ +}#endif /* CONFIG_MMU */ #endif /* !CONFIG_KSM */
mm/Makefile +1 -0
@@ -43,6 +43,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ obj-y += init-mm.o obj-y += memblock.o +obj-y += exchange.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
mm/exchange.c new file +846 -0
846 lines not on this page · raw message
mm/internal.h +6 -0
@@ -543,4 +543,10 @@ static inline bool is_migrate_highatomic_page(struct page *page)void setup_zone_pageset (struct zone *zone);extern struct page *alloc_new_node_page (struct page *page,unsigned long node); + +bool buffer_migrate_lock_buffers (struct buffer_head *head, +enum migrate_mode mode); +int writeout (struct address_space *mapping,struct page *page); +extern int exchange_two_pages (struct page *page1,struct page *page2); +#endif /* __MM_INTERNAL_H */
mm/ksm.c +35 -0
@@ -2665,6 +2665,41 @@ void ksm_migrate_page(struct page *newpage, struct page *oldpage) set_page_stable_node(oldpage, NULL); } } + +void ksm_exchange_page (struct page *to_page,struct page *from_page) +{ +struct stable_node *to_stable_node, *from_stable_node; + +VM_BUG_ON_PAGE (!PageLocked (to_page), to_page); +VM_BUG_ON_PAGE (!PageLocked (from_page), from_page); + + to_stable_node =page_stable_node (to_page); + from_stable_node =page_stable_node (from_page); +if (to_stable_node) { +VM_BUG_ON_PAGE (to_stable_node->kpfn !=page_to_pfn (from_page), + from_page); + to_stable_node->kpfn =page_to_pfn (to_page); +/* +* newpage->mapping was set in advance; now we need smp_wmb() +* to make sure that the new stable_node->kpfn is visible +* to get_ksm_page() before it can see that oldpage->mapping +* has gone stale (or that PageSwapCache has been cleared). +*/ +smp_wmb (); + } +if (from_stable_node) { +VM_BUG_ON_PAGE (from_stable_node->kpfn !=page_to_pfn (to_page), + to_page); + from_stable_node->kpfn =page_to_pfn (from_page); +/* +* newpage->mapping was set in advance; now we need smp_wmb() +* to make sure that the new stable_node->kpfn is visible +* to get_ksm_page() before it can see that oldpage->mapping +* has gone stale (or that PageSwapCache has been cleared). +*/ +smp_wmb (); + } +}#endif /* CONFIG_MIGRATION */ #ifdef CONFIG_MEMORY_HOTREMOVE
mm/migrate.c +2 -2
@@ -701,7 +701,7 @@ EXPORT_SYMBOL(migrate_page);#ifdef CONFIG_BLOCK /* Returns true if all buffers are successfully locked */ -static bool buffer_migrate_lock_buffers (struct buffer_head *head, +bool buffer_migrate_lock_buffers (struct buffer_head *head,enum migrate_mode mode) {struct buffer_head *bh = head;
@@ -849,7 +849,7 @@ int buffer_migrate_page_norefs(struct address_space *mapping,/* * Writeback a page to clean the dirty state */ -static int writeout (struct address_space *mapping,struct page *page) +int writeout (struct address_space *mapping,struct page *page) {struct writeback_control wbc = { .sync_mode =WB_SYNC_NONE ,
Signature · 1 line
2.20.1
Matthew Wilcox
maintainerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 6 lines · Zi Yan
mm/exchange.c new file +6
-0+struct page_flags { +unsigned int page_error :1 ; +unsigned int page_referenced:1 ; +unsigned int page_uptodate:1 ; +unsigned int page_active:1 ; +unsigned int page_unevictable:1 ;
mm/exchange.c line 52
Quoted 10 lines
mm/exchange.c new file +10
-0+ unsigned int page_checked:1; + unsigned int page_mappedtodisk:1; + unsigned int page_dirty:1; + unsigned int page_is_young:1; + unsigned int page_is_idle:1; + unsigned int page_swapcache:1; + unsigned int page_writeback:1; + unsigned int page_private:1; + unsigned int __pad:3; +};
I'm not sure how to feel about this. It's a bit fragile versus somebody adding new page flags. I don't know whether it's needed or whether you can just copy page->flags directly because you're holding PageLock.
mm/exchange.c line 65
Quoted 11 lines
mm/exchange.c new file +11
-0+static void exchange_page (char *to,char *from) +{ +u64 tmp; +int i; + +for (i =0 ; i <PAGE_SIZE ; i +=sizeof (tmp)) { + tmp = *((u64 *)(from + i)); + *((u64 *)(from + i)) = *((u64 *)(to + i)); + *((u64 *)(to + i)) = tmp; + } +}
I have a suspicion you'd be better off allocating a temporary page and using copy_page(). Some architectures have put a lot of effort into making copy_page() run faster.
mm/exchange.c line 326
Quoted 4 lines
mm/exchange.c new file +4
-0+xa_lock_irq (&to_mapping->i_pages); + + to_pslot =radix_tree_lookup_slot (&to_mapping->i_pages, +page_index (to_page));
This needs to be converted to the XArray. radix_tree_lookup_slot() is going away soon. You probably need:
XA_STATE(to_xas, &to_mapping->i_pages, page_index(to_page));
This is a lot of code and I'm still trying to get my head aroud it all. Thanks for putting in this work; it's good to see this approach being explored.
Zi Yan
reviewerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 13 lines · Matthew Wilcox
Quoted 12 lines · Zi Yan
mm/exchange.c new file +12
-0+struct page_flags { +unsigned int page_error :1 ; +unsigned int page_referenced:1 ; +unsigned int page_uptodate:1 ; +unsigned int page_active:1 ; +unsigned int page_unevictable:1 ; +unsigned int page_checked:1 ; +unsigned int page_mappedtodisk:1 ; +unsigned int page_dirty:1 ; +unsigned int page_is_young:1 ; +unsigned int page_is_idle:1 ; +unsigned int page_swapcache:1 ;
Quoted 8 lines
mm/exchange.c line 52
Quoted 4 lines
mm/exchange.c new file +4
-0+ unsigned int page_writeback:1; + unsigned int page_private:1; + unsigned int __pad:3; +};I'm not sure how to feel about this. It's a bit fragile versus somebody adding new page flags. I don't know whether it's needed or whether you can just copy
page->flagsdirectly because you're holding PageLock.
I agree with you that current way of copying page flags individually could miss new page flags. I will try to come up with something better. Copying page->flags as a whole might not simply work, since the upper part of page->flags has the page node information, which should not be changed. I think I need to add a helper function to just copy/exchange all page flags, like calling migrate_page_stats() twice.
Quoted 7 lines
Quoted 7 lines
mm/exchange.c new file +7
-0+static void exchange_page (char *to,char *from) +{ +u64 tmp; +int i; + +for (i =0 ; i <PAGE_SIZE ; i +=sizeof (tmp)) { + tmp = *((u64 *)(from + i));
Quoted 8 lines
mm/exchange.c line 65
Quoted 4 lines
mm/exchange.c new file +4
-0+ *((u64 *)(from + i)) = *((u64 *)(to + i)); + *((u64 *)(to + i)) = tmp; + } +}I have a suspicion you'd be better off allocating a temporary page and using
copy_page(). Some architectures have put a lot of effort into makingcopy_page()run faster.
When I am doing exchange_pages() between two NUMA nodes on a x86_64 machine, I actually can saturate the QPI bandwidth with this operation. I think cache prefetching was doing its job.
The purpose of proposing exchange_pages() is to avoid allocating any new page, so that we would not trigger any potential page reclaim or memory compaction. Allocating a temporary page defeats the purpose.
Quoted 10 lines
mm/exchange.c line 326
Quoted 5 lines
mm/exchange.c new file +4
-0+xa_lock_irq (&to_mapping->i_pages); + + to_pslot =radix_tree_lookup_slot (&to_mapping->i_pages, +page_index (to_page));This needs to be converted to the XArray.
radix_tree_lookup_slot()is going away soon. You probably need:XA_STATE(to_xas, &to_mapping->i_pages, page_index(to_page));
Thank you for pointing this out. I will do the change.
Quoted 4 lines
This is a lot of code and I'm still trying to get my head aroud it all. Thanks for putting in this work; it's good to see this approach being explored.
Thank you for taking a look at the code.
--
Best Regards,
Yan Zi
Re: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 5 lines · Zi Yan
The purpose of proposing
exchange_pages()is to avoid allocating any new page,
so that we would not trigger any potential page reclaim or memory compaction.
Allocating a temporary page defeats the purpose.
Compaction can only happen for order > 0 temporary pages. Even if you used single order = 0 page to gradually exchange e.g. a THP, it should be better than u64. Allocating order = 0 should be a non-issue. If it's an issue, then the system is in a bad state and physically contiguous layout is a secondary concern.
Zi Yan
reviewerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 11 lines · Vlastimil Babka
Quoted 5 lines · Zi Yan
The purpose of proposing
exchange_pages()is to avoid allocating any new page,
so that we would not trigger any potential page reclaim or memory compaction.
Allocating a temporary page defeats the purpose.Compaction can only happen for order > 0 temporary pages. Even if you used single order = 0 page to gradually exchange e.g. a THP, it should be better than u64. Allocating order = 0 should be a non-issue. If it's an issue, then the system is in a bad state and physically contiguous layout is a secondary concern.
You are right if we only need to allocate one order-0 page. But this also means we can only exchange two pages at a time. We need to add a lock to make sure the temporary page is used exclusively or we need to keep allocating temporary pages when multiple exchange_pages() are happening at the same time.
--
Best Regards,
Yan Zi
Matthew Wilcox
maintainerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 18 lines · Zi Yan
Quoted 8 lines · Vlastimil Babka
Quoted 6 lines · Zi Yan
The purpose of proposing
exchange_pages()is to avoid allocating any new
page,
so that we would not trigger any potential page reclaim or memory compaction.
Allocating a temporary page defeats the purpose.Quoted 8 lines
Compaction can only happen for order > 0 temporary pages. Even if you used
single order = 0 page to gradually exchange e.g. a THP, it should be better than
u64. Allocating order = 0 should be a non-issue. If it's an issue, then the
system is in a bad state and physically contiguous layout is a secondary concern.
Quoted 6 lines
You are right if we only need to allocate one order-0 page. But this also means
we can only exchange two pages at a time. We need to add a lock to make sure the temporary page is used exclusively or we need to keep allocating temporary pages
when multipleexchange_pages()are happening at the same time.
You allocate one temporary page per thread that's doing an exchange_page().
Zi Yan
reviewerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 19 lines · Matthew Wilcox
Quoted 18 lines · Zi Yan
Quoted 16 lines · Vlastimil Babka
Quoted 6 lines · Zi Yan
The purpose of proposing
exchange_pages()is to avoid allocating any new
page,
so that we would not trigger any potential page reclaim or memory compaction.
Allocating a temporary page defeats the purpose.Compaction can only happen for order > 0 temporary pages. Even if you used
single order = 0 page to gradually exchange e.g. a THP, it should be better than
u64. Allocating order = 0 should be a non-issue. If it's an issue, then the
system is in a bad state and physically contiguous layout is a secondary concern.
Quoted 8 lines
Quoted 6 lines
You are right if we only need to allocate one order-0 page. But this also means
we can only exchange two pages at a time. We need to add a lock to make sure the temporary page is used exclusively or we need to keep allocating temporary pages
when multipleexchange_pages()are happening at the same time.You allocate one temporary page per thread that's doing an
exchange_page().
Yeah, you are right. I think at most I need NR_CPU order-0 pages. I will try it. Thanks.
--
Best Regards,
Yan Zi
Re: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 28 lines · Zi Yan
Quoted 26 lines · Matthew Wilcox
Quoted 24 lines · Zi Yan
Quoted 8 lines · Vlastimil Babka
Quoted 6 lines · Zi Yan
The purpose of proposing
exchange_pages()is to avoid allocating any new
page,
so that we would not trigger any potential page reclaim or memory compaction.
Allocating a temporary page defeats the purpose.Quoted 8 lines
Compaction can only happen for order > 0 temporary pages. Even if you used
single order = 0 page to gradually exchange e.g. a THP, it should be better than
u64. Allocating order = 0 should be a non-issue. If it's an issue, then the
system is in a bad state and physically contiguous layout is a secondary concern.You are right if we only need to allocate one order-0 page. But this also means
we can only exchange two pages at a time. We need to add a lock to make sure the temporary page is used exclusively or we need to keep allocating temporary pages
when multipleexchange_pages()are happening at the same time.
Quoted 4 lines
Quoted 1 line
You allocate one temporary page per thread that's doing an
exchange_page().Yeah, you are right. I think at most I need
NR_CPUorder-0 pages. I will try it. Thanks.
But the location of this temp page matters as well because you would like to saturate the inter node interface. It needs to be either of the nodes where the source or destination page belongs. Any other node would generate two internode copy process which is not what you intend here I guess.
Matthew Wilcox
maintainerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 4 lines · Anshuman Khandual
But the location of this temp page matters as well because you would like to saturate the inter node interface. It needs to be either of the nodes where the source or destination page belongs. Any other node would generate two internode copy process which is not what you intend here I guess.
That makes no sense. It should be allocated on the local node of the CPU performing the copy. If the CPU is in node A, the destination is in node B and the source is in node C, then you're doing 4k worth of reads from node C, 4k worth of reads from node B, 4k worth of writes to node C followed by 4k worth of writes to node B. Eventually the 4k of dirty cachelines on node A will be written back from cache to the local memory (... or not, if that page gets reused for some other purpose first).
If you allocate the page on node B or node C, that's an extra 4k of writes to be sent across the inter-node link.
Re: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 13 lines · Matthew Wilcox
Quoted 4 lines · Anshuman Khandual
But the location of this temp page matters as well because you would like to saturate the inter node interface. It needs to be either of the nodes where the source or destination page belongs. Any other node would generate two internode copy process which is not what you intend here I guess.
That makes no sense. It should be allocated on the local node of the CPU performing the copy. If the CPU is in node A, the destination is in node B and the source is in node C, then you're doing 4k worth of reads from node C, 4k worth of reads from node B, 4k worth of writes to node C followed by 4k worth of writes to node B. Eventually the 4k of dirty cachelines on node A will be written back from cache to the local memory (... or not, if that page gets reused for some other purpose first).
Quoted 2 lines
If you allocate the page on node B or node C, that's an extra 4k of writes to be sent across the inter-node link.
Thats right there will be an extra remote write. My assumption was that the CPU performing the copy belongs to either node B or node C.
Zi Yan
reviewerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 14 lines · Anshuman Khandual
Quoted 13 lines · Matthew Wilcox
Quoted 4 lines · Anshuman Khandual
But the location of this temp page matters as well because you would like to saturate the inter node interface. It needs to be either of the nodes where the source or destination page belongs. Any other node would generate two internode copy process which is not what you intend here I guess.
That makes no sense. It should be allocated on the local node of the CPU performing the copy. If the CPU is in node A, the destination is in node B and the source is in node C, then you're doing 4k worth of reads from node C, 4k worth of reads from node B, 4k worth of writes to node C followed by 4k worth of writes to node B. Eventually the 4k of dirty cachelines on node A will be written back from cache to the local memory (... or not, if that page gets reused for some other purpose first).
Quoted 5 lines
Quoted 2 lines
If you allocate the page on node B or node C, that's an extra 4k of writes to be sent across the inter-node link.
Thats right there will be an extra remote write. My assumption was that the CPU performing the copy belongs to either node B or node C.
I have some interesting throughput results for exchange per u64 and exchange per 4KB page. What I discovered is that using a 4KB page as the temporary storage for exchanging 2MB THPs does not improve the throughput. On contrary, when we are exchanging more than 2^4=16 THPs, exchanging per 4KB page has lower throughput than exchanging per u64. Please see results below.
The experiments are done on a two socket machine with two Intel Xeon E5-2640 v3 CPUs. All exchanges are done via the QPI link across two sockets.
Results
===
Throughput (GB/s) of exchanging 2 order-N 2MB pages between two NUMA nodes
| 2mb_page_order | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | u64 | 5.31 | 5.58 | 5.89 | 5.69 | 8.97 | 9.51 | 9.21 | 9.50 | 9.57 | 9.62 | per_page | 5.85 | 6.48 | 6.20 | 5.26 | 7.22 | 7.25 | 7.28 | 7.30 | 7.32 | 7.31
Normalized throughput (to per_page)
2mb_page_order | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
| 8 | 9
u64 | 0.90 | 0.86 | 0.94 | 1.08 | 1.24 | 1.31 |1.26 | 1.30
| 1.30 | 1.31Exchange page code
===
For exchanging per u64, I use the following function:
static void exchange_page(char *to, char *from)
{
u64 tmp;
int i;
for (i = 0; i < PAGE_SIZE; i += sizeof(tmp)) {
tmp = *((u64 *)(from + i));
*((u64 *)(from + i)) = *((u64 *)(to + i));
*((u64 *)(to + i)) = tmp;
}
}For exchange per 4KB, I use the following function:
static void exchange_page2(char *to, char *from)
{
int cpu = smp_processor_id();
VM_BUG_ON(!in_atomic());
if (!page_tmp[cpu]) {
int nid = cpu_to_node(cpu);
struct page *page_tmp_page = alloc_pages_node(nid, GFP_KERNEL, 0);
if (!page_tmp_page) {
exchange_page(to, from);
return;
}
page_tmp[cpu] = kmap(page_tmp_page);
}
copy_page(page_tmp[cpu], to);
copy_page(to, from);
copy_page(from, page_tmp[cpu]);
}where page_tmp is pre-allocated local to each CPU and alloc_pages_node() above is for hot-added CPUs, which is not used in the tests.
The kernel is available at: https://gitlab.com/ziy/linux-contig-mem-rfc
To do a comparison, you can clone this repo: https://gitlab.com/ziy/thp-migration-bench,
then make, ./run_test.sh, and ./get_results.sh using the kernel from above.
Let me know if I missed anything or did something wrong. Thanks.
--
Best Regards,
Yan Zi
Re: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
In reply to Zi YanQuoted 4 lines · Zi Yan
From: Zi Yan <[email protected]>
In stead of using two
migrate_pages(), a singleexchange_pages()would be sufficient and without allocating new pages.
So i believe it would be better to arrange the code differently instead of having one function that special case combination, define function for each one ie:
exchange_anon_to_share()
exchange_anon_to_anon()
exchange_share_to_share()
Then you could define function to test if a page is in correct states:
can_exchange_anon_page() // return true if page can be exchange can_exchange_share_page()
In fact both of this function can be factor out as common helpers with the existing migrate code within migrate.c This way we would have one place only where we need to handle all the special casing, test and exceptions.
Other than that i could not spot anything obviously wrong but i did not spent enough time to check everything. Re-architecturing the code like i propose above would make this a lot easier to review i believe.
Cheers,
Jérôme
Quoted 11 lines
Signed-off-by Zi Yan <[email protected]>--- include/linux/ksm.h | 5 + mm/Makefile | 1 + mm/exchange.c | 846 ++++++++++++++++++++++++++++++++++++++++++++ mm/internal.h | 6 + mm/ksm.c | 35 ++ mm/migrate.c | 4 +- 6 files changed, 895 insertions(+), 2 deletions(-) create mode 100644 mm/exchange.c
[...]
mm/exchange.c line 602
Quoted 10 lines
mm/exchange.c new file +10
-0+ from_page_count =page_count (from_page); + from_map_count =page_mapcount (from_page); + to_page_count =page_count (to_page); + to_map_count =page_mapcount (to_page); + from_flags = from_page->flags; + to_flags = to_page->flags; + from_mapping = from_page->mapping; + to_mapping = to_page->mapping; + from_index = from_page->index; + to_index = to_page->index;
Those are not use anywhere ...
Zi Yan
reviewerRe: [RFC PATCH 01/31] mm: migrate: Add exchange_pages to exchange two lists of pages.
Quoted 17 lines · Jerome Glisse
Quoted 4 lines · Zi Yan
From: Zi Yan <[email protected]>
In stead of using two
migrate_pages(), a singleexchange_pages()would be sufficient and without allocating new pages.So i believe it would be better to arrange the code differently instead of having one function that special case combination, define function for each one ie:
exchange_anon_to_share()
exchange_anon_to_anon()
exchange_share_to_share()Then you could define function to test if a page is in correct states:
can_exchange_anon_page()// return true if page can be exchangecan_exchange_share_page()
Quoted 8 lines
In fact both of this function can be factor out as common helpers with the existing migrate code within
migrate.cThis way we would have one place only where we need to handle all the special casing, test and exceptions.Other than that i could not spot anything obviously wrong but i did not spent enough time to check everything. Re-architecturing the code like i propose above would make this a lot easier to review i believe.
Thank you for reviewing the patch. Your suggestions are very helpful. I will restructure the code to help people review it.
Quoted 4 lines
Quoted 4 lines
mm/exchange.c new file +4
-0+ from_page_count =page_count (from_page); + from_map_count =page_mapcount (from_page); + to_page_count =page_count (to_page); + to_map_count =page_mapcount (to_page);
Quoted 8 lines
mm/exchange.c line 602
Quoted 6 lines
mm/exchange.c new file +6
-0+ from_flags = from_page->flags; + to_flags = to_page->flags; + from_mapping = from_page->mapping; + to_mapping = to_page->mapping; + from_index = from_page->index; + to_index = to_page->index;Those are not use anywhere ...
Will remove them. Thanks.
--
Best Regards,
Yan Zi
RFC PATCH 2/31 mm: migrate: Add THP exchange support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Add support for exchange between two THPs.
--- mm/exchange.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-)
mm/exchange.c +38 -9
@@ -48,7 +48,8 @@ struct page_flags { unsigned int page_swapcache:1; unsigned int page_writeback:1; unsigned int page_private:1; - unsigned int __pad:3; + unsigned int page_doublemap:1; + unsigned int __pad:2; };
@@ -125,20 +126,23 @@ static void exchange_huge_page(struct page *dst, struct page *src)static void exchange_page_flags (struct page *to_page,struct page *from_page) {int from_cpupid, to_cpupid; -struct page_flags from_page_flags, to_page_flags; +struct page_flags from_page_flags = {0 }, to_page_flags = {0 };struct mem_cgroup *to_memcg =page_memcg (to_page), *from_memcg =page_memcg (from_page); from_cpupid =page_cpupid_xchg_last (from_page,-1 ); - from_page_flags.page_error =TestClearPageError (from_page); + from_page_flags.page_error =PageError (from_page); +if (from_page_flags.page_error) +ClearPageError (from_page); from_page_flags.page_referenced =TestClearPageReferenced (from_page); from_page_flags.page_uptodate =PageUptodate (from_page);ClearPageUptodate (from_page); from_page_flags.page_active =TestClearPageActive (from_page); from_page_flags.page_unevictable =TestClearPageUnevictable (from_page); from_page_flags.page_checked =PageChecked (from_page); -ClearPageChecked (from_page); +if (from_page_flags.page_checked) +ClearPageChecked (from_page); from_page_flags.page_mappedtodisk =PageMappedToDisk (from_page);ClearPageMappedToDisk (from_page); from_page_flags.page_dirty =PageDirty (from_page);
@@ -148,18 +152,22 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page)clear_page_idle (from_page); from_page_flags.page_swapcache =PageSwapCache (from_page); from_page_flags.page_writeback =test_clear_page_writeback (from_page); + from_page_flags.page_doublemap =PageDoubleMap (from_page); to_cpupid =page_cpupid_xchg_last (to_page,-1 ); - to_page_flags.page_error =TestClearPageError (to_page); + to_page_flags.page_error =PageError (to_page); +if (to_page_flags.page_error) +ClearPageError (to_page); to_page_flags.page_referenced =TestClearPageReferenced (to_page); to_page_flags.page_uptodate =PageUptodate (to_page);ClearPageUptodate (to_page); to_page_flags.page_active =TestClearPageActive (to_page); to_page_flags.page_unevictable =TestClearPageUnevictable (to_page); to_page_flags.page_checked =PageChecked (to_page); -ClearPageChecked (to_page); +if (to_page_flags.page_checked) +ClearPageChecked (to_page); to_page_flags.page_mappedtodisk =PageMappedToDisk (to_page);ClearPageMappedToDisk (to_page); to_page_flags.page_dirty =PageDirty (to_page);
@@ -169,6 +177,7 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page)clear_page_idle (to_page); to_page_flags.page_swapcache =PageSwapCache (to_page); to_page_flags.page_writeback =test_clear_page_writeback (to_page); + to_page_flags.page_doublemap =PageDoubleMap (to_page);/* set to_page */ if (from_page_flags.page_error)
@@ -195,6 +204,8 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page)set_page_young (to_page);if (from_page_flags.page_is_idle)set_page_idle (to_page); +if (from_page_flags.page_doublemap) +SetPageDoubleMap (to_page);/* set from_page */ if (to_page_flags.page_error)
@@ -221,6 +232,8 @@ static void exchange_page_flags(struct page *to_page, struct page *from_page)set_page_young (from_page);if (to_page_flags.page_is_idle)set_page_idle (from_page); +if (to_page_flags.page_doublemap) +SetPageDoubleMap (from_page);/* * Copy NUMA information to the new page, to prevent over-eager
@@ -280,6 +293,7 @@ static int exchange_page_move_mapping(struct address_space *to_mapping,VM_BUG_ON_PAGE (to_mapping !=page_mapping (to_page), to_page);VM_BUG_ON_PAGE (from_mapping !=page_mapping (from_page), from_page); +VM_BUG_ON (PageCompound (from_page) !=PageCompound (to_page));if (!to_mapping) {/* Anonymous page without mapping */
@@ -600,7 +614,6 @@ static int unmap_and_exchange(struct page *from_page, to_mapping = to_page->mapping; from_index = from_page->index; to_index = to_page->index; -/* * Corner case handling: * 1. When a new swap-cache page is read into, it is added to the LRU
@@ -691,6 +704,23 @@ static int unmap_and_exchange(struct page *from_page,return rc; } +static bool can_be_exchanged (struct page *from,struct page *to) +{ +if (PageCompound (from) !=PageCompound (to)) +return false; + +if (PageHuge (from) !=PageHuge (to)) +return false; + +if (PageHuge (from) ||PageHuge (to)) +return false; + +if (compound_order (from) !=compound_order (to)) +return false; + +return true; +} +/* * Exchange pages in the exchange_list *
@@ -751,9 +781,8 @@ static int exchange_pages(struct list_head *exchange_list,continue ; } -/* TODO: compound page not supported */ /* to_page can be file-backed page */ -if (PageCompound (from_page) || +if (!can_be_exchanged (from_page, to_page) ||page_mapping (from_page) ) { ++failed;
Signature · 1 line
2.20.1
RFC PATCH 3/31 mm: migrate: Add tmpfs exchange support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
tmpfs uses the same migrate routine as anonymous pages, enabling exchange pages for it is easy.
--- mm/exchange.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/exchange.c +4 -1
@@ -466,7 +466,10 @@ static int exchange_from_to_pages(struct page *to_page, struct page *from_page, rc =exchange_page_move_mapping (to_page_mapping, from_page_mapping, to_page, from_page,NULL ,NULL , mode,0 ,0 ); }else { -if (to_page_mapping->a_ops->migratepage == buffer_migrate_page) { +/* shmem */ +if (to_page_mapping->a_ops->migratepage == migrate_page) + goto exchange_mappings; +else if (to_page_mapping->a_ops->migratepage == buffer_migrate_page) {if (!page_has_buffers (to_page)) goto exchange_mappings;
Signature · 1 line
2.20.1
RFC PATCH 4/31 mm: add mem_defrag functionality.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Create contiguous physical memory regions by migrating/exchanging pages.
1. It scans all VMAs in one process and determines an anchor pair (VPN, PFN) for each VMA.
2. Then, it migrates/exchanges pages in each VMA to make them aligned with the anchor pair.
At the end of day, a physically contiguous region should be created for each VMA in the physical address range [PFN, PFN + VMA size).
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/exec.c | 4 + include/linux/mem_defrag.h | 60 + include/linux/mm.h | 12 + include/linux/mm_types.h | 4 + include/linux/sched/coredump.h | 3 + include/linux/syscalls.h | 3 + include/linux/vm_event_item.h | 23 + include/uapi/asm-generic/mman-common.h | 3 + kernel/fork.c | 9 + kernel/sysctl.c | 79 +- mm/Makefile | 1 + mm/compaction.c | 17 +- mm/huge_memory.c | 4 + mm/internal.h | 28 + mm/khugepaged.c | 1 + mm/madvise.c | 15 + mm/mem_defrag.c | 1782 ++++++++++++++++++++++++ mm/memory.c | 7 + mm/mmap.c | 29 + mm/page_alloc.c | 4 +- mm/vmstat.c | 21 + 22 files changed, 2096 insertions(+), 14 deletions(-) create mode 100644 include/linux/mem_defrag.h create mode 100644 mm/mem_defrag.c
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -343,6 +343,7 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +335 common scan_process_memory __x64_sys_scan_process_memory # # x32-specific system call numbers start at 512 to avoid cache impact
fs/exec.c +4 -0
@@ -1010,7 +1010,11 @@ static int exec_mmap(struct mm_struct *mm) {struct task_struct *tsk;struct mm_struct *old_mm, *active_mm; +int move_mem_defrag = current->mm ? +test_bit (MMF_VM_MEM_DEFRAG_ALL , ¤t->mm->flags):0 ; +if (move_mem_defrag && mm) +set_bit (MMF_VM_MEM_DEFRAG_ALL , &mm->flags);/* Notify parent that we're no longer interested in the old VM */ tsk = current; old_mm = current->mm;
include/linux/mem_defrag.h new file +60 -0
60 lines not on this page · raw message
include/linux/mm.h +12 -0
@@ -251,13 +251,20 @@ extern unsigned int kobjsize(const void *objp);#define VM_HIGH_ARCH_BIT_2 34/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_3 35/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_BIT_4 36/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_5 37/* bit only usable on 64-bit architectures */ +#define VM_HIGH_ARCH_BIT_6 38/* bit only usable on 64-bit architectures */ #define VM_HIGH_ARCH_0 BIT(VM_HIGH_ARCH_BIT_0)#define VM_HIGH_ARCH_1 BIT(VM_HIGH_ARCH_BIT_1)#define VM_HIGH_ARCH_2 BIT(VM_HIGH_ARCH_BIT_2)#define VM_HIGH_ARCH_3 BIT(VM_HIGH_ARCH_BIT_3)#define VM_HIGH_ARCH_4 BIT(VM_HIGH_ARCH_BIT_4) +#define VM_HIGH_ARCH_5 BIT(VM_HIGH_ARCH_BIT_5) +#define VM_HIGH_ARCH_6 BIT(VM_HIGH_ARCH_BIT_6)#endif /* CONFIG_ARCH_USES_HIGH_VMA_FLAGS */ +#define VM_MEMDEFRAG VM_HIGH_ARCH_5/* memory defrag */ +#define VM_NOMEMDEFRAG VM_HIGH_ARCH_6/* no memory defrag */ +#ifdef CONFIG_ARCH_HAS_PKEYS # define VM_PKEY_SHIFT VM_HIGH_ARCH_BIT_0# define VM_PKEY_BIT0 VM_HIGH_ARCH_0/* A protection key is a 4-bit value */
@@ -487,6 +494,9 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); + vma->anchor_page_rb =RB_ROOT_CACHED ; + vma->vma_create_jiffies = jiffies; + vma->vma_defrag_jiffies =0 ; }static inline void vma_set_anonymous (struct vm_area_struct *vma)
@@ -2837,6 +2847,8 @@ static inline bool debug_guardpage_enabled(void) { return false; }static inline bool page_is_guard (struct page *page) {return false; }#endif /* CONFIG_DEBUG_PAGEALLOC */ +void free_anchor_pages (struct vm_area_struct *vma); +#if MAX_NUMNODES >1 void __init setup_nr_node_ids (void );#else
include/linux/mm_types.h +4 -0
@@ -328,6 +328,10 @@ struct vm_area_struct {struct mempolicy *vm_policy;/* NUMA policy for the VMA */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +struct rb_root_cached anchor_page_rb; +unsigned long vma_create_jiffies;/* life time of the vma */ +unsigned long vma_modify_jiffies;/* being modified time of the vma */ +unsigned long vma_defrag_jiffies;/* being defragged time of the vma */ } __randomize_layout;struct core_thread {
include/linux/sched/coredump.h +3 -0
@@ -76,5 +76,8 @@ static inline int get_dumpable(struct mm_struct *mm)#define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK) +#define MMF_VM_MEM_DEFRAG 26/* set mm is added to do mem defrag */ +#define MMF_VM_MEM_DEFRAG_ALL 27/* all vmas in the mm will be mem defrag */ +#endif /* _LINUX_SCHED_COREDUMP_H */
include/linux/syscalls.h +3 -0
@@ -926,6 +926,8 @@ asmlinkage long sys_statx(int dfd, const char __user *path, unsigned flags, unsigned mask, struct statx __user *buffer);asmlinkage long sys_rseq (struct rseq __user *rseq,uint32_t rseq_len,int flags,uint32_t sig); +asmlinkage long sys_scan_process_memory (pid_t pid,char __user *out_buf, +int buf_len,int action);/* * Architecture-specific system calls
@@ -1315,4 +1317,5 @@ static inline unsigned int ksys_personality(unsigned int personality) return old; } +#endif
include/linux/vm_event_item.h +23 -0
@@ -109,6 +109,29 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,#ifdef CONFIG_SWAP SWAP_RA ,SWAP_RA_HIT , +#endif +/* MEM_DEFRAG STATS */ +MEM_DEFRAG_DEFRAG_NUM , +MEM_DEFRAG_SCAN_NUM , +MEM_DEFRAG_DST_FREE_PAGES , +MEM_DEFRAG_DST_ANON_PAGES , +MEM_DEFRAG_DST_FILE_PAGES , +MEM_DEFRAG_DST_NONLRU_PAGES , +MEM_DEFRAG_DST_FREE_PAGES_FAILED , +MEM_DEFRAG_DST_FREE_PAGES_OVERFLOW_FAILED , +MEM_DEFRAG_DST_ANON_PAGES_FAILED , +MEM_DEFRAG_DST_FILE_PAGES_FAILED , +MEM_DEFRAG_DST_NONLRU_PAGES_FAILED , +MEM_DEFRAG_SRC_ANON_PAGES_FAILED , +MEM_DEFRAG_SRC_COMP_PAGES_FAILED , +MEM_DEFRAG_DST_SPLIT_HUGEPAGES , +#ifdef CONFIG_COMPACTION +/* memory compaction */ +COMPACT_MIGRATE_PAGES , +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +/* thp collapse */ +THP_COLLAPSE_MIGRATE_PAGES , #endifNR_VM_EVENT_ITEMS };
include/uapi/asm-generic/mman-common.h +3 -0
@@ -66,6 +66,9 @@#define MADV_WIPEONFORK 18/* Zero memory on fork, child only */ #define MADV_KEEPONFORK 19/* Undo MADV_WIPEONFORK */ +#define MADV_MEMDEFRAG 20/* Worth backing with hugepages */ +#define MADV_NOMEMDEFRAG 21/* Not worth backing with hugepages */ +/* compatibility flags */ #define MAP_FILE 0
kernel/fork.c +9 -0
@@ -92,6 +92,7 @@#include <linux/livepatch.h> #include <linux/thread_info.h> #include <linux/stackleak.h> +#include <linux/mem_defrag.h> #include <asm/pgtable.h> #include <asm/pgalloc.h>
@@ -343,12 +344,16 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)if (new) { *new = *orig;INIT_LIST_HEAD (&new->anon_vma_chain); + new->anchor_page_rb =RB_ROOT_CACHED ; + new->vma_create_jiffies = jiffies; + new->vma_defrag_jiffies =0 ; }return new; }void vm_area_free (struct vm_area_struct *vma) { +free_anchor_pages (vma);kmem_cache_free (vm_area_cachep, vma); }
@@ -496,6 +501,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,if (retval) goto out; retval =khugepaged_fork (mm, oldmm); +if (retval) + goto out; + retval =kmem_defragd_fork (mm, oldmm);if (retval) goto out;
@@ -1044,6 +1052,7 @@ static inline void __mmput(struct mm_struct *mm)exit_aio (mm);ksm_exit (mm);khugepaged_exit (mm);/* must run before exit_mmap */ +kmem_defragd_exit (mm);exit_mmap (mm);mm_put_huge_zero_page (mm);set_mm_exe_file (mm,NULL );
kernel/sysctl.c +69 -10
137 lines not on this page · raw message
mm/Makefile +1 -0
@@ -44,6 +44,7 @@ obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ obj-y += init-mm.o obj-y += memblock.o obj-y += exchange.o +obj-y += mem_defrag.o ifdef CONFIG_MMU obj-$(CONFIG_ADVISE_SYSCALLS) += madvise.o
mm/compaction.c +15 -2
@@ -50,7 +50,7 @@ static inline void count_compact_events(enum vm_event_item item, long delta)#define pageblock_start_pfn (pfn) block_start_pfn(pfn, pageblock_order)#define pageblock_end_pfn (pfn) block_end_pfn(pfn, pageblock_order) -static unsigned long release_freepages (struct list_head *freelist) +unsigned long release_freepages (struct list_head *freelist) {struct page *page, *next;unsigned long high_pfn =0 ;
@@ -58,7 +58,10 @@ static unsigned long release_freepages(struct list_head *freelist)list_for_each_entry_safe (page, next, freelist, lru) {unsigned long pfn =page_to_pfn (page);list_del (&page->lru); -__free_page (page); +if (PageCompound (page)) +__free_pages (page,compound_order (page)); +else +__free_page (page);if (pfn > high_pfn) high_pfn = pfn; }
@@ -1593,6 +1596,8 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_controwhile ((ret =compact_finished (zone, cc)) ==COMPACT_CONTINUE ) {int err; +int num_migrated_pages =0 ; +struct page *iter;switch (isolate_migratepages (zone, cc)) {case ISOLATE_ABORT :
@@ -1611,6 +1616,9 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro ; } +list_for_each_entry (iter, &cc->migratepages, lru) + num_migrated_pages++; + err =migrate_pages (&cc->migratepages, compaction_alloc, compaction_free, (unsigned long )cc, cc->mode,MR_COMPACTION );
@@ -1618,6 +1626,11 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_controtrace_mm_compaction_migratepages (cc->nr_migratepages, err, &cc->migratepages); +list_for_each_entry (iter, &cc->migratepages, lru) + num_migrated_pages--; + +count_vm_events (COMPACT_MIGRATE_PAGES , num_migrated_pages); +/* All pages were either migrated or will be released */ cc->nr_migratepages =0 ;if (err) {
mm/huge_memory.c +4 -0
@@ -33,6 +33,7 @@#include <linux/page_idle.h> #include <linux/shmem_fs.h> #include <linux/oom.h> +#include <linux/mem_defrag.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
@@ -695,6 +696,9 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf)return VM_FAULT_OOM ;if (unlikely (khugepaged_enter (vma, vma->vm_flags)))return VM_FAULT_OOM ; +/* Make it defrag */ +if (unlikely (kmem_defragd_enter (vma, vma->vm_flags))) +return VM_FAULT_OOM ;if (!(vmf->flags &FAULT_FLAG_WRITE ) && !mm_forbids_zeropage (vma->vm_mm) &&transparent_hugepage_use_zero_page ()) {
mm/internal.h +28 -0
@@ -15,6 +15,7 @@#include <linux/mm.h> #include <linux/pagemap.h> #include <linux/tracepoint-defs.h> +#include <linux/interval_tree.h> /* * The set of flags that only affect watermark checking and reclaim
@@ -549,4 +550,31 @@ bool buffer_migrate_lock_buffers(struct buffer_head *head,int writeout (struct address_space *mapping,struct page *page);extern int exchange_two_pages (struct page *page1,struct page *page2); +struct anchor_page_info { +struct list_head list; +struct page *anchor_page; +unsigned long vaddr; +unsigned long start; +unsigned long end; +}; + +struct anchor_page_node { +struct interval_tree_node node; +unsigned long anchor_pfn;/* struct page can be calculated from pfn_to_page() */ +unsigned long anchor_vpn; +}; + +unsigned long release_freepages (struct list_head *freelist); + +void free_anchor_pages (struct vm_area_struct *vma); + +extern int exchange_two_pages (struct page *page1,struct page *page2); + +void expand (struct zone *zone,struct page *page, +int low,int high,struct free_area *area, +int migratetype); + +void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags, +unsigned int alloc_flags); +#endif /* __MM_INTERNAL_H */
mm/khugepaged.c +1 -0
@@ -660,6 +660,7 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); +count_vm_event (THP_COLLAPSE_MIGRATE_PAGES );VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/*
mm/madvise.c +15 -0
@@ -24,6 +24,7 @@#include <linux/swapops.h> #include <linux/shmem_fs.h> #include <linux/mmu_notifier.h> +#include <linux/mem_defrag.h> #include <asm/tlb.h>
@@ -616,6 +617,13 @@ static long madvise_remove(struct vm_area_struct *vma,return error; } +static long madvise_memdefrag (struct vm_area_struct *vma, +struct vm_area_struct **prev, +unsigned long start,unsigned long end,int behavior) +{ + *prev = vma; +return memdefrag_madvise (vma, &vma->vm_flags, behavior); +}#ifdef CONFIG_MEMORY_FAILURE /* * Error injection support for memory error handling.
@@ -697,6 +705,9 @@ madvise_vma(struct vm_area_struct *vma, struct vm_area_struct **prev,case MADV_FREE :case MADV_DONTNEED :return madvise_dontneed_free (vma, prev, start, end, behavior); +case MADV_MEMDEFRAG : +case MADV_NOMEMDEFRAG : +return madvise_memdefrag (vma, prev, start, end, behavior);default :return madvise_behavior (vma, prev, start, end, behavior); }
@@ -731,6 +742,8 @@ madvise_behavior_valid(int behavior)case MADV_SOFT_OFFLINE :case MADV_HWPOISON :#endif +case MADV_MEMDEFRAG : +case MADV_NOMEMDEFRAG :return true;default :
@@ -785,6 +798,8 @@ madvise_behavior_valid(int behavior)* MADV_DONTDUMP - the application wants to prevent pages in the given range * from being included in its core dump. * MADV_DODUMP - cancel MADV_DONTDUMP: no longer exclude from core dump. +* MADV_MEMDEFRAG - allow mem defrag running on this region. +* MADV_NOMEMDEFRAG - no mem defrag here. * * return values: * zero - success
mm/mem_defrag.c new file +1782 -0
1,783 lines not on this page · raw message
mm/memory.c +7 -0
@@ -69,6 +69,7 @@#include <linux/userfaultfd_k.h> #include <linux/dax.h> #include <linux/oom.h> +#include <linux/mem_defrag.h> #include <asm/io.h> #include <asm/mmu_context.h>
@@ -2926,6 +2927,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)/* Allocate our own private page. */ if (unlikely (anon_vma_prepare (vma))) goto oom; +/* Make it defrag */ +if (unlikely (kmem_defragd_enter (vma, vma->vm_flags))) + goto oom; page =alloc_zeroed_user_highpage_movable (vma, vmf->address);if (!page) goto oom;
@@ -3844,6 +3848,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,p4d_t *p4d;vm_fault_t ret; +/* Zi: page faults modify vma */ + vma->vma_modify_jiffies = jiffies; + pgd =pgd_offset (mm, address); p4d =p4d_alloc (mm, pgd, address);if (!p4d)
mm/mmap.c +29 -0
@@ -169,6 +169,28 @@ void unlink_file_vma(struct vm_area_struct *vma) } } +void free_anchor_pages (struct vm_area_struct *vma) +{ +struct interval_tree_node *node; + +if (!vma) +return ; + +if (RB_EMPTY_ROOT (&vma->anchor_page_rb.rb_root)) +return ; + +for (node =interval_tree_iter_first (&vma->anchor_page_rb, +0 , (unsigned long )-1 ); + node;) { +struct anchor_page_node *anchor_node =container_of (node, + struct anchor_page_node, node); +interval_tree_remove (node, &vma->anchor_page_rb); + node =interval_tree_iter_next (node,0 , (unsigned long )-1 ); +kfree (anchor_node); + } + +} +/* * Close a vm structure and free it, returning the next. */
@@ -181,6 +203,7 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma) vma->vm_ops->close (vma);if (vma->vm_file)fput (vma->vm_file); +free_anchor_pages (vma);mpol_put (vma_policy (vma));vm_area_free (vma);return next;
@@ -725,10 +748,15 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,long adjust_next =0 ;int remove_next =0 ; +/*free_anchor_pages(vma);*/ + + vma->vma_modify_jiffies = jiffies; +if (next && !insert) {struct vm_area_struct *exporter =NULL , *importer =NULL ;if (end >= next->vm_end) { +/*free_anchor_pages(next);*/ /* * vma expands, overlapping all the next, and * perhaps the one after too (mprotect case 6).
@@ -775,6 +803,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, exporter = next->vm_next; }else if (end > next->vm_start) { +/*free_anchor_pages(next);*/ /* * vma expands, overlapping part of the next: * mprotect case 5 shifting the boundary up.
mm/page_alloc.c +2 -2
@@ -1828,7 +1828,7 @@ void __init init_cma_reserved_pageblock(struct page *page)* * -- nyc */ -static inline void expand (struct zone *zone,struct page *page, +inline void expand (struct zone *zone,struct page *page,int low,int high,struct free_area *area,int migratetype) {
@@ -1950,7 +1950,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order,set_page_owner (page, order, gfp_flags); } -static void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags, +void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags,unsigned int alloc_flags) {int i;
mm/vmstat.c +21 -0
@@ -1293,6 +1293,27 @@ const char * const vmstat_text[] = { "swap_ra", "swap_ra_hit", #endif + "memdefrag_defrag", + "memdefrag_scan", + "memdefrag_dest_free_pages", + "memdefrag_dest_anon_pages", + "memdefrag_dest_file_pages", + "memdefrag_dest_non_lru_pages", + "memdefrag_dest_free_pages_failed", + "memdefrag_dest_free_pages_overflow_failed", + "memdefrag_dest_anon_pages_failed", + "memdefrag_dest_file_pages_failed", + "memdefrag_dest_nonlru_pages_failed", + "memdefrag_src_anon_pages_failed", + "memdefrag_src_compound_pages_failed", + "memdefrag_dst_split_hugepage", +#ifdef CONFIG_COMPACTION + "compact_migrate_pages", +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + "thp_collapse_migrate_pages" +#endif + #endif /* CONFIG_VM_EVENTS_COUNTERS */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA */
Signature · 1 line
2.20.1
RFC PATCH 5/31 mem_defrag: split a THP if either src or dst is THP only.
In reply to Zi YanFrom: Zi Yan <[email protected]>
During the process of generating physically contiguous memory, it is possible that we want to move a THP to a place with 512 base pages. Exchange pages has not implemented the exchange of a THP and 512 base pages. Instead, we can split the THP and exchange 512 base pages. This increases the chance of creating a large contiguous region. A split THP could be promoted back after all 512 pages are moved to the destination or if none of its subpages is moved.
In-place THP promotion will be introduced later in this patch serie.
--- mm/internal.h | 4 ++ mm/mem_defrag.c | 155 +++++++++++++++++++++++++++++++++++++----------- mm/page_alloc.c | 45 ++++++++++++++ 3 files changed, 168 insertions(+), 36 deletions(-)
mm/internal.h +4 -0
@@ -574,6 +574,10 @@ void expand(struct zone *zone, struct page *page, int low, int high, struct free_area *area, int migratetype); +int expand_free_page (struct zone *zone,struct page *buddy_head, +struct page *page,int buddy_order,int page_order, +struct free_area *area,int migratetype); +void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags,unsigned int alloc_flags);
mm/mem_defrag.c +119 -36
238 lines not on this page · raw message
mm/page_alloc.c +45 -0
51 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 6/31 mm: Make MAX_ORDER configurable in Kconfig for buddy allocator.
In reply to Zi YanFrom: Zi Yan <[email protected]>
To test 1GB THP implemented in the following patches, this patch enables changing MAX_ORDER of the buddy allocator.
It should be dropped later when we solely rely on mem_defrag to generate 1GB THPs.
--- arch/x86/Kconfig | 15 +++++++++++++++ arch/x86/include/asm/sparsemem.h | 4 ++-- 2 files changed, 17 insertions(+), 2 deletions(-)
arch/x86/Kconfig +15 -0
@@ -1665,6 +1665,21 @@ config X86_PMEM_LEGACY Say Y if unsure. +config FORCE_MAX_ZONEORDER + int "Maximum zone order" + range 11 20 + default "11" + help + The kernel memory allocator divides physically contiguous memory + blocks into "zones", where each zone is a power of two number of + pages. This option selects the largest power of two that the kernel + keeps in the memory allocator. If you need to allocate very large + blocks of physically contiguous memory, then you may need to + increase this value. + + This config option is actually maximum order plus one. For example, + a value of 11 means that the largest free memory block is 2^10 pages. + config HIGHPTE bool "Allocate 3rd-level pagetables from highmem" depends on HIGHMEM
arch/x86/include/asm/sparsemem.h +2 -2
@@ -21,12 +21,12 @@# define MAX_PHYSADDR_BITS 36# define MAX_PHYSMEM_BITS 36# else -# define SECTION_SIZE_BITS 26 +# define SECTION_SIZE_BITS 31# define MAX_PHYSADDR_BITS 32# define MAX_PHYSMEM_BITS 32# endif #else /* CONFIG_X86_32 */ -# define SECTION_SIZE_BITS 27/* matt - 128 is convenient right now */ +# define SECTION_SIZE_BITS 31/* matt - 128 is convenient right now */ # define MAX_PHYSADDR_BITS (pgtable_l5_enabled() ? 52 : 44)# define MAX_PHYSMEM_BITS (pgtable_l5_enabled() ? 52 : 46)#endif
Signature · 1 line
2.20.1
RFC PATCH 7/31 mm: deallocate pages with order > MAX_ORDER.
In reply to Zi YanFrom: Zi Yan <[email protected]>
When MAX_ORDER is not set to allocate 1GB pages and 1GB THPs are created from in-place promotion, we need this to properly free 1GB THPs.
--- mm/page_alloc.c | 36 ++++++++++++++++++++++++++++++------ 1 file changed, 30 insertions(+), 6 deletions(-)
mm/page_alloc.c +30 -6
@@ -1287,6 +1287,24 @@ void __meminit reserve_bootmem_region(phys_addr_t start, phys_addr_t end) } } +static void destroy_compound_gigantic_page (struct page *page, +unsigned int order) +{ +int i; +int nr_pages =1 << order; +struct page *p = page +1 ; + +atomic_set (compound_mapcount_ptr (page),0 ); +for (i =1 ; i < nr_pages; i++, p =mem_map_next (p, page, i)) { +clear_compound_head (p); +set_page_refcounted (p); + } + +set_compound_order (page,0 ); +__ClearPageHead (page); +set_page_refcounted (page); +} +static void __free_pages_ok (struct page *page,unsigned int order) {unsigned long flags;
@@ -1296,11 +1314,16 @@ static void __free_pages_ok(struct page *page, unsigned int order)if (!free_pages_prepare (page, order, true))return ; - migratetype =get_pfnblock_migratetype (page, pfn); -local_irq_save (flags); -__count_vm_events (PGFREE ,1 << order); -free_one_page (page_zone (page), page, pfn, order, migratetype); -local_irq_restore (flags); +if (order >MAX_ORDER ) { +destroy_compound_gigantic_page (page, order); +free_contig_range (page_to_pfn (page),1 << order); + }else { + migratetype =get_pfnblock_migratetype (page, pfn); +local_irq_save (flags); +__count_vm_events (PGFREE ,1 << order); +free_one_page (page_zone (page), page, pfn, order, migratetype); +local_irq_restore (flags); + } }static void __init __free_pages_boot_core (struct page *page,unsigned int order)
@@ -8281,6 +8304,8 @@ int alloc_contig_range(unsigned long start, unsigned long end, return ret; } +#endif +void free_contig_range (unsigned long pfn,unsigned nr_pages) {unsigned int count =0 ;
@@ -8293,7 +8318,6 @@ void free_contig_range(unsigned long pfn, unsigned nr_pages) }WARN (count !=0 ,"%d pages are still in use!\n" , count); } -#endif #ifdef CONFIG_MEMORY_HOTPLUG /*
Signature · 1 line
2.20.1
RFC PATCH 8/31 mm: add pagechain container for storing multiple pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
When depositing page table pages for 1GB THPs, we need 512 PTE pages + 1 PMD page. Instead of counting and depositing 513 pages, we can use the PMD page as a leader page and chain the rest 512 PTE pages with ->lru. This, however, prevents us depositing PMD pages with ->lru, which is currently used by depositing PTE pages for 2MB THPs. So add a new pagechain container for PMD pages.
--- include/linux/pagechain.h | 73 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 include/linux/pagechain.h
include/linux/pagechain.h new file +73 -0
@@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* include/linux/pagechain.h +* +* In many places it is efficient to batch an operation up against multiple +* pages. A pagechain is a multipage container which is used for that. +*/ + +#ifndef _LINUX_PAGECHAIN_H +#define _LINUX_PAGECHAIN_H + +#include <linux/slab.h> + +/* 14 pointers + two long's align the pagechain structure to a power of two */ +#define PAGECHAIN_SIZE 13 + +struct page ; + +struct pagechain { +struct list_head list; +unsigned int nr; +struct page *pages[PAGECHAIN_SIZE ]; +}; + +static inline void pagechain_init (struct pagechain *pchain) +{ + pchain->nr =0 ; +INIT_LIST_HEAD (&pchain->list); +} + +static inline void pagechain_reinit (struct pagechain *pchain) +{ + pchain->nr =0 ; +} + +static inline unsigned int pagechain_count (struct pagechain *pchain) +{ +return pchain->nr; +} + +static inline unsigned int pagechain_space (struct pagechain *pchain) +{ +return PAGECHAIN_SIZE - pchain->nr; +} + +static inline bool pagechain_empty (struct pagechain *pchain) +{ +return pchain->nr ==0 ; +} + +/* +* Add a page to a pagechain. Returns the number of slots still available. +*/ +static inline unsigned int pagechain_deposit (struct pagechain *pchain,struct page *page) +{ +VM_BUG_ON (!pagechain_space (pchain)); + pchain->pages[pchain->nr++] = page; +return pagechain_space (pchain); +} + +static inline struct page *pagechain_withdraw (struct pagechain *pchain) +{ +if (!pagechain_count (pchain)) +return NULL ; +return pchain->pages[--pchain->nr]; +} + +void __init pagechain_cache_init (void ); +struct pagechain *pagechain_alloc (void ); +void pagechain_free (struct pagechain *pchain); + +#endif /* _LINUX_PAGECHAIN_H */ +
Signature · 1 line
2.20.1
RFC PATCH 9/31 mm: thp: 1GB anonymous page implementation.
In reply to Zi YanFrom: Zi Yan <[email protected]>
This adds 1GB THP support for anonymous pages. Applications can get 1GB pages during page faults when their VMAs are larger than 1GB. For read-only 1GB zero THP, a shared 1GB zero THP is created for all readers.
--- arch/x86/include/asm/pgalloc.h | 58 +++++++ arch/x86/include/asm/pgtable.h | 2 + arch/x86/mm/pgtable.c | 25 +++ drivers/base/node.c | 4 +- fs/proc/meminfo.c | 3 +- include/asm-generic/pgtable.h | 3 + include/linux/huge_mm.h | 17 ++- include/linux/mm.h | 4 + include/linux/mm_types.h | 1 + include/linux/mmzone.h | 1 + include/linux/sched/coredump.h | 1 + include/linux/vm_event_item.h | 2 + kernel/fork.c | 5 + mm/huge_memory.c | 267 ++++++++++++++++++++++++++++++++- mm/memory.c | 28 +++- mm/page_alloc.c | 3 +- mm/pgtable-generic.c | 47 +++++- mm/rmap.c | 28 +++- mm/vmstat.c | 3 + 19 files changed, 484 insertions(+), 18 deletions(-)
arch/x86/include/asm/pgalloc.h +58 -0
@@ -49,6 +49,7 @@ extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);extern pte_t *pte_alloc_one_kernel (struct mm_struct *);extern pgtable_t pte_alloc_one (struct mm_struct *); +extern pgtable_t pte_alloc_order (struct mm_struct *,unsigned long ,int );/* Should really implement gc for free page table pages. This could be done with a reference count in struct page. */
@@ -65,6 +66,17 @@ static inline void pte_free(struct mm_struct *mm, struct page *pte)__free_page (pte); } +static inline void pte_free_order (struct mm_struct *mm,struct page *pte, +int order) +{ +int i; + +for (i =0 ; i < (1 <<order); i++) { +pgtable_page_dtor (&pte[i]); +__free_page (&pte[i]); + } +} +extern void ___pte_free_tlb (struct mmu_gather *tlb,struct page *pte); static inline void __pte_free_tlb(struct mmu_gather *tlb, struct page *pte,
@@ -123,6 +135,52 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)free_page ((unsigned long )pmd); } +static inline pmd_t *pmd_alloc_one_page_with_ptes (struct mm_struct *mm,unsigned long addr) +{ +pgtable_t pte_pgtables; +pmd_t *pmd; +spinlock_t *pmd_ptl; +int i; + + pte_pgtables =pte_alloc_order (mm, addr, +HPAGE_PUD_ORDER -HPAGE_PMD_ORDER ); +if (!pte_pgtables) +return NULL ; + + pmd =pmd_alloc_one (mm, addr); +if (unlikely (!pmd)) { +pte_free_order (mm, pte_pgtables, +HPAGE_PUD_ORDER -HPAGE_PMD_ORDER ); +return NULL ; + } + pmd_ptl =pmd_lock (mm, pmd); + +for (i =0 ; i < (1 <<(HPAGE_PUD_ORDER -HPAGE_PMD_ORDER )); i++) +pgtable_trans_huge_deposit (mm, pmd, pte_pgtables + i); + +spin_unlock (pmd_ptl); + +return pmd; +} + +static inline void pmd_free_page_with_ptes (struct mm_struct *mm,pmd_t *pmd) +{ +spinlock_t *pmd_ptl; +int i; + +BUG_ON ((unsigned long )pmd & (PAGE_SIZE -1 )); + pmd_ptl =pmd_lock (mm, pmd); + +for (i =0 ; i < (1 <<(HPAGE_PUD_ORDER -HPAGE_PMD_ORDER )); i++) { +pgtable_t pte_pgtable; + + pte_pgtable =pgtable_trans_huge_withdraw (mm, pmd); +pte_free (mm, pte_pgtable); + } + +spin_unlock (pmd_ptl); +pmd_free (mm, pmd); +}extern void ___pmd_free_tlb (struct mmu_gather *tlb,pmd_t *pmd); static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
arch/x86/include/asm/pgtable.h +2 -0
@@ -1165,6 +1165,8 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned longreturn native_pmdp_get_and_clear (pmdp); } +#define mk_pud (page, pgprot) pfn_pud(page_to_pfn(page), (pgprot)) +#define __HAVE_ARCH_PUDP_HUGE_GET_AND_CLEARstatic inline pud_t pudp_huge_get_and_clear (struct mm_struct *mm,unsigned long addr,pud_t *pudp)
arch/x86/mm/pgtable.c +25 -0
@@ -42,6 +42,31 @@ pgtable_t pte_alloc_one(struct mm_struct *mm)return pte; } +pgtable_t pte_alloc_order (struct mm_struct *mm,unsigned long address,int order) +{ +struct page *pte; +int i; + + pte =alloc_pages (__userpte_alloc_gfp, order); +if (!pte) +return NULL ; +split_page (pte, order); +for (i =1 ; i < (1 << order); i++) +set_page_private (pte + i,0 ); + +for (i =0 ; i < (1 <<order); i++) { +if (!pgtable_page_ctor (&pte[i])) { +__free_page (&pte[i]); +while (--i >=0 ) { +pgtable_page_dtor (&pte[i]); +__free_page (&pte[i]); + } +return NULL ; + } + } +return pte; +} +static int __init setup_userpte (char *arg) {if (!arg)
drivers/base/node.c +3 -1
@@ -150,7 +150,9 @@ static ssize_t node_read_meminfo(struct device *dev,#ifdef CONFIG_TRANSPARENT_HUGEPAGE , nid,K (node_page_state (pgdat,NR_ANON_THPS ) * -HPAGE_PMD_NR ), +HPAGE_PMD_NR ) + +K (node_page_state (pgdat,NR_ANON_THPS_PUD ) * +HPAGE_PUD_NR ), nid,K (node_page_state (pgdat,NR_SHMEM_THPS ) *HPAGE_PMD_NR ), nid, K(node_page_state(pgdat, NR_SHMEM_PMDMAPPED) *
fs/proc/meminfo.c +2 -1
@@ -131,7 +131,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v)#ifdef CONFIG_TRANSPARENT_HUGEPAGE show_val_kb (m,"AnonHugePages: " , -global_node_page_state (NR_ANON_THPS ) *HPAGE_PMD_NR ); +global_node_page_state (NR_ANON_THPS ) *HPAGE_PMD_NR + +global_node_page_state (NR_ANON_THPS_PUD ) *HPAGE_PUD_NR );show_val_kb (m,"ShmemHugePages: " ,global_node_page_state (NR_SHMEM_THPS ) *HPAGE_PMD_NR ); show_val_kb(m, "ShmemPmdMapped: ",
include/asm-generic/pgtable.h +3 -0
@@ -303,10 +303,13 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,#ifndef __HAVE_ARCH_PGTABLE_DEPOSITextern void pgtable_trans_huge_deposit (struct mm_struct *mm,pmd_t *pmdp,pgtable_t pgtable); +extern void pgtable_trans_huge_pud_deposit (struct mm_struct *mm,pud_t *pudp, +pgtable_t pgtable);#endif #ifndef __HAVE_ARCH_PGTABLE_WITHDRAWextern pgtable_t pgtable_trans_huge_withdraw (struct mm_struct *mm,pmd_t *pmdp); +extern pgtable_t pgtable_trans_huge_pud_withdraw (struct mm_struct *mm,pud_t *pudp);#endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE
include/linux/huge_mm.h +14 -3
@@ -18,10 +18,15 @@ extern int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD extern void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud); +extern int do_huge_pud_anonymous_page (struct vm_fault *vmf);#else static inline void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud) { } +extern int do_huge_pud_anonymous_page (struct vm_fault *vmf) +{ +return VM_FAULT_FALLBACK ; +}#endif extern vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd);
@@ -80,6 +85,9 @@ extern struct kobj_attribute shmem_enabled_attr;#define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT)#define HPAGE_PMD_NR (1<<HPAGE_PMD_ORDER) +#define HPAGE_PUD_ORDER (HPAGE_PUD_SHIFT-PAGE_SHIFT) +#define HPAGE_PUD_NR (1<<HPAGE_PUD_ORDER) +#ifdef CONFIG_TRANSPARENT_HUGEPAGE #define HPAGE_PMD_SHIFT PMD_SHIFT#define HPAGE_PMD_SIZE ((1UL) << HPAGE_PMD_SHIFT)
@@ -214,7 +222,7 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud,static inline int hpage_nr_pages (struct page *page) {if (unlikely (PageTransHuge (page))) -return HPAGE_PMD_NR ; +return (1 <<page[1 ].compound_order);return 1 ; }
@@ -226,10 +234,12 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr,extern vm_fault_t do_huge_pmd_numa_page (struct vm_fault *vmf,pmd_t orig_pmd);extern struct page *huge_zero_page; +extern struct page *huge_pud_zero_page;static inline bool is_huge_zero_page (struct page *page) { -return READ_ONCE (huge_zero_page) == page; +return (READ_ONCE (huge_zero_page) == page) || + (READ_ONCE (huge_pud_zero_page) == page); }static inline bool is_huge_zero_pmd (pmd_t pmd)
@@ -239,13 +249,14 @@ static inline bool is_huge_zero_pmd(pmd_t pmd)static inline bool is_huge_zero_pud (pud_t pud) { -return false; +return is_huge_zero_page (pud_page (pud)); }struct page *mm_get_huge_zero_page (struct mm_struct *mm);void mm_put_huge_zero_page (struct mm_struct *mm);#define mk_huge_pmd (page, prot) pmd_mkhuge(mk_pmd(page, prot)) +#define mk_huge_pud (page, prot) pud_mkhuge(mk_pud(page, prot))static inline bool thp_migration_supported (void ) {
include/linux/mm.h +4 -0
@@ -26,6 +26,7 @@#include <linux/page_ref.h> #include <linux/memremap.h> #include <linux/overflow.h> +#include <linux/pagechain.h> struct mempolicy ;struct anon_vma ;
@@ -1985,6 +1986,7 @@ static inline void pgtable_init(void) {ptlock_cache_init ();pgtable_cache_init (); +pagechain_cache_init (); }static inline bool pgtable_page_ctor (struct page *page)
@@ -2101,6 +2103,8 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud)return ptl; } +#define pud_huge_pte (mm, pud) ((mm)->pud_huge_pte) +extern void __init pagecache_init (void );extern void free_area_init (unsigned long * zones_size); extern void __init free_area_init_node(int nid, unsigned long * zones_size,
include/linux/mm_types.h +1 -0
@@ -466,6 +466,7 @@ struct mm_struct {#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS pgtable_t pmd_huge_pte;/* protected by page_table_lock */ #endif +struct list_head pud_huge_pte;/* protected by page_table_lock */ #ifdef CONFIG_NUMA_BALANCING /* * numa_next_scan is the next time that the PTEs will be marked
include/linux/mmzone.h +1 -0
@@ -177,6 +177,7 @@ enum node_stat_item {NR_SHMEM_THPS ,NR_SHMEM_PMDMAPPED ,NR_ANON_THPS , +NR_ANON_THPS_PUD ,NR_UNSTABLE_NFS ,/* NFS unstable pages */ NR_VMSCAN_WRITE , NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */
include/linux/sched/coredump.h +1 -0
@@ -73,6 +73,7 @@ static inline int get_dumpable(struct mm_struct *mm)#define MMF_OOM_VICTIM 25/* mm is the oom victim */ #define MMF_OOM_REAP_QUEUED 26/* mm was queued for oom_reaper */ #define MMF_DISABLE_THP_MASK (1 << MMF_DISABLE_THP) +#define MMF_HUGE_PUD_ZERO_PAGE 26/* mm has ever used the global huge pud zero page */ #define MMF_INIT_MASK (MMF_DUMPABLE_MASK | MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK)
include/linux/vm_event_item.h +2 -0
@@ -82,6 +82,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, THP_DEFERRED_SPLIT_PAGE, THP_SPLIT_PMD,#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +THP_FAULT_ALLOC_PUD , +THP_FAULT_FALLBACK_PUD ,THP_SPLIT_PUD , #endif THP_ZERO_PAGE_ALLOC,
kernel/fork.c +5 -0
@@ -662,6 +662,10 @@ static void check_mm(struct mm_struct *mm)#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS VM_BUG_ON_MM (mm->pmd_huge_pte, mm);#endif +VM_BUG_ON_MM (!list_empty (&mm->pud_huge_pte) && + !pagechain_empty (list_first_entry (&mm->pud_huge_pte, + struct pagechain, list)), + mm); }#define allocate_mm () (kmem_cache_alloc(mm_cachep, GFP_KERNEL))
@@ -1003,6 +1007,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) && !USE_SPLIT_PMD_PTLOCKS mm->pmd_huge_pte =NULL ;#endif +INIT_LIST_HEAD (&mm->pud_huge_pte);mm_init_uprobes_state (mm);if (current->mm) {
mm/huge_memory.c +264 -3
312 lines not on this page · raw message
mm/memory.c +27 -1
@@ -3710,7 +3710,7 @@ static vm_fault_t create_huge_pud(struct vm_fault *vmf)#ifdef CONFIG_TRANSPARENT_HUGEPAGE /* No support for anonymous transparent PUD pages yet */ if (vma_is_anonymous (vmf->vma)) -return VM_FAULT_FALLBACK ; +return do_huge_pud_anonymous_page (vmf);if (vmf->vma->vm_ops->huge_fault)return vmf->vma->vm_ops->huge_fault (vmf,PE_SIZE_PUD );#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
@@ -4593,3 +4593,29 @@ void ptlock_free(struct page *page) kmem_cache_free(page_ptl_cachep, page->ptl); }#endif + +static struct kmem_cache *pagechain_cachep; + +void __init pagechain_cache_init (void ) +{ + pagechain_cachep =kmem_cache_create ("pagechain" , +sizeof (struct pagechain ),0 ,SLAB_PANIC ,NULL ); +} + +struct pagechain *pagechain_alloc (void ) +{ +struct pagechain *chain; + + chain =kmem_cache_alloc (pagechain_cachep,GFP_ATOMIC ); + +if (!chain) +return NULL ; + +pagechain_init (chain); +return chain; +} + +void pagechain_free (struct pagechain *pchain) +{ +kmem_cache_free (pagechain_cachep, pchain); +}
mm/page_alloc.c +2 -1
@@ -5157,7 +5157,8 @@ void show_free_areas(unsigned int filter, nodemask_t *nodemask)K (node_page_state (pgdat,NR_SHMEM_THPS ) *HPAGE_PMD_NR ),K (node_page_state (pgdat,NR_SHMEM_PMDMAPPED ) *HPAGE_PMD_NR ), -K (node_page_state (pgdat,NR_ANON_THPS ) *HPAGE_PMD_NR ), +K (node_page_state (pgdat,NR_ANON_THPS ) *HPAGE_PMD_NR + +node_page_state (pgdat,NR_ANON_THPS_PUD ) *HPAGE_PUD_NR ), #endifK (node_page_state (pgdat,NR_WRITEBACK_TEMP )), K(node_page_state(pgdat, NR_UNSTABLE_NFS)),
mm/pgtable-generic.c +46 -1
71 lines not on this page · raw message
mm/rmap.c +21 -7
@@ -712,6 +712,7 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address)pgd_t *pgd;p4d_t *p4d;pud_t *pud; +pud_t pude;pmd_t *pmd =NULL ;pmd_t pmde;
@@ -724,7 +725,10 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address) goto out; pud =pud_offset (p4d, address); -if (!pud_present (*pud)) + + pude = *pud; +barrier (); +if (!pud_present (pude) ||pud_trans_huge (pude)) goto out; pmd =pmd_offset (pud, address);
@@ -1121,8 +1125,12 @@ void do_page_add_anon_rmap(struct page *page,* pte lock(a spinlock) is held, which implies preemption * disabled. */ -if (compound) -__inc_node_page_state (page,NR_ANON_THPS ); +if (compound) { +if (nr ==HPAGE_PMD_NR ) +__inc_node_page_state (page,NR_ANON_THPS ); +else +__inc_node_page_state (page,NR_ANON_THPS_PUD ); + }__mod_node_page_state (page_pgdat (page),NR_ANON_MAPPED , nr); }if (unlikely (PageKsm (page)))
@@ -1160,7 +1168,10 @@ void page_add_new_anon_rmap(struct page *page,VM_BUG_ON_PAGE (!PageTransHuge (page), page);/* increment count (starts at -1) */ atomic_set (compound_mapcount_ptr (page),0 ); -__inc_node_page_state (page,NR_ANON_THPS ); +if (nr ==HPAGE_PMD_NR ) +__inc_node_page_state (page,NR_ANON_THPS ); +else +__inc_node_page_state (page,NR_ANON_THPS_PUD ); }else {/* Anon THP always mapped first with PMD */ VM_BUG_ON_PAGE (PageTransCompound (page), page);
@@ -1265,19 +1276,22 @@ static void page_remove_anon_compound_rmap(struct page *page)if (!IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE ))return ; -__dec_node_page_state (page,NR_ANON_THPS ); +if (hpage_nr_pages (page) ==HPAGE_PMD_NR ) +__dec_node_page_state (page,NR_ANON_THPS ); +else +__dec_node_page_state (page,NR_ANON_THPS_PUD );if (TestClearPageDoubleMap (page)) {/* * Subpages can be mapped with PTEs too. Check how many of * themi are still mapped. */ -for (i =0 , nr =0 ; i <HPAGE_PMD_NR ; i++) { +for (i =0 , nr =0 ; i <hpage_nr_pages (page); i++) {if (atomic_add_negative (-1 , &page[i]._mapcount)) nr++; } }else { - nr =HPAGE_PMD_NR ; + nr =hpage_nr_pages (page); }if (unlikely (PageMlocked (page)))
mm/vmstat.c +3 -0
@@ -1158,6 +1158,7 @@ const char * const vmstat_text[] = {"nr_shmem_hugepages" ,"nr_shmem_pmdmapped" ,"nr_anon_transparent_hugepages" , +"nr_anon_transparent_pud_hugepages" ,"nr_unstable" ,"nr_vmscan_write" , "nr_vmscan_immediate_reclaim",
@@ -1259,6 +1260,8 @@ const char * const vmstat_text[] = {"thp_deferred_split_page" ,"thp_split_pmd" , #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +"thp_fault_alloc_pud" , +"thp_fault_fallback_pud" ,"thp_split_pud" , #endif "thp_zero_page_alloc",
Signature · 1 line
2.20.1
RFC PATCH 10/31 mm: proc: add 1GB THP kpageflag.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Bit 27 is used to identify 1GB THP.
--- fs/proc/page.c | 2 ++ include/uapi/linux/kernel-page-flags.h | 2 ++ 2 files changed, 4 insertions(+)
fs/proc/page.c +2 -0
@@ -138,6 +138,8 @@ u64 stable_page_flags(struct page *page) u |=1 <<KPF_ZERO_PAGE ; u |=1 <<KPF_THP ; } +if (compound_order (head) ==HPAGE_PUD_ORDER ) + u |=1 <<KPF_PUD_THP ; }else if (is_zero_pfn (page_to_pfn (page))) u |=1 <<KPF_ZERO_PAGE ;
include/uapi/linux/kernel-page-flags.h +2 -0
@@ -36,5 +36,7 @@#define KPF_ZERO_PAGE 24#define KPF_IDLE 25#define KPF_PGTABLE 26 +#define KPF_PUD_THP 27 +#endif /* _UAPILINUX_KERNEL_PAGE_FLAGS_H */
Signature · 1 line
2.20.1
RFC PATCH 11/31 mm: debug: print compound page order in dump_page().
In reply to Zi YanFrom: Zi Yan <[email protected]>
Since we have more than just PMD-level THPs, printing compound page order is helpful to check the actual compound page sizes.
--- mm/debug.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
mm/debug.c +6 -2
@@ -68,8 +68,12 @@ void __dump_page(struct page *page, const char *reason)pr_warn ("page:%px count:%d mapcount:%d mapping:%px index:%#lx" , page,page_ref_count (page), mapcount, page->mapping,page_to_pgoff (page)); -if (PageCompound (page)) -pr_cont (" compound_mapcount: %d" ,compound_mapcount (page)); +if (PageCompound (page)) { +struct page *head =compound_head (page); + +pr_cont (" compound_mapcount: %d, order: %d" ,compound_mapcount (page), +compound_order (head)); + }pr_cont ("\n" );if (PageAnon (page))pr_warn ("anon " );
Signature · 1 line
2.20.1
RFC PATCH 12/31 mm: stats: Separate PMD THP and PUD THP stats.
In reply to Zi YanFrom: Zi Yan <[email protected]>
PMD THPs and PUD THPs are shown in separate stats.
--- drivers/base/node.c | 5 +++-- fs/proc/meminfo.c | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-)
drivers/base/node.c +3 -2
@@ -127,6 +127,7 @@ static ssize_t node_read_meminfo(struct device *dev,"Node %d SUnreclaim: %8lu kB\n" #ifdef CONFIG_TRANSPARENT_HUGEPAGE"Node %d AnonHugePages: %8lu kB\n" +"Node %d AnonHugePages(1GB): %8lu kB\n" "Node %d ShmemHugePages: %8lu kB\n" "Node %d ShmemPmdMapped: %8lu kB\n" #endif
@@ -150,8 +151,8 @@ static ssize_t node_read_meminfo(struct device *dev,#ifdef CONFIG_TRANSPARENT_HUGEPAGE , nid,K (node_page_state (pgdat,NR_ANON_THPS ) * -HPAGE_PMD_NR ) + -K (node_page_state (pgdat,NR_ANON_THPS_PUD ) * +HPAGE_PMD_NR ), + nid,K (node_page_state (pgdat,NR_ANON_THPS_PUD ) *HPAGE_PUD_NR ), nid,K (node_page_state (pgdat,NR_SHMEM_THPS ) * HPAGE_PMD_NR),
fs/proc/meminfo.c +2 -1
@@ -131,7 +131,8 @@ static int meminfo_proc_show(struct seq_file *m, void *v)#ifdef CONFIG_TRANSPARENT_HUGEPAGE show_val_kb (m,"AnonHugePages: " , -global_node_page_state (NR_ANON_THPS ) *HPAGE_PMD_NR + +global_node_page_state (NR_ANON_THPS ) *HPAGE_PMD_NR ); +show_val_kb (m,"AnonHugePages(1GB): " ,global_node_page_state (NR_ANON_THPS_PUD ) *HPAGE_PUD_NR );show_val_kb (m,"ShmemHugePages: " ,global_node_page_state (NR_SHMEM_THPS ) *HPAGE_PMD_NR );
Signature · 1 line
2.20.1
RFC PATCH 13/31 mm: thp: 1GB THP copy on write implementation.
In reply to Zi YanFrom: Zi Yan <[email protected]>
COW on 1GB THPs will fall back to 2MB THPs if 1GB THP is not available.
--- arch/x86/include/asm/pgalloc.h | 9 + include/linux/huge_mm.h | 5 + mm/huge_memory.c | 319 ++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 4 files changed, 331 insertions(+), 4 deletions(-)
arch/x86/include/asm/pgalloc.h +9 -0
@@ -110,6 +110,15 @@ static inline void pmd_populate(struct mm_struct *mm, pmd_t *pmd,#define pmd_pgtable (pmd) pmd_page(pmd) +static inline void pud_populate_with_pgtable (struct mm_struct *mm,pud_t *pud, +struct page *pte) +{ +unsigned long pfn =page_to_pfn (pte); + +paravirt_alloc_pmd (mm, pfn); +set_pud (pud,__pud (((pteval_t )pfn <<PAGE_SHIFT ) | _PAGE_TABLE)); +} +#if CONFIG_PGTABLE_LEVELS >2 static inline pmd_t *pmd_alloc_one (struct mm_struct *mm,unsigned long addr) {
include/linux/huge_mm.h +5 -0
@@ -19,6 +19,7 @@ extern int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD extern void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud);extern int do_huge_pud_anonymous_page (struct vm_fault *vmf); +extern int do_huge_pud_wp_page (struct vm_fault *vmf,pud_t orig_pud);#else static inline void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud) {
@@ -27,6 +28,10 @@ extern int do_huge_pud_anonymous_page(struct vm_fault *vmf) {return VM_FAULT_FALLBACK ; } +extern int do_huge_pud_wp_page (struct vm_fault *vmf,pud_t orig_pud) +{ +return VM_FAULT_FALLBACK ; +}#endif extern vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd);
mm/huge_memory.c +316 -3
350 lines not on this page · raw message
mm/memory.c +1 -1
@@ -3722,7 +3722,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud)#ifdef CONFIG_TRANSPARENT_HUGEPAGE /* No support for anonymous transparent PUD pages yet */ if (vma_is_anonymous (vmf->vma)) -return VM_FAULT_FALLBACK ; +return do_huge_pud_wp_page (vmf, orig_pud);if (vmf->vma->vm_ops->huge_fault)return vmf->vma->vm_ops->huge_fault (vmf,PE_SIZE_PUD );#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
Signature · 1 line
2.20.1
RFC PATCH 14/31 mm: thp: handling 1GB THP reference bit.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Add PUD-level TLB flush ops and teach page_vma_mapped_talk about 1GB THPs.
--- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/mm/pgtable.c | 13 +++++++++++++ include/asm-generic/pgtable.h | 14 ++++++++++++++ include/linux/mmu_notifier.h | 13 +++++++++++++ include/linux/rmap.h | 1 + mm/page_vma_mapped.c | 33 +++++++++++++++++++++++++++++---- mm/rmap.c | 12 +++++++++--- 7 files changed, 82 insertions(+), 7 deletions(-)
arch/x86/include/asm/pgtable.h +3 -0
@@ -1151,6 +1151,9 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma,extern int pmdp_clear_flush_young (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); +#define __HAVE_ARCH_PUDP_CLEAR_YOUNG_FLUSH +extern int pudp_clear_flush_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp);#define pmd_write pmd_writestatic inline int pmd_write (pmd_t pmd)
arch/x86/mm/pgtable.c +13 -0
@@ -643,6 +643,19 @@ int pmdp_clear_flush_young(struct vm_area_struct *vma, return young; } +int pudp_clear_flush_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp) +{ +int young; + +VM_BUG_ON (address & ~HPAGE_PUD_MASK ); + + young =pudp_test_and_clear_young (vma, address, pudp); +if (young) +flush_tlb_range (vma, address, address +HPAGE_PUD_SIZE ); + +return young; +}#endif /**
include/asm-generic/pgtable.h +14 -0
@@ -121,6 +121,20 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma,#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef __HAVE_ARCH_PUDP_CLEAR_YOUNG_FLUSH +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +extern int pudp_clear_flush_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp); +#else +int pudp_clear_flush_young (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp) +{ +BUILD_BUG (); +return 0 ; +} +#endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ +#endif +#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address,
include/linux/mmu_notifier.h +13 -0
@@ -353,6 +353,19 @@ static inline void mmu_notifier_range_init(struct mmu_notifier_range *range, __young; \ }) +#define pudp_clear_flush_young_notify (__vma, __address, __pudp) \ +({ \ + int __young; \ + struct vm_area_struct *___vma = __vma; \ + unsigned long ___address = __address; \ + __young = pudp_clear_flush_young(___vma, ___address, __pudp); \ + __young |= mmu_notifier_clear_flush_young(___vma->vm_mm, \ + ___address, \ + ___address + \ + PUD_SIZE); \ + __young; \ +}) +#define ptep_clear_young_notify (__vma, __address, __ptep) \ ({ \ int __young; \
include/linux/rmap.h +1 -0
@@ -206,6 +206,7 @@ struct page_vma_mapped_walk {struct page *page;struct vm_area_struct *vma;unsigned long address; +pud_t *pud;pmd_t *pmd;pte_t *pte;spinlock_t *ptl;
mm/page_vma_mapped.c +29 -4
@@ -141,9 +141,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)struct page *page = pvmw->page;pgd_t *pgd;p4d_t *p4d; -pud_t *pud; +pud_t pude;pmd_t pmde; +if (!pvmw->pte && !pvmw->pmd && pvmw->pud) +return not_found (pvmw); +/* The only possible pmd mapping has been handled on last iteration */ if (pvmw->pmd && !pvmw->pte)return not_found (pvmw);
@@ -171,10 +174,31 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) p4d =p4d_offset (pgd, pvmw->address);if (!p4d_present (*p4d))return false; - pud =pud_offset (p4d, pvmw->address); -if (!pud_present (*pud)) + pvmw->pud =pud_offset (p4d, pvmw->address); + +/* +* Make sure the pud value isn't cached in a register by the +* compiler and used as a stale value after we've observed a +* subsequent update. +*/ + pude =READ_ONCE (*pvmw->pud); +if (pud_trans_huge (pude)) { + pvmw->ptl =pud_lock (mm, pvmw->pud); +if (likely (pud_trans_huge (*pvmw->pud))) { +if (pvmw->flags &PVMW_MIGRATION ) +return not_found (pvmw); +if (pud_page (*pvmw->pud) != page) +return not_found (pvmw); +return true; + }else { +/* THP pud was split under us: handle on pmd level */ +spin_unlock (pvmw->ptl); + pvmw->ptl =NULL ; + } + }else if (!pud_present (pude))return false; - pvmw->pmd =pmd_offset (pud, pvmw->address); + + pvmw->pmd =pmd_offset (pvmw->pud, pvmw->address);/* * Make sure the pmd value isn't cached in a register by the * compiler and used as a stale value after we've observed a
@@ -210,6 +234,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) }else if (!pmd_present (pmde)) {return false; } +if (!map_pte (pvmw)) goto next_pte;while (1 ) {
mm/rmap.c +9 -3
@@ -789,9 +789,15 @@ static bool page_referenced_one(struct page *page, struct vm_area_struct *vma, referenced++; } }else if (IS_ENABLED (CONFIG_TRANSPARENT_HUGEPAGE )) { -if (pmdp_clear_flush_young_notify (vma, address, - pvmw.pmd)) - referenced++; +if (pvmw.pmd) { +if (pmdp_clear_flush_young_notify (vma, address, + pvmw.pmd)) + referenced++; + }else if (pvmw.pud) { +if (pudp_clear_flush_young_notify (vma, address, + pvmw.pud)) + referenced++; + } }else {/* unexpected pmd-mapped page? */ WARN_ON_ONCE (1 );
Signature · 1 line
2.20.1
RFC PATCH 15/31 mm: thp: add 1GB THP split_huge_pud_page() function.
In reply to Zi YanFrom: Zi Yan <[email protected]>
It mimics PMD-level THP split. In addition, to support PMD-mapped PUD THP, PMDPageInPUD() is used. For the mapcount of PMD-mapped PUD THP, sub_compound_mapcount() is used, which uses (head_page+3).compound_mapcount, since each base page's mapcount is used for PTE mapping. PagePUDDoubleMap() is used for both PUD-mapped and PMD-mapped PUD THPs.
page_xxx_rmap() functions now have an extra page order parameter to distinguish different THP sizes.
--- arch/x86/include/asm/pgtable.h | 15 + include/asm-generic/pgtable.h | 83 ++++ include/linux/huge_mm.h | 31 +- include/linux/memcontrol.h | 5 + include/linux/mm.h | 18 + include/linux/page-flags.h | 79 +++- include/linux/rmap.h | 9 +- include/linux/swap.h | 2 + include/linux/vm_event_item.h | 4 + kernel/events/uprobes.c | 4 +- mm/huge_memory.c | 695 ++++++++++++++++++++++++++++++--- mm/hugetlb.c | 4 +- mm/khugepaged.c | 4 +- mm/ksm.c | 4 +- mm/memcontrol.c | 13 + mm/memory.c | 16 +- mm/migrate.c | 8 +- mm/page_alloc.c | 18 +- mm/pgtable-generic.c | 11 + mm/rmap.c | 108 +++-- mm/swap.c | 38 ++ mm/swapfile.c | 4 +- mm/userfaultfd.c | 2 +- mm/util.c | 7 + mm/vmstat.c | 4 + 25 files changed, 1079 insertions(+), 107 deletions(-)
arch/x86/include/asm/pgtable.h +15 -0
@@ -1269,6 +1269,21 @@ static inline p4d_t *user_to_kernel_p4dp(p4d_t *p4dp) }#endif /* CONFIG_PAGE_TABLE_ISOLATION */ +#ifndef pudp_establish +#define pudp_establish pudp_establish +static inline pud_t pudp_establish (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp,pud_t pud) +{ +if (IS_ENABLED (CONFIG_SMP )) { +return xchg (pudp, pud); + }else { +pud_t old = *pudp; + *pudp = pud; +return old; + } +} +#endif +/* * clone_pgd_range(pgd_t *dst, pgd_t *src, int count); *
include/asm-generic/pgtable.h +83 -0
101 lines not on this page · raw message
include/linux/huge_mm.h +29 -2
@@ -178,17 +178,27 @@ void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address,bool freeze,struct page *page); +bool can_split_huge_pud_page (struct page *page,int *pextra_pins); +int split_huge_pud_page_to_list (struct page *page,struct list_head *list); +static inline int split_huge_pud_page (struct page *page) +{ +return split_huge_pud_page_to_list (page,NULL ); +}void __split_huge_pud (struct vm_area_struct *vma,pud_t *pud, -unsigned long address); +unsigned long address,bool freeze,struct page *page);#define split_huge_pud (__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ if (pud_trans_huge(*____pud) \ || pud_devmap(*____pud)) \ - __split_huge_pud(__vma, __pud, __address); \ + __split_huge_pud(__vma, __pud, __address, \ + false, NULL); \ } while (0) +void split_huge_pud_address (struct vm_area_struct *vma,unsigned long address, +bool freeze,struct page *page); +extern int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice); extern void vma_adjust_trans_huge(struct vm_area_struct *vma,
@@ -319,8 +329,25 @@ static inline void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,static inline void split_huge_pmd_address (struct vm_area_struct *vma,unsigned long address,bool freeze,struct page *page) {} +static inline bool +can_split_huge_pud_page (struct page *page,int *pextra_pins) +{ +BUILD_BUG (); +return false; +} +static inline int +split_huge_pud_page_to_list (struct page *page,struct list_head *list) +{ +return 0 ; +} +static inline int split_huge_pud_page (struct page *page) +{ +return 0 ; +}#define split_huge_pud (__vma, __pmd, __address) \ do { } while (0) +static inline void split_huge_pud_address (struct vm_area_struct *vma, +unsigned long address,bool freeze,struct page *page) {}static inline int hugepage_madvise (struct vm_area_struct *vma,unsigned long *vm_flags,int advice)
include/linux/memcontrol.h +5 -0
@@ -790,6 +790,7 @@ static inline void memcg_memory_event_mm(struct mm_struct *mm,#ifdef CONFIG_TRANSPARENT_HUGEPAGE void mem_cgroup_split_huge_fixup (struct page *head); +void mem_cgroup_split_huge_pud_fixup (struct page *head);#endif #else /* CONFIG_MEMCG */
@@ -1098,6 +1099,10 @@ static inline void mem_cgroup_split_huge_fixup(struct page *head) { } +static inline void mem_cgroup_split_huge_pud_fixup (struct page *head) +{ +} +static inline void count_memcg_events (struct mem_cgroup *memcg,enum vm_event_item idx,unsigned long count)
include/linux/mm.h +18 -0
@@ -652,6 +652,24 @@ static inline int compound_mapcount(struct page *page)return atomic_read (compound_mapcount_ptr (page)) +1 ; } +static inline unsigned int compound_order (struct page *page); +static inline atomic_t *sub_compound_mapcount_ptr (struct page *page,int sub_level) +{ +struct page *head =compound_head (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page); +VM_BUG_ON_PAGE (compound_order (head) !=HPAGE_PUD_ORDER , page); +VM_BUG_ON_PAGE ((page - head) %HPAGE_PMD_NR , page); +VM_BUG_ON_PAGE (sub_level !=1 , page); +return &page[2 + sub_level].compound_mapcount; +} + +/* Only works for PUD pages */ +static inline int sub_compound_mapcount (struct page *page) +{ +return atomic_read (sub_compound_mapcount_ptr (page,1 )) +1 ; +} +/* * The atomic page->_mapcount, starts from -1: so that transitions * both from it and to it can be tracked, using atomic_inc_and_test
include/linux/page-flags.h +71 -8
116 lines not on this page · raw message
include/linux/rmap.h +5 -4
@@ -99,6 +99,7 @@ enum ttu_flags {TTU_RMAP_LOCKED =0x80 ,/* do not grab rmap lock: * caller holds it */ TTU_SPLIT_FREEZE =0x100 ,/* freeze pte under splitting thp */ + TTU_SPLIT_HUGE_PUD = 0x200, /* split huge PUD if any */ };#ifdef CONFIG_MMU
@@ -171,13 +172,13 @@ struct anon_vma *page_get_anon_vma(struct page *page);*/ void page_move_anon_rmap (struct page *,struct vm_area_struct *);void page_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long ,bool ); +unsigned long ,bool ,int );void do_page_add_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long ,int ); +unsigned long ,int ,int );void page_add_new_anon_rmap (struct page *,struct vm_area_struct *, -unsigned long ,bool ); +unsigned long ,bool ,int );void page_add_file_rmap (struct page *,bool ); -void page_remove_rmap (struct page *,bool ); +void page_remove_rmap (struct page *,bool ,int );void hugepage_add_anon_rmap (struct page *,struct vm_area_struct *,unsigned long );
include/linux/swap.h +2 -0
@@ -333,6 +333,8 @@ extern void lru_cache_add_anon(struct page *page);extern void lru_cache_add_file (struct page *page);extern void lru_add_page_tail (struct page *page,struct page *page_tail,struct lruvec *lruvec,struct list_head *head); +extern void lru_add_pud_page_tail (struct page *page,struct page *page_tail, +struct lruvec *lruvec,struct list_head *head);extern void activate_page (struct page *);extern void mark_page_accessed (struct page *);extern void lru_add_drain (void );
include/linux/vm_event_item.h +4 -0
@@ -85,6 +85,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_FAULT_ALLOC_PUD ,THP_FAULT_FALLBACK_PUD ,THP_SPLIT_PUD , +THP_SPLIT_PUD_PAGE , +THP_SPLIT_PUD_PAGE_FAILED , +THP_ZERO_PUD_PAGE_ALLOC , +THP_ZERO_PUD_PAGE_ALLOC_FAILED , #endifTHP_ZERO_PAGE_ALLOC , THP_ZERO_PAGE_ALLOC_FAILED,
kernel/events/uprobes.c +2 -2
@@ -195,7 +195,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,VM_BUG_ON_PAGE (addr != pvmw.address, old_page);get_page (new_page); -page_add_new_anon_rmap (new_page, vma, addr, false); +page_add_new_anon_rmap (new_page, vma, addr, false,0 );mem_cgroup_commit_charge (new_page, memcg, false, false);lru_cache_add_active_or_unevictable (new_page, vma);
@@ -209,7 +209,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr,set_pte_at_notify (mm, addr, pvmw.pte,mk_pte (new_page, vma->vm_page_prot)); -page_remove_rmap (old_page, false); +page_remove_rmap (old_page, false,0 );if (!page_mapped (old_page))try_to_free_swap (old_page);page_vma_mapped_walk_done (&pvmw);
mm/huge_memory.c +651 -44
913 lines not on this page · raw message
mm/hugetlb.c +2 -2
@@ -3418,7 +3418,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,set_page_dirty (page);hugetlb_count_sub (pages_per_huge_page (h), mm); -page_remove_rmap (page, true); +page_remove_rmap (page, true,huge_page_order (h));spin_unlock (ptl);tlb_remove_page_size (tlb, page,huge_page_size (h));
@@ -3643,7 +3643,7 @@ static vm_fault_t hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,mmu_notifier_invalidate_range (mm, range.start, range.end);set_huge_pte_at (mm, haddr, ptep,make_huge_pte (vma, new_page,1 )); -page_remove_rmap (old_page, true); +page_remove_rmap (old_page, true,huge_page_order (h));hugepage_add_new_anon_rmap (new_page, vma, haddr);/* Make the old page be freed below */ new_page = old_page;
mm/khugepaged.c +2 -2
@@ -674,7 +674,7 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page,* superfluous. */ pte_clear (vma->vm_mm, address, _pte); -page_remove_rmap (src_page, false); +page_remove_rmap (src_page, false,0 );spin_unlock (ptl);free_page_and_swap_cache (src_page); }
@@ -1073,7 +1073,7 @@ static void collapse_huge_page(struct mm_struct *mm,spin_lock (pmd_ptl);BUG_ON (!pmd_none (*pmd)); -page_add_new_anon_rmap (new_page, vma, address, true); +page_add_new_anon_rmap (new_page, vma, address, true,HPAGE_PMD_ORDER );mem_cgroup_commit_charge (new_page, memcg, false, true);lru_cache_add_active_or_unevictable (new_page, vma);pgtable_trans_huge_deposit (mm, pmd, pgtable);
mm/ksm.c +2 -2
@@ -1154,7 +1154,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,*/ if (!is_zero_pfn (page_to_pfn (kpage))) {get_page (kpage); -page_add_anon_rmap (kpage, vma, addr, false); +page_add_anon_rmap (kpage, vma, addr, false,0 ); newpte =mk_pte (kpage, vma->vm_page_prot); }else { newpte = pte_mkspecial(pfn_pte(page_to_pfn(kpage),
@@ -1178,7 +1178,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,ptep_clear_flush (vma, addr, ptep);set_pte_at_notify (mm, addr, ptep, newpte); -page_remove_rmap (page, false); +page_remove_rmap (page, false,0 );if (!page_mapped (page))try_to_free_swap (page);put_page (page);
mm/memcontrol.c +13 -0
@@ -2678,6 +2678,19 @@ void mem_cgroup_split_huge_fixup(struct page *head) __mod_memcg_state(head->mem_cgroup, MEMCG_RSS_HUGE, -HPAGE_PMD_NR); } + +void mem_cgroup_split_huge_pud_fixup (struct page *head) +{ +int i; + +if (mem_cgroup_disabled ()) +return ; + +for (i =HPAGE_PMD_NR ; i <HPAGE_PUD_NR ; i +=HPAGE_PMD_NR ) + head[i].mem_cgroup = head->mem_cgroup; + +/*__mod_memcg_state(head->mem_cgroup, MEMCG_RSS_HUGE, -HPAGE_PUD_NR);*/ +}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #ifdef CONFIG_MEMCG_SWAP
mm/memory.c +8 -8
@@ -1088,7 +1088,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,mark_page_accessed (page); } rss[mm_counter (page)]--; -page_remove_rmap (page, false); +page_remove_rmap (page, false,0 );if (unlikely (page_mapcount (page) <0 ))print_bad_pte (vma, addr, ptent, page);if (unlikely (__tlb_remove_page (tlb, page))) {
@@ -1116,7 +1116,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,pte_clear_not_present_full (mm, addr, pte, tlb->fullmm); rss[mm_counter (page)]--; -page_remove_rmap (page, false); +page_remove_rmap (page, false,0 );put_page (page);continue ; }
@@ -2300,7 +2300,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)* thread doing COW. */ ptep_clear_flush_notify (vma, vmf->address, vmf->pte); -page_add_new_anon_rmap (new_page, vma, vmf->address, false); +page_add_new_anon_rmap (new_page, vma, vmf->address, false,0 );mem_cgroup_commit_charge (new_page, memcg, false, false);lru_cache_add_active_or_unevictable (new_page, vma);/*
@@ -2333,7 +2333,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf)* mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ -page_remove_rmap (old_page, false); +page_remove_rmap (old_page, false,0 ); }/* Free the old page.. */
@@ -2816,11 +2816,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)/* ksm created a completely new copy */ if (unlikely (page != swapcache && swapcache)) { -page_add_new_anon_rmap (page, vma, vmf->address, false); +page_add_new_anon_rmap (page, vma, vmf->address, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);lru_cache_add_active_or_unevictable (page, vma); }else { -do_page_add_anon_rmap (page, vma, vmf->address, exclusive); +do_page_add_anon_rmap (page, vma, vmf->address, exclusive,0 );mem_cgroup_commit_charge (page, memcg, true, false);activate_page (page); }
@@ -2967,7 +2967,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) }inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, vmf->address, false); +page_add_new_anon_rmap (page, vma, vmf->address, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);lru_cache_add_active_or_unevictable (page, vma); setpte:
@@ -3241,7 +3241,7 @@ vm_fault_t alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,/* copy-on-write page */ if (write && !(vma->vm_flags &VM_SHARED )) {inc_mm_counter_fast (vma->vm_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, vmf->address, false); +page_add_new_anon_rmap (page, vma, vmf->address, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);lru_cache_add_active_or_unevictable (page, vma); }else {
mm/migrate.c +4 -4
@@ -268,7 +268,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,set_pte_at (vma->vm_mm, pvmw.address, pvmw.pte, pte);if (PageAnon (new)) -page_add_anon_rmap (new, vma, pvmw.address, false); +page_add_anon_rmap (new, vma, pvmw.address, false,0 );else page_add_file_rmap (new, false); }
@@ -2067,7 +2067,7 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,page_ref_unfreeze (page,2 );mlock_migrate_page (new_page, page); -page_remove_rmap (page, true); +page_remove_rmap (page, true,HPAGE_PMD_ORDER );set_page_owner_migrate_reason (new_page,MR_NUMA_MISPLACED );spin_unlock (ptl);
@@ -2297,7 +2297,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,* drop page refcount. Page won't be freed, as we took * a reference just above. */ -page_remove_rmap (page, false); +page_remove_rmap (page, false,0 );put_page (page);if (pte_present (pte))
@@ -2688,7 +2688,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, }inc_mm_counter (mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, vma, addr, false); +page_add_new_anon_rmap (page, vma, addr, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);if (!is_zone_device_page (page))lru_cache_add_active_or_unevictable (page, vma);
mm/page_alloc.c +17 -1
@@ -626,6 +626,9 @@ void prep_compound_page(struct page *page, unsigned int order)set_compound_head (p, page); }atomic_set (compound_mapcount_ptr (page),-1 ); +if (order ==HPAGE_PUD_ORDER ) +for (i =0 ; i <HPAGE_PUD_NR ; i +=HPAGE_PMD_NR ) +atomic_set (sub_compound_mapcount_ptr (&page[i],1 ),-1 ); }#ifdef CONFIG_DEBUG_PAGEALLOC
@@ -1001,6 +1004,13 @@ static int free_tail_pages_check(struct page *head_page, struct page *page)*/ break ;default : +/* sub_compound_map_ptr store here */ +if (compound_order (head_page) ==HPAGE_PUD_ORDER && + (page - head_page) %HPAGE_PMD_NR ==3 ) { +if (unlikely (atomic_read (&page->compound_mapcount) !=-1 )) +bad_page (page,"nonzero sub_compound_mapcount" ,0 ); +break ; + }if (page->mapping !=TAIL_MAPPING ) {bad_page (page,"corrupted mapping in tail page" ,0 ); goto out;
@@ -1041,8 +1051,14 @@ static __always_inline bool free_pages_prepare(struct page *page,VM_BUG_ON_PAGE (compound &&compound_order (page) != order, page); -if (compound) +if (compound) {ClearPageDoubleMap (page); +if (order ==HPAGE_PUD_ORDER ) { +ClearPagePUDDoubleMap (page); +for (i =0 ; i <HPAGE_PUD_NR ; i +=HPAGE_PMD_NR ) +ClearPageDoubleMap (&page[i]); + } + }for (i =1 ; i < (1 << order); i++) {if (compound) bad +=free_tail_pages_check (page, page + i);
mm/pgtable-generic.c +11 -0
@@ -236,6 +236,17 @@ pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, }#endif +#ifndef __HAVE_ARCH_PUDP_INVALIDATE +pud_t pudp_invalidate (struct vm_area_struct *vma,unsigned long address, +pud_t *pudp) +{ +pud_t old =pudp_establish (vma, address, pudp,pud_mknotpresent (*pudp)); + +flush_pud_tlb_range (vma, address, address +HPAGE_PUD_SIZE ); +return old; +} +#endif +#ifndef pmdp_collapse_flushpmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp)
mm/rmap.c +83 -25
192 lines not on this page · raw message
mm/swap.c +38 -0
@@ -851,6 +851,44 @@ void lru_add_page_tail(struct page *page, struct page *page_tail,if (!PageUnevictable (page)) update_page_reclaim_stat(lruvec, file, PageActive(page_tail)); } + +/* used by __split_pud_huge_page_tail() */ +void lru_add_pud_page_tail (struct page *page,struct page *page_tail, +struct lruvec *lruvec,struct list_head *list) +{ +const int file =0 ; + +VM_BUG_ON_PAGE (!PageHead (page), page); +VM_BUG_ON_PAGE (PageLRU (page_tail), page); +VM_BUG_ON (NR_CPUS !=1 && + !spin_is_locked (&lruvec_pgdat (lruvec)->lru_lock)); + +if (!list) +SetPageLRU (page_tail); + +if (likely (PageLRU (page))) +list_add_tail (&page_tail->lru, &page->lru); +else if (list) { +/* page reclaim is reclaiming a huge page */ +get_page (page_tail); +list_add_tail (&page_tail->lru, list); + }else { +struct list_head *list_head; +/* +* Head page has not yet been counted, as an hpage, +* so we must account for each subpage individually. +* +* Use the standard add function to put page_tail on the list, +* but then correct its position so they all end up in order. +*/ +add_page_to_lru_list (page_tail, lruvec,page_lru (page_tail)); + list_head = page_tail->lru.prev; +list_move_tail (&page_tail->lru, list_head); + } + +if (!PageUnevictable (page)) +update_page_reclaim_stat (lruvec, file,PageActive (page_tail)); +}#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec,
mm/swapfile.c +2 -2
@@ -1775,10 +1775,10 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,set_pte_at (vma->vm_mm, addr, pte,pte_mkold (mk_pte (page, vma->vm_page_prot)));if (page == swapcache) { -page_add_anon_rmap (page, vma, addr, false); +page_add_anon_rmap (page, vma, addr, false,0 );mem_cgroup_commit_charge (page, memcg, true, false); }else {/* ksm created a completely new copy */ -page_add_new_anon_rmap (page, vma, addr, false); +page_add_new_anon_rmap (page, vma, addr, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);lru_cache_add_active_or_unevictable (page, vma); }
mm/userfaultfd.c +1 -1
@@ -90,7 +90,7 @@ static int mcopy_atomic_pte(struct mm_struct *dst_mm, goto out_release_uncharge_unlock;inc_mm_counter (dst_mm,MM_ANONPAGES ); -page_add_new_anon_rmap (page, dst_vma, dst_addr, false); +page_add_new_anon_rmap (page, dst_vma, dst_addr, false,0 );mem_cgroup_commit_charge (page, memcg, false, false);lru_cache_add_active_or_unevictable (page, dst_vma);
mm/util.c +7 -0
@@ -536,8 +536,15 @@ struct address_space *page_mapping_file(struct page *page)int __page_mapcount (struct page *page) {int ret; +struct page *head =compound_head (page); ret =atomic_read (&page->_mapcount) +1 ; +if (compound_order (head) ==HPAGE_PUD_ORDER ) { +struct page *sub_compound_page = head + + (((page - head) /HPAGE_PMD_NR ) *HPAGE_PMD_NR ); + + ret +=sub_compound_mapcount (sub_compound_page); + }/* * For file THP page->_mapcount contains total number of mapping * of the page: no need to look into compound_mapcount.
mm/vmstat.c +4 -0
@@ -1263,6 +1263,10 @@ const char * const vmstat_text[] = {"thp_fault_alloc_pud" ,"thp_fault_fallback_pud" ,"thp_split_pud" , +"thp_split_pud_page" , +"thp_split_pud_page_failed" , +"thp_zero_pud_page_alloc" , +"thp_zero_pud_page_alloc_failed" , #endif"thp_zero_page_alloc" , "thp_zero_page_alloc_failed",
Signature · 1 line
2.20.1
RFC PATCH 16/31 mm: thp: check compound_mapcount of PMD-mapped PUD THPs at free time.
In reply to Zi YanFrom: Zi Yan <[email protected]>
PMD mappings on a PUD THP should be zero when the page is freed.
--- mm/page_alloc.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
mm/page_alloc.c +3 -1
@@ -1007,8 +1007,10 @@ static int free_tail_pages_check(struct page *head_page, struct page *page)/* sub_compound_map_ptr store here */ if (compound_order (head_page) ==HPAGE_PUD_ORDER && (page - head_page) %HPAGE_PMD_NR ==3 ) { -if (unlikely (atomic_read (&page->compound_mapcount) !=-1 )) +if (unlikely (atomic_read (&page->compound_mapcount) !=-1 )) { +pr_err ("sub_compound_mapcount: %d\n" ,atomic_read (&page->compound_mapcount) +1 );bad_page (page,"nonzero sub_compound_mapcount" ,0 ); + }break ; }if (page->mapping !=TAIL_MAPPING ) {
Signature · 1 line
2.20.1
RFC PATCH 17/31 mm: thp: split properly PMD-mapped PUD THP to PTE-mapped PUD THP.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Page count increase needs to goto the head of the PUD page.
--- mm/huge_memory.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/huge_memory.c +3 -2
@@ -3198,7 +3198,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) {struct mm_struct *mm = vma->vm_mm; -struct page *page; +struct page *page, *head;pgtable_t pgtable;pmd_t old_pmd, _pmd;bool young, write, soft_dirty, pmd_migration = false;
@@ -3285,7 +3285,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, soft_dirty =pmd_soft_dirty (old_pmd); }VM_BUG_ON_PAGE (!page_count (page), page); -page_ref_add (page,HPAGE_PMD_NR -1 ); + head =compound_head (page); +page_ref_add (head,HPAGE_PMD_NR -1 );/* * Withdraw the table only after we mark the pmd entry invalid.
Signature · 1 line
2.20.1
RFC PATCH 18/31 mm: page_vma_walk: teach it about PMD-mapped PUD THP.
In reply to Zi YanFrom: Zi Yan <[email protected]>
We now have PMD-mapped PUD THP and PTE-mapped PUD THP, page_vma_walk should handle them properly.
--- mm/page_vma_mapped.c | 116 ++++++++++++++++++++++++++++++------------- 1 file changed, 82 insertions(+), 34 deletions(-)
mm/page_vma_mapped.c +82 -34
@@ -52,6 +52,22 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw)return true; } +static bool map_pmd (struct page_vma_mapped_walk *pvmw) +{ +pmd_t pmde; + + pvmw->pmd =pmd_offset (pvmw->pud, pvmw->address); + pmde =READ_ONCE (*pvmw->pmd); +if (pmd_trans_huge (pmde) ||is_pmd_migration_entry (pmde)) { + pvmw->ptl =pmd_lock (pvmw->vma->vm_mm, pvmw->pmd); +return true; + }else if (!pmd_present (pmde)) +return false; + + pvmw->ptl =pmd_lock (pvmw->vma->vm_mm, pvmw->pmd); +return true; +} +static inline bool pfn_in_hpage (struct page *hpage,unsigned long pfn) {unsigned long hpage_pfn =page_to_pfn (hpage);
@@ -111,6 +127,38 @@ static bool check_pte(struct page_vma_mapped_walk *pvmw)return pfn_in_hpage (pvmw->page, pfn); } +/* 0: not mapped, 1: pmd_page, 2: pmd */ +static int check_pmd (struct page_vma_mapped_walk *pvmw) +{ +unsigned long pfn; + +if (likely (pmd_trans_huge (*pvmw->pmd))) { +if (pvmw->flags &PVMW_MIGRATION ) +return 0 ; + pfn =pmd_pfn (*pvmw->pmd); +if (!pfn_in_hpage (pvmw->page, pfn)) +return 0 ; +return 1 ; + }else if (!pmd_present (*pvmw->pmd)) { +if (thp_migration_supported ()) { +if (!(pvmw->flags &PVMW_MIGRATION )) +return 0 ; +if (is_migration_entry (pmd_to_swp_entry (*pvmw->pmd))) { +swp_entry_t entry =pmd_to_swp_entry (*pvmw->pmd); + + pfn =migration_entry_to_pfn (entry); +if (!pfn_in_hpage (pvmw->page, pfn)) +return 0 ; +return 1 ; + } + } +return 0 ; + } +/* THP pmd was split under us: handle on pte level */ +spin_unlock (pvmw->ptl); + pvmw->ptl =NULL ; +return 2 ; +}/** * page_vma_mapped_walk - check if @pvmw->page is mapped in @pvmw->vma at * @pvmw->address
@@ -142,14 +190,14 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)pgd_t *pgd;p4d_t *p4d;pud_t pude; -pmd_t pmde; +int pmd_res;if (!pvmw->pte && !pvmw->pmd && pvmw->pud)return not_found (pvmw);/* The only possible pmd mapping has been handled on last iteration */ if (pvmw->pmd && !pvmw->pte) -return not_found (pvmw); + goto next_pmd;if (pvmw->pte) goto next_pte;
@@ -198,43 +246,43 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) }else if (!pud_present (pude))return false; - pvmw->pmd =pmd_offset (pvmw->pud, pvmw->address); -/* -* Make sure the pmd value isn't cached in a register by the -* compiler and used as a stale value after we've observed a -* subsequent update. -*/ - pmde =READ_ONCE (*pvmw->pmd); -if (pmd_trans_huge (pmde) ||is_pmd_migration_entry (pmde)) { - pvmw->ptl =pmd_lock (mm, pvmw->pmd); -if (likely (pmd_trans_huge (*pvmw->pmd))) { -if (pvmw->flags &PVMW_MIGRATION ) -return not_found (pvmw); -if (pmd_page (*pvmw->pmd) != page) -return not_found (pvmw); +if (!map_pmd (pvmw)) + goto next_pmd; +/* pmd locked after map_pmd */ +while (1 ) { + pmd_res =check_pmd (pvmw); +if (pmd_res ==1 )/* pmd_page */ return true; - }else if (!pmd_present (*pvmw->pmd)) { -if (thp_migration_supported ()) { -if (!(pvmw->flags &PVMW_MIGRATION )) -return not_found (pvmw); -if (is_migration_entry (pmd_to_swp_entry (*pvmw->pmd))) { -swp_entry_t entry =pmd_to_swp_entry (*pvmw->pmd); - -if (migration_entry_to_page (entry) != page) -return not_found (pvmw); -return true; +else if (pmd_res ==2 )/* pmd entry */ + goto pte_level; +next_pmd: +/* Only PMD-mapped PUD THP has next pmd */ +if (!(PageTransHuge (pvmw->page) &&compound_order (pvmw->page) ==HPAGE_PUD_ORDER )) +return not_found (pvmw); +do { + pvmw->address +=HPAGE_PMD_SIZE ; +if (pvmw->address >= pvmw->vma->vm_end || + pvmw->address >= +__vma_address (pvmw->page, pvmw->vma) + +hpage_nr_pages (pvmw->page) *PAGE_SIZE ) +return not_found (pvmw); +/* Did we cross page table boundary? */ +if (pvmw->address %PUD_SIZE ==0 ) { +if (pvmw->ptl) { +spin_unlock (pvmw->ptl); + pvmw->ptl =NULL ; } + goto restart; + }else { + pvmw->pmd++; } -return not_found (pvmw); - }else { -/* THP pmd was split under us: handle on pte level */ -spin_unlock (pvmw->ptl); - pvmw->ptl =NULL ; - } - }else if (!pmd_present (pmde)) { -return false; + }while (pmd_none (*pvmw->pmd)); + +if (!pvmw->ptl) + pvmw->ptl =pmd_lock (mm, pvmw->pmd); } +pte_level:if (!map_pte (pvmw)) goto next_pte;while (1 ) {
Signature · 1 line
2.20.1
RFC PATCH 19/31 mm: thp: 1GB THP support in try_to_unmap().
In reply to Zi YanFrom: Zi Yan <[email protected]>
Unmap different subpages in different sized THPs properly in the try_to_unmap() function.
--- mm/migrate.c | 2 +- mm/rmap.c | 140 +++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 103 insertions(+), 39 deletions(-)
mm/migrate.c +1 -1
@@ -223,7 +223,7 @@ static bool remove_migration_pte(struct page *page, struct vm_area_struct *vma,#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION /* PMD-mapped THP migration entry */ -if (!pvmw.pte) { +if (!pvmw.pte && pvmw.pmd) {VM_BUG_ON_PAGE (PageHuge (page) || !PageTransCompound (page), page);remove_migration_pmd (&pvmw, new);continue ;
mm/rmap.c +102 -38
203 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 20/31 mm: thp: split 1GB THPs at page reclaim.
In reply to Zi YanFrom: Zi Yan <[email protected]>
We cannot swap 1GB THPs, so split them before swap them out.
--- mm/swap_slots.c | 2 ++ mm/vmscan.c | 55 ++++++++++++++++++++++++++++++++++++------------- 2 files changed, 43 insertions(+), 14 deletions(-)
mm/swap_slots.c +2 -0
@@ -315,6 +315,8 @@ swp_entry_t get_swap_page(struct page *page) entry.val =0 ;if (PageTransHuge (page)) { +if (compound_order (page) ==HPAGE_PUD_ORDER ) +return entry;if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,HPAGE_PMD_NR ); goto out;
mm/vmscan.c +41 -14
80 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 21/31 mm: thp: 1GB zero page shrinker.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Remove 1GB zero page when we are under memory pressure.
--- mm/huge_memory.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
mm/huge_memory.c +31 -0
@@ -207,6 +207,32 @@ static struct shrinker huge_zero_page_shrinker = { .seeks = DEFAULT_SEEKS, }; +static unsigned long shrink_huge_pud_zero_page_count (struct shrinker *shrink, +struct shrink_control *sc) +{ +/* we can free zero page only if last reference remains */ +return atomic_read (&huge_pud_zero_refcount) ==1 ?HPAGE_PUD_NR :0 ; +} + +static unsigned long shrink_huge_pud_zero_page_scan (struct shrinker *shrink, +struct shrink_control *sc) +{ +if (atomic_cmpxchg (&huge_pud_zero_refcount,1 ,0 ) ==1 ) { +struct page *zero_page =xchg (&huge_pud_zero_page,NULL ); +BUG_ON (zero_page ==NULL ); +__free_pages (zero_page,compound_order (zero_page)); +return HPAGE_PUD_NR ; + } + +return 0 ; +} + +static struct shrinker huge_pud_zero_page_shrinker = { + .count_objects = shrink_huge_pud_zero_page_count, + .scan_objects = shrink_huge_pud_zero_page_scan, + .seeks =DEFAULT_SEEKS , +}; +#ifdef CONFIG_SYSFS static ssize_t enabled_show (struct kobject *kobj,struct kobj_attribute *attr,char *buf)
@@ -474,6 +500,9 @@ static int __init hugepage_init(void) err =register_shrinker (&huge_zero_page_shrinker);if (err) goto err_hzp_shrinker; + err =register_shrinker (&huge_pud_zero_page_shrinker); +if (err) + goto err_hpzp_shrinker; err =register_shrinker (&deferred_split_shrinker);if (err) goto err_split_shrinker;
@@ -496,6 +525,8 @@ static int __init hugepage_init(void) err_khugepaged:unregister_shrinker (&deferred_split_shrinker); err_split_shrinker: +unregister_shrinker (&huge_pud_zero_page_shrinker); +err_hpzp_shrinker:unregister_shrinker (&huge_zero_page_shrinker); err_hzp_shrinker:khugepaged_destroy ();
Signature · 1 line
2.20.1
RFC PATCH 22/31 mm: thp: 1GB THP follow_p*d_page() support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Add follow_page support for 1GB THPs.
--- include/linux/huge_mm.h | 11 +++++++ mm/gup.c | 60 ++++++++++++++++++++++++++++++++- mm/huge_memory.c | 73 ++++++++++++++++++++++++++++++++++++++++- 3 files changed, 142 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +11 -0
@@ -20,6 +20,10 @@ extern int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm,extern void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud);extern int do_huge_pud_anonymous_page (struct vm_fault *vmf);extern int do_huge_pud_wp_page (struct vm_fault *vmf,pud_t orig_pud); +extern struct page *follow_trans_huge_pud (struct vm_area_struct *vma, +unsigned long addr, +pud_t *pud, +unsigned int flags);#else static inline void huge_pud_set_accessed (struct vm_fault *vmf,pud_t orig_pud) {
@@ -32,6 +36,13 @@ extern int do_huge_pud_wp_page(struct vm_fault *vmf, pud_t orig_pud) {return VM_FAULT_FALLBACK ; } +struct page *follow_trans_huge_pud (struct vm_area_struct *vma, +unsigned long addr, +pud_t *pud, +unsigned int flags) +{ +return NULL ; +}#endif extern vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd);
mm/gup.c +59 -1
69 lines not on this page · raw message
mm/huge_memory.c +72 -1
85 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 23/31 mm: support 1GB THP pagemap support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Print page flags properly.
--- fs/proc/task_mmu.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
fs/proc/task_mmu.c +42 -0
@@ -1373,6 +1373,45 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,return err; } +static int pagemap_pud_range (pud_t *pudp,unsigned long addr,unsigned long end, +struct mm_walk *walk) +{ +struct vm_area_struct *vma = walk->vma; +struct pagemapread *pm = walk->private; +int err =0 ; +u64 flags =0 , frame =0 ; +pud_t pud = *pudp; +struct page *page =NULL ; + +if (vma->vm_flags &VM_SOFTDIRTY ) + flags |=PM_SOFT_DIRTY ; + +if (pud_present (pud)) { + page =pud_page (pud); + + flags |=PM_PRESENT ; +if (pud_soft_dirty (pud)) + flags |=PM_SOFT_DIRTY ; +if (pm->show_pfn) + frame =pud_pfn (pud) + + ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); + } + +if (page &&page_mapcount (page) ==1 ) + flags |=PM_MMAP_EXCLUSIVE ; + +for (; addr != end; addr +=PAGE_SIZE ) { +pagemap_entry_t pme =make_pme (frame, flags); + + err =add_to_pagemap (addr, &pme, pm); +if (err) +break ; +if (pm->show_pfn && (flags &PM_PRESENT )) + frame++; + } +return err; +} +#ifdef CONFIG_HUGETLB_PAGE /* This function walks within one hugetlb entry in the single call */ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask,
@@ -1479,6 +1518,9 @@ static ssize_t pagemap_read(struct file *file, char __user *buf,if (!pm.buffer) goto out_mm; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + pagemap_walk.pud_entry = pagemap_pud_range; +#endif pagemap_walk.pmd_entry = pagemap_pmd_range; pagemap_walk.pte_hole = pagemap_pte_hole;#ifdef CONFIG_HUGETLB_PAGE
Signature · 1 line
2.20.1
RFC PATCH 24/31 sysctl: add an option to only print the head page virtual address.
In reply to Zi YanFrom: Zi Yan <[email protected]>
It can help distinguish between PUD-mapped, PMD-mapped THPs, and PTE-mapped THPs.
--- fs/proc/task_mmu.c | 7 +++++-- kernel/sysctl.c | 11 +++++++++++ 2 files changed, 16 insertions(+), 2 deletions(-)
fs/proc/task_mmu.c +5 -2
@@ -27,6 +27,9 @@#define SEQ_PUT_DEC (str, val) \ seq_put_decimal_ull_width(m, str, (val) << (PAGE_SHIFT-10), 8) + +int only_print_head_pfn; +void task_mem (struct seq_file *m,struct mm_struct *mm) {unsigned long text, lib, swap, anon, file, shmem;
@@ -1308,7 +1311,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, flags |=PM_SOFT_DIRTY ;if (pm->show_pfn) frame =pmd_pfn (pmd) + - ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); + (only_print_head_pfn?0 :((addr & ~PMD_MASK ) >>PAGE_SHIFT )); }#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION else if (is_swap_pmd(pmd)) {
@@ -1394,7 +1397,7 @@ static int pagemap_pud_range(pud_t *pudp, unsigned long addr, unsigned long end, flags |=PM_SOFT_DIRTY ;if (pm->show_pfn) frame =pud_pfn (pud) + - ((addr & ~PMD_MASK ) >>PAGE_SHIFT ); + (only_print_head_pfn?0 :((addr & ~PUD_MASK ) >>PAGE_SHIFT )); }if (page &&page_mapcount (page) ==1 )
kernel/sysctl.c +11 -0
@@ -122,6 +122,8 @@ extern int vma_no_repeat_defrag;extern int num_breakout_chunks;extern int defrag_size_threshold; +extern int only_print_head_pfn; +/* Constants used for minimum and maximum */ #ifdef CONFIG_LOCKUP_DETECTOR static int sixty =60 ;
@@ -1750,6 +1752,15 @@ static struct ctl_table vm_table[] = { .proc_handler = proc_dointvec_minmax, .extra1 = &zero, }, + { + .procname = "only_print_head_pfn", + .data = &only_print_head_pfn, + .maxlen = sizeof(only_print_head_pfn), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, + }, { } };
Signature · 1 line
2.20.1
RFC PATCH 25/31 mm: thp: add a knob to enable/disable 1GB THPs.
In reply to Zi YanFrom: Zi Yan <[email protected]>
It does not affect existing 1GB THPs. It is similar to the knob for 2MB THPs.
--- include/linux/huge_mm.h | 14 ++++++++++++++ mm/huge_memory.c | 42 ++++++++++++++++++++++++++++++++++++++++- mm/memory.c | 2 +- 3 files changed, 56 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +14 -0
@@ -84,6 +84,8 @@ enum transparent_hugepage_flag {#ifdef CONFIG_DEBUG_VM TRANSPARENT_HUGEPAGE_DEBUG_COW_FLAG , #endif +TRANSPARENT_PUD_HUGEPAGE_FLAG , + TRANSPARENT_PUD_HUGEPAGE_REQ_MADV_FLAG, };struct kobject ;
@@ -146,6 +148,18 @@ static inline bool __transparent_hugepage_enabled(struct vm_area_struct *vma) }bool transparent_hugepage_enabled (struct vm_area_struct *vma); +static inline bool transparent_pud_hugepage_enabled (struct vm_area_struct *vma) +{ +if (transparent_hugepage_enabled (vma)) { +if (transparent_hugepage_flags & (1 <<TRANSPARENT_PUD_HUGEPAGE_FLAG )) +return true; +if (transparent_hugepage_flags & + (1 <<TRANSPARENT_HUGEPAGE_REQ_MADV_FLAG )) +return !!(vma->vm_flags &VM_HUGEPAGE ); + } + +return false; +}#define transparent_hugepage_use_zero_page () \ (transparent_hugepage_flags & \
mm/huge_memory.c +41 -1
69 lines not on this page · raw message
mm/memory.c +1 -1
@@ -3859,7 +3859,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, vmf.pud =pud_alloc (mm, p4d, address);if (!vmf.pud)return VM_FAULT_OOM ; -if (pud_none (*vmf.pud) &&__transparent_hugepage_enabled (vma)) { +if (pud_none (*vmf.pud) &&transparent_pud_hugepage_enabled (vma)) { ret =create_huge_pud (&vmf);if (!(ret &VM_FAULT_FALLBACK ))return ret;
Signature · 1 line
2.20.1
RFC PATCH 26/31 mm: thp: promote PTE-mapped THP to PMD-mapped THP.
In reply to Zi YanFrom: Zi Yan <[email protected]>
First promote 512 base pages to a PTE-mapped THP, then promote the PTE-mapped THP to a PMD-mapped THP.
--- include/linux/khugepaged.h | 1 + mm/filemap.c | 8 + mm/huge_memory.c | 419 +++++++++++++++++++++++++++++++++++++ mm/internal.h | 6 + mm/khugepaged.c | 2 +- 5 files changed, 435 insertions(+), 1 deletion(-)
include/linux/khugepaged.h +1 -0
@@ -55,6 +55,7 @@ static inline int khugepaged_enter(struct vm_area_struct *vma,return -ENOMEM ; return 0; } +void release_pte_pages (pte_t *pte,pte_t *_pte);#else /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int khugepaged_fork (struct mm_struct *mm,struct mm_struct *oldmm) {
mm/filemap.c +8 -0
@@ -1236,6 +1236,14 @@ static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem#endif +void __unlock_page (struct page *page) +{ +BUILD_BUG_ON (PG_waiters !=7 ); +VM_BUG_ON_PAGE (!PageLocked (page), page); +if (clear_bit_unlock_is_negative_byte (PG_locked, &page->flags)) +wake_up_page_bit (page, PG_locked); +} +/** * unlock_page - unlock a locked page * @page: the page
mm/huge_memory.c +419 -0
422 lines not on this page · raw message
mm/internal.h +6 -0
@@ -581,4 +581,10 @@ int expand_free_page(struct zone *zone, struct page *buddy_head,void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags,unsigned int alloc_flags); +void __unlock_page (struct page *page); + +int promote_huge_pmd_address (struct vm_area_struct *vma,unsigned long haddr); + +int promote_huge_page_address (struct vm_area_struct *vma,unsigned long haddr); +#endif /* __MM_INTERNAL_H */
mm/khugepaged.c +1 -1
@@ -508,7 +508,7 @@ static void release_pte_page(struct page *page)putback_lru_page (page); } -static void release_pte_pages (pte_t *pte,pte_t *_pte) +void release_pte_pages (pte_t *pte,pte_t *_pte) {while (--_pte >= pte) {pte_t pteval = *_pte;
Signature · 1 line
2.20.1
RFC PATCH 27/31 mm: thp: promote PMD-mapped PUD pages to PUD-mapped PUD pages.
In reply to Zi YanFrom: Zi Yan <[email protected]>
First promote 512 PMD-mapped THPs to a PMD-mapped PUD THP, then promote a PMD-mapped PUD THP to a PUD-mapped PUD THP.
--- arch/x86/include/asm/pgalloc.h | 2 + include/asm-generic/pgtable.h | 10 + mm/huge_memory.c | 497 ++++++++++++++++++++++++++++++++- mm/internal.h | 2 + mm/pgtable-generic.c | 20 ++ mm/rmap.c | 23 +- 6 files changed, 540 insertions(+), 14 deletions(-)
arch/x86/include/asm/pgalloc.h +2 -0
@@ -119,6 +119,8 @@ static inline void pud_populate_with_pgtable(struct mm_struct *mm, pud_t *pud,set_pud (pud,__pud (((pteval_t )pfn <<PAGE_SHIFT ) | _PAGE_TABLE)); } +#define pud_pgtable (pud) pud_page(pud) +#if CONFIG_PGTABLE_LEVELS >2 static inline pmd_t *pmd_alloc_one (struct mm_struct *mm,unsigned long addr) {
include/asm-generic/pgtable.h +10 -0
@@ -302,6 +302,8 @@ static inline void pudp_set_wrprotect(struct mm_struct *mm,#ifdef CONFIG_TRANSPARENT_HUGEPAGE extern pmd_t pmdp_collapse_flush (struct vm_area_struct *vma,unsigned long address,pmd_t *pmdp); +extern pud_t pudp_collapse_flush (struct vm_area_struct *vma, +unsigned long address,pud_t *pudp);#else static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address,
@@ -310,7 +312,15 @@ static inline pmd_t pmdp_collapse_flush(struct vm_area_struct *vma,BUILD_BUG ();return *pmdp; } +static inline pud_t pudp_collapse_flush (struct vm_area_struct *vma, +unsigned long address, +pud_t *pudp) +{ +BUILD_BUG (); +return *pudp; +}#define pmdp_collapse_flush pmdp_collapse_flush +#define pudp_collapse_flush pudp_collapse_flush#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif
mm/huge_memory.c +491 -6
536 lines not on this page · raw message
mm/internal.h +2 -0
@@ -584,7 +584,9 @@ void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,void __unlock_page (struct page *page);int promote_huge_pmd_address (struct vm_area_struct *vma,unsigned long haddr); +int promote_huge_pud_address (struct vm_area_struct *vma,unsigned long haddr);int promote_huge_page_address (struct vm_area_struct *vma,unsigned long haddr); +int promote_huge_pud_page_address (struct vm_area_struct *vma,unsigned long haddr);#endif /* __MM_INTERNAL_H */
mm/pgtable-generic.c +20 -0
@@ -266,4 +266,24 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address,return pmd; }#endif + +#ifndef pudp_collapse_flush +pud_t pudp_collapse_flush (struct vm_area_struct *vma,unsigned long address, +pud_t *pudp) +{ +/* +* pud and hugepage pte format are same. So we could +* use the same function. +*/ +pud_t pud; + +VM_BUG_ON (address & ~HPAGE_PUD_MASK ); +VM_BUG_ON (pud_trans_huge (*pudp)); + pud =pudp_huge_get_and_clear (vma->vm_mm, address, pudp); + +/* collapse entails shooting down ptes not pmd */ +flush_tlb_range (vma, address, address +HPAGE_PUD_SIZE ); +return pud; +} +#endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
mm/rmap.c +15 -8
@@ -1112,12 +1112,13 @@ void do_page_add_anon_rmap(struct page *page, {bool compound = flags &RMAP_COMPOUND ;bool first; +struct page *head =compound_head (page);if (compound) {atomic_t *mapcount;VM_BUG_ON_PAGE (!PageLocked (page), page); -VM_BUG_ON_PAGE (!PageTransHuge (page), page); -if (compound_order (page) ==HPAGE_PUD_ORDER ) { +VM_BUG_ON_PAGE (!PMDPageInPUD (page) && !PageTransHuge (page), page); +if (compound_order (head) ==HPAGE_PUD_ORDER ) {if (order ==HPAGE_PUD_ORDER ) { mapcount =compound_mapcount_ptr (page); }else if (order ==HPAGE_PMD_ORDER ) {
@@ -1125,7 +1126,7 @@ void do_page_add_anon_rmap(struct page *page, mapcount =sub_compound_mapcount_ptr (page,1 ); }else VM_BUG_ON (1 ); - }else if (compound_order (page) ==HPAGE_PMD_ORDER ) { + }else if (compound_order (head) ==HPAGE_PMD_ORDER ) { mapcount =compound_mapcount_ptr (page); }else VM_BUG_ON (1 );
@@ -1135,7 +1136,8 @@ void do_page_add_anon_rmap(struct page *page, }if (first) { -int nr = compound ?hpage_nr_pages (page) :1 ; +/*int nr = compound ? hpage_nr_pages(page) : 1;*/ +int nr =1 <<order;/* * We use the irq-unsafe __{inc|mod}_zone_page_stat because * these counters are not modified in interrupt context, and
@@ -1429,6 +1431,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,bool ret = true;struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )arg; +int order =0 ;/* munlock has nothing to gain from examining un-locked vmas */ if ((flags &TTU_MUNLOCK ) && !(vma->vm_flags &VM_LOCKED ))
@@ -1505,12 +1508,16 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,/* Unexpected PMD-mapped THP? */ -if (pvmw.pte) +if (pvmw.pte) { subpage = page -page_to_pfn (page) +pte_pfn (*pvmw.pte); -else if (!pvmw.pte && pvmw.pmd) + order =0 ; + }else if (!pvmw.pte && pvmw.pmd) { subpage = page -page_to_pfn (page) +pmd_pfn (*pvmw.pmd); -else if (!pvmw.pte && !pvmw.pmd && pvmw.pud) + order =HPAGE_PMD_ORDER ; + }else if (!pvmw.pte && !pvmw.pmd && pvmw.pud) { subpage = page -page_to_pfn (page) +pud_pfn (*pvmw.pud); + order =HPAGE_PUD_ORDER ; + }VM_BUG_ON (!subpage); address = pvmw.address;
@@ -1794,7 +1801,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,* * See Documentation/vm/mmu_notifier.rst */ -page_remove_rmap (subpage,PageHuge (page),0 ); +page_remove_rmap (subpage,PageHuge (page) || order >=HPAGE_PMD_ORDER , order);put_page (page); }
Signature · 1 line
2.20.1
RFC PATCH 28/31 mm: vmstats: add page promotion stats.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Count all four types of page promotion.
--- include/linux/vm_event_item.h | 4 ++++ mm/huge_memory.c | 8 ++++++++ mm/vmstat.c | 4 ++++ 3 files changed, 16 insertions(+)
include/linux/vm_event_item.h +4 -0
@@ -81,6 +81,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_SPLIT_PAGE_FAILED ,THP_DEFERRED_SPLIT_PAGE ,THP_SPLIT_PMD , +THP_PROMOTE_PMD , +THP_PROMOTE_PAGE , #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUDTHP_FAULT_ALLOC_PUD , THP_FAULT_FALLBACK_PUD,
@@ -89,6 +91,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,THP_SPLIT_PUD_PAGE_FAILED ,THP_ZERO_PUD_PAGE_ALLOC ,THP_ZERO_PUD_PAGE_ALLOC_FAILED , +THP_PROMOTE_PUD , +THP_PROMOTE_PUD_PAGE , #endifTHP_ZERO_PAGE_ALLOC , THP_ZERO_PAGE_ALLOC_FAILED,
mm/huge_memory.c +8 -0
@@ -4403,6 +4403,8 @@ int promote_huge_pmd_address(struct vm_area_struct *vma, unsigned long haddr) out_unlock:anon_vma_unlock_write (vma->anon_vma); out: +if (!ret) +count_vm_event (THP_PROMOTE_PMD );return ret; }
@@ -4644,6 +4646,8 @@ int promote_list_to_huge_page(struct page *head, struct list_head *list)put_anon_vma (anon_vma); } out: +if (!ret) +count_vm_event (THP_PROMOTE_PAGE );return ret; }
@@ -4842,6 +4846,8 @@ int promote_huge_pud_address(struct vm_area_struct *vma, unsigned long haddr) out_unlock:anon_vma_unlock_write (vma->anon_vma); out: +if (!ret) +count_vm_event (THP_PROMOTE_PUD );return ret; }
@@ -5169,6 +5175,8 @@ int promote_list_to_huge_pud_page(struct page *head, struct list_head *list)unlock_page (p);putback_lru_page (p); } +if (!ret) +count_vm_event (THP_PROMOTE_PUD_PAGE );return ret; }
mm/vmstat.c +4 -0
@@ -1259,6 +1259,8 @@ const char * const vmstat_text[] = {"thp_split_page_failed" ,"thp_deferred_split_page" ,"thp_split_pmd" , +"thp_promote_pmd" , +"thp_promote_page" , #ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD"thp_fault_alloc_pud" , "thp_fault_fallback_pud",
@@ -1267,6 +1269,8 @@ const char * const vmstat_text[] = {"thp_split_pud_page_failed" ,"thp_zero_pud_page_alloc" ,"thp_zero_pud_page_alloc_failed" , +"thp_promote_pud" , +"thp_promote_pud_page" , #endif"thp_zero_page_alloc" , "thp_zero_page_alloc_failed",
Signature · 1 line
2.20.1
RFC PATCH 29/31 mm: madvise: add madvise options to split PMD and PUD THPs.
In reply to Zi YanFrom: Zi Yan <[email protected]>
--- include/uapi/asm-generic/mman-common.h | 12 +++ mm/madvise.c | 106 +++++++++++++++++++++++++ 2 files changed, 118 insertions(+)
include/uapi/asm-generic/mman-common.h +12 -0
@@ -69,6 +69,18 @@#define MADV_MEMDEFRAG 20/* Worth backing with hugepages */ #define MADV_NOMEMDEFRAG 21/* Not worth backing with hugepages */ +#define MADV_SPLITHUGEPAGE 24/* Split huge page in range once */ +#define MADV_PROMOTEHUGEPAGE 25/* Promote range into huge page */ + +#define MADV_SPLITHUGEMAP 26/* Split huge page table entry in range once */ +#define MADV_PROMOTEHUGEMAP 27/* Promote range into huge page table entry */ + +#define MADV_SPLITHUGEPUDPAGE 28/* Split huge page in range once */ +#define MADV_PROMOTEHUGEPUDPAGE 29/* Promote range into huge page */ + +#define MADV_SPLITHUGEPUDMAP 30/* Split huge page table entry in range once */ +#define MADV_PROMOTEHUGEPUDMAP 31/* Promote range into huge page table entry */ +/* compatibility flags */ #define MAP_FILE 0
mm/madvise.c +106 -0
124 lines not on this page · raw message
Signature · 1 line
2.20.1
RFC PATCH 30/31 mm: mem_defrag: thp: PMD THP and PUD THP in-place promotion support.
In reply to Zi YanFrom: Zi Yan <[email protected]>
PMD THPs will get PMD page table entry promotion as well. PUD THPs only gets PUD page table entry promotion when the toggle is on, which is off by default. Since 1GB THP performs not so good due to shortage of 1GB TLB entries.
--- mm/mem_defrag.c | 79 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 73 insertions(+), 6 deletions(-)
mm/mem_defrag.c +73 -6
@@ -56,6 +56,7 @@ struct defrag_result_stats {unsigned long dst_non_lru_failed;unsigned long dst_non_moveable_failed;unsigned long not_defrag_vpn; +unsigned int aligned_max_order; };enum {
@@ -689,6 +690,10 @@ int defrag_address_range(struct mm_struct *mm, struct vm_area_struct *vma, page_size =get_contig_page_size (scan_page); +if (compound_order (compound_head (scan_page)) ==HPAGE_PUD_ORDER ) { + defrag_stats->aligned_max_order =HPAGE_PUD_ORDER ; + goto quit_defrag; + }/* PTE-mapped THP not allowed */ if ((scan_page ==compound_head (scan_page)) &&PageTransHuge (scan_page) && !PageHuge (scan_page))
@@ -714,6 +719,8 @@ int defrag_address_range(struct mm_struct *mm, struct vm_area_struct *vma,/* already in the contiguous pos */ if (page_dist == (long long )(scan_page - anchor_page)) { defrag_stats->aligned += (page_size/PAGE_SIZE ); + defrag_stats->aligned_max_order =max (defrag_stats->aligned_max_order, +compound_order (scan_page));continue ; }else {/* migrate pages according to the anchor pages in the vma. */ struct page *dest_page = anchor_page + page_dist;
@@ -901,6 +908,10 @@ int defrag_address_range(struct mm_struct *mm, struct vm_area_struct *vma, }else {/* exchange */ int err = -EBUSY ; +if (compound_order (compound_head (dest_page)) ==HPAGE_PUD_ORDER ) { + defrag_stats->aligned_max_order =HPAGE_PUD_ORDER ; + goto quit_defrag; + }/* PTE-mapped THP not allowed */ if ((dest_page ==compound_head (dest_page)) &&PageTransHuge (dest_page) && !PageHuge (dest_page))
@@ -1486,10 +1497,13 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc)up_read (&vma->vm_mm->mmap_sem); }else if (sc->action ==MEM_DEFRAG_DO_DEFRAG ) {/* go to nearest 1GB aligned address */ +unsigned long defrag_begin = *scan_address; unsigned long defrag_end = min_t(unsigned long, (*scan_address +HPAGE_PUD_SIZE ) &HPAGE_PUD_MASK , vend);int defrag_result; +int nr_fails_in_1gb_range =0 ; +int skip_promotion =0 ; anchor_node =get_anchor_page_node_from_vma (vma, *scan_address);
@@ -1583,14 +1597,47 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc)* skip the page which cannot be defragged and restart * from the next page */ -if (defrag_stats.not_defrag_vpn && - defrag_stats.not_defrag_vpn < defrag_sub_chunk_end) { +if (defrag_stats.not_defrag_vpn) {VM_BUG_ON (defrag_sub_chunk_end != defrag_end && defrag_stats.not_defrag_vpn > defrag_sub_chunk_end); - - *scan_address = defrag_stats.not_defrag_vpn; - defrag_stats.not_defrag_vpn =0 ; - goto continue_defrag; +find_anchor_pages_in_vma (mm, vma, defrag_stats.not_defrag_vpn); + + nr_fails_in_1gb_range +=1 ; +if (defrag_stats.not_defrag_vpn < defrag_sub_chunk_end) { +/* reset and continue */ + *scan_address = defrag_stats.not_defrag_vpn; + defrag_stats.not_defrag_vpn =0 ; + goto continue_defrag; + } + }else { +/* defrag works for the whole chunk, +* promote to THP in place +*/ +if (!defrag_result && +/* skip existing THPs */ + defrag_stats.aligned_max_order <HPAGE_PMD_ORDER && + !(*scan_address & (HPAGE_PMD_SIZE -1 )) && + !(defrag_sub_chunk_end & (HPAGE_PMD_SIZE -1 ))) { +int ret =0 ; +/* find a range to promote pmd */ +down_write (&mm->mmap_sem); + ret =promote_huge_page_address (vma, *scan_address); +if (!ret) { +/* +* promote to 2MB THP successful, but it is +* still PTE pointed +*/ +/* promote PTE-mapped THP to PMD-mapped */ +promote_huge_pmd_address (vma, *scan_address); + } +up_write (&mm->mmap_sem); + } +/* skip PUD pages */ +if (defrag_stats.aligned_max_order ==HPAGE_PUD_ORDER ) { + *scan_address = defrag_end; + skip_promotion =1 ; +continue ; + } }/* Done with current 2MB chunk */
@@ -1606,6 +1653,26 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc) } } +/* defrag works for the whole chunk, promote to PUD THP in place */ +if (!nr_fails_in_1gb_range && + !skip_promotion &&/* avoid existing THP */ + !(defrag_begin & (HPAGE_PUD_SIZE -1 )) && + !(defrag_end & (HPAGE_PUD_SIZE -1 ))) { +int ret =0 ; +/* find a range to promote pud */ +down_write (&mm->mmap_sem); + ret =promote_huge_pud_page_address (vma, defrag_begin); +if (!ret) { +/* +* promote to 1GB THP successful, but it is +* still PMD pointed +*/ +/* promote PMD-mapped THP to PUD-mapped */ +if (mem_defrag_promote_1gb_thp) +promote_huge_pud_address (vma, defrag_begin); + } +up_write (&mm->mmap_sem); + } } } done_one_vma:
Signature · 1 line
2.20.1
RFC PATCH 31/31 sysctl: toggle to promote PUD-mapped 1GB THP or not.
In reply to Zi YanFrom: Zi Yan <[email protected]>
Only promotion PMD THP by default.
--- kernel/sysctl.c | 11 +++++++++++ mm/mem_defrag.c | 17 +++++++++++++---- 2 files changed, 24 insertions(+), 4 deletions(-)
kernel/sysctl.c +11 -0
@@ -121,6 +121,7 @@ extern int vma_scan_threshold_type;extern int vma_no_repeat_defrag;extern int num_breakout_chunks;extern int defrag_size_threshold; +extern int mem_defrag_promote_thp;extern int only_print_head_pfn;
@@ -135,6 +136,7 @@ static int zero;static int __maybe_unused one =1 ;static int __maybe_unused two =2 ;static int __maybe_unused four =4 ; +static int __maybe_unused fifteen =15 ;static unsigned long one_ul =1 ;static int one_hundred =100 ;static int one_thousand =1000 ;
@@ -1761,6 +1763,15 @@ static struct ctl_table vm_table[] = { .extra1 = &zero, .extra2 = &one, }, + { + .procname = "mem_defrag_promote_thp", + .data = &mem_defrag_promote_thp, + .maxlen = sizeof(mem_defrag_promote_thp), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &fifteen, + }, { } };
mm/mem_defrag.c +13 -4
@@ -64,12 +64,18 @@ enum { VMA_THRESHOLD_TYPE_SIZE, }; +#define PROMOTE_PMD_MAP (0x8) +#define PROMOTE_PMD_PAGE (0x4) +#define PROMOTE_PUD_MAP (0x2) +#define PROMOTE_PUD_PAGE (0x1) +int num_breakout_chunks;int vma_scan_percentile =100 ;int vma_scan_threshold_type =VMA_THRESHOLD_TYPE_TIME ;int vma_no_repeat_defrag;int kmem_defragd_always;int defrag_size_threshold =5 ; +int mem_defrag_promote_thp = (PROMOTE_PMD_MAP |PROMOTE_PMD_PAGE );static DEFINE_SPINLOCK (kmem_defragd_mm_lock);#define MM_SLOTS_HASH_BITS 10
@@ -1613,7 +1619,8 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc)/* defrag works for the whole chunk, * promote to THP in place */ - if (!defrag_result && + if ((mem_defrag_promote_thp & PROMOTE_PMD_PAGE) && + !defrag_result && /* skip existing THPs */ defrag_stats.aligned_max_order < HPAGE_PMD_ORDER && !(*scan_address & (HPAGE_PMD_SIZE-1)) &&
@@ -1628,7 +1635,8 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc)* still PTE pointed */ /* promote PTE-mapped THP to PMD-mapped */ -promote_huge_pmd_address (vma, *scan_address); +if (mem_defrag_promote_thp &PROMOTE_PMD_MAP ) +promote_huge_pmd_address (vma, *scan_address); }up_write (&mm->mmap_sem); }
@@ -1654,7 +1662,8 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc) }/* defrag works for the whole chunk, promote to PUD THP in place */ -if (!nr_fails_in_1gb_range && +if ((mem_defrag_promote_thp &PROMOTE_PUD_PAGE ) && + !nr_fails_in_1gb_range && !skip_promotion &&/* avoid existing THP */ !(defrag_begin & (HPAGE_PUD_SIZE -1 )) && !(defrag_end & (HPAGE_PUD_SIZE -1 ))) {
@@ -1668,7 +1677,7 @@ static int kmem_defragd_scan_mm(struct defrag_scan_control *sc)* still PMD pointed */ /* promote PMD-mapped THP to PUD-mapped */ -if (mem_defrag_promote_1gb_thp) +if (mem_defrag_promote_thp &PROMOTE_PUD_MAP )promote_huge_pud_address (vma, defrag_begin); }up_write (&mm->mmap_sem);
Signature · 1 line
2.20.1
On 2/15/19 2:08 PM, Zi Yan wrote:
Thanks for working on this issue!
I have not yet had a chance to take a look at the code. However, I do have some general questions/comments on the approach.
Quoted 13 lines
Patch structure
----The patchset I developed to generate physically contiguous memory/arbitrary sized pages merely moves pages around. There are three components in this patchset:
1) a new page migration mechanism, called exchange pages, that exchanges the content of two in-use pages instead of performing two back-to-back page migration. It saves on overheads and avoids page reclaim and memory compaction in the page allocation path, although it is not strictly required if enough free memory is available in the system.
Quoted 6 lines
2) a new mechanism that utilizes both page migration and exchange pages to produce physically contiguous memory/arbitrary sized pages without allocating any new pages, unlike what khugepaged does. It works on per-VMA basis, creating physically contiguous memory out of each VMA, which is virtually contiguous. A simple range tree is used to ensure no two VMAs are overlapping with each other in the physical address space.
This appears to be a new approach to generating contiguous areas. Previous attempts had relied on finding a contiguous area that can then be used for various purposes including user mappings. Here, you take an existing mapping and make it contiguous. [RFC PATCH 04/31] mm: add mem_defrag functionality talks about creating a (VPN, PFN) anchor pair for each vma and then using this pair as the base for creating a contiguous area.
I'm curious, how 'fixed' is the anchor? As you know, there could be a non-movable page in the PFN range. As a result, you will not be able to create a contiguous area starting at that PFN. In such a case, do we try another PFN? I know this could result in much page shuffling. I'm just trying to figure out how we satisfy a user who really wants a contiguous area. Is there some method to keep trying?
My apologies if this is addressed in the code. This was just one of the first thoughts that came to mine when giving the series a quick look.
Signature · 1 line
Mike Kravetz
Zi Yan
reviewerQuoted 6 lines · Mike Kravetz
On 2/15/19 2:08 PM, Zi Yan wrote:
Thanks for working on this issue!
I have not yet had a chance to take a look at the code. However, I do have some general questions/comments on the approach.
Thanks for replying. The code is very intrusive and has a lot of hacks, so it is OK for us to discuss the general idea first. :)
Quoted 27 lines
Quoted 13 lines
Patch structure
----The patchset I developed to generate physically contiguous memory/arbitrary sized pages merely moves pages around. There are three components in this patchset:
1) a new page migration mechanism, called exchange pages, that exchanges the content of two in-use pages instead of performing two back-to-back page migration. It saves on overheads and avoids page reclaim and memory compaction in the page allocation path, although it is not strictly required if enough free memory is available in the system.
Quoted 6 lines
2) a new mechanism that utilizes both page migration and exchange pages to produce physically contiguous memory/arbitrary sized pages without allocating any new pages, unlike what khugepaged does. It works on per-VMA basis, creating physically contiguous memory out of each VMA, which is virtually contiguous. A simple range tree is used to ensure no two VMAs are overlapping with each other in the physical address space.
This appears to be a new approach to generating contiguous areas. Previous attempts had relied on finding a contiguous area that can then be used for various purposes including user mappings. Here, you take an existing mapping and make it contiguous. [RFC PATCH 04/31] mm: add
mem_defragfunctionality talks about creating a (VPN, PFN) anchor pair for each vma and then using this pair as the base for creating a contiguous area.
Quoted 6 lines
I'm curious, how 'fixed' is the anchor? As you know, there could be a non-movable page in the PFN range. As a result, you will not be able to create a contiguous area starting at that PFN. In such a case, do we try another PFN? I know this could result in much page shuffling. I'm just trying to figure out how we satisfy a user who really wants a contiguous area. Is there some method to keep trying?
Good question. The anchor is determined on a per-VMA basis, which can be changed easily, but in this patchiest, I used a very simple strategy — making all VMAs not overlapping in the physical address space to get maximum overall contiguity and not changing anchors even if non-moveable pages are encountered when generating physically contiguous pages.
Basically, first VMA1 in the virtual address space has its anchor as (VMA1_start_VPN, ZONE_start_PFN), second VMA1 has its anchor as (VMA2_start_VPN, ZONE_start_PFN + VMA1_size), and so on. This makes all VMA not overlapping in physical address space during contiguous memory generation. When there is a non-moveable page, the anchor will not be changed, because no matter whether we assign a new anchor or not, the contiguous pages stops at
the non-moveable page. If we are trying to get a new anchor, more effort is needed to avoid overlapping new anchor with existing contiguous pages. Any overlapping will nullify the existing contiguous pages.
To satisfy a user who wants a contiguous area with N pages, the minimal distance between any two non-moveable pages should be bigger than N pages in the system memory. Otherwise, nothing would work. If there is such an area (PFN1, PFN1+N) in the physical address space, you can set the anchor to (VPN_USER, PFN1) and use exchange_pages() to generate a contiguous area with N pages. Instead, alloc_contig_pages(PFN1, PFN1+N, …) could also work, but only at page allocation time. It also requires the system has N free pages when alloc_contig_pages() are migrating the pages in (PFN1, PFN1+N) away, or you need to swap pages to make the space.
Let me know if this makes sense to you.
--
Best Regards,
Yan Zi
Quoted 85 lines · Zi Yan
Quoted 6 lines · Mike Kravetz
On 2/15/19 2:08 PM, Zi Yan wrote:
Thanks for working on this issue!
I have not yet had a chance to take a look at the code. However, I do have some general questions/comments on the approach.
Thanks for replying. The code is very intrusive and has a lot of hacks, so it is OK for us to discuss the general idea first. :)
Quoted 27 lines
Quoted 13 lines
Patch structure
----The patchset I developed to generate physically contiguous memory/arbitrary sized pages merely moves pages around. There are three components in this patchset:
1) a new page migration mechanism, called exchange pages, that exchanges the content of two in-use pages instead of performing two back-to-back page migration. It saves on overheads and avoids page reclaim and memory compaction in the page allocation path, although it is not strictly required if enough free memory is available in the system.
Quoted 6 lines
2) a new mechanism that utilizes both page migration and exchange pages to produce physically contiguous memory/arbitrary sized pages without allocating any new pages, unlike what khugepaged does. It works on per-VMA basis, creating physically contiguous memory out of each VMA, which is virtually contiguous. A simple range tree is used to ensure no two VMAs are overlapping with each other in the physical address space.
This appears to be a new approach to generating contiguous areas. Previous attempts had relied on finding a contiguous area that can then be used for various purposes including user mappings. Here, you take an existing mapping and make it contiguous. [RFC PATCH 04/31] mm: add
mem_defragfunctionality talks about creating a (VPN, PFN) anchor pair for each vma and then using this pair as the base for creating a contiguous area.Quoted 6 lines
I'm curious, how 'fixed' is the anchor? As you know, there could be a non-movable page in the PFN range. As a result, you will not be able to create a contiguous area starting at that PFN. In such a case, do we try another PFN? I know this could result in much page shuffling. I'm just trying to figure out how we satisfy a user who really wants a contiguous area. Is there some method to keep trying?
Good question. The anchor is determined on a per-VMA basis, which can be changed easily,
but in this patchiest, I used a very simple strategy — making all VMAs not overlapping
in the physical address space to get maximum overall contiguity and not changing anchors
even if non-moveable pages are encountered when generating physically contiguous pages.Basically, first VMA1 in the virtual address space has its anchor as (
VMA1_start_VPN,ZONE_start_PFN),
second VMA1 has its anchor as (VMA2_start_VPN,ZONE_start_PFN+VMA1_size), and so on.
This makes all VMA not overlapping in physical address space during contiguous memory
generation. When there is a non-moveable page, the anchor will not be changed, because
no matter whether we assign a new anchor or not, the contiguous pages stops at the non-moveable page. If we are trying to get a new anchor, more effort is needed to
avoid overlapping new anchor with existing contiguous pages. Any overlapping will nullify the existing contiguous pages.To satisfy a user who wants a contiguous area with N pages, the minimal distance between
any two non-moveable pages should be bigger than N pages in the system memory. Otherwise,
nothing would work. If there is such an area (PFN1, PFN1+N) in the physical address space,
you can set the anchor to (VPN_USER, PFN1) and useexchange_pages()to generate a contiguous
area with N pages. Instead,alloc_contig_pages(PFN1, PFN1+N, …)could also work, but
only at page allocation time. It also requires the system has N free pages whenalloc_contig_pages()are migrating the pages in (PFN1, PFN1+N) away, or you need to swap
pages to make the space.
Quoted 2 lines
Let me know if this makes sense to you.
Yes, that is how I expected the implementation would work. Thank you.
Another high level question. One of the benefits of this approach is that exchanging pages does not require N free pages as you describe above. This assumes that the vma which we are trying to make contiguous is already populated. If it is not populated, then you also need to have N free pages. Correct? If this is true, then is the expected use case to first populate a vma, and then try to make contiguous? I would assume that if it is not populated and a request to make contiguous is given, we should try to allocate/populate the vma with contiguous pages at that time?
Signature · 1 line
Mike Kravetz
Zi Yan
reviewerQuoted 91 lines · Mike Kravetz
Quoted 85 lines · Zi Yan
Quoted 6 lines · Mike Kravetz
On 2/15/19 2:08 PM, Zi Yan wrote:
Thanks for working on this issue!
I have not yet had a chance to take a look at the code. However, I do have some general questions/comments on the approach.
Thanks for replying. The code is very intrusive and has a lot of hacks, so it is OK for us to discuss the general idea first. :)
Quoted 27 lines
Quoted 13 lines
Patch structure
----The patchset I developed to generate physically contiguous memory/arbitrary sized pages merely moves pages around. There are three components in this patchset:
1) a new page migration mechanism, called exchange pages, that exchanges the content of two in-use pages instead of performing two back-to-back page migration. It saves on overheads and avoids page reclaim and memory compaction in the page allocation path, although it is not strictly required if enough free memory is available in the system.
Quoted 6 lines
2) a new mechanism that utilizes both page migration and exchange pages to produce physically contiguous memory/arbitrary sized pages without allocating any new pages, unlike what khugepaged does. It works on per-VMA basis, creating physically contiguous memory out of each VMA, which is virtually contiguous. A simple range tree is used to ensure no two VMAs are overlapping with each other in the physical address space.
This appears to be a new approach to generating contiguous areas. Previous attempts had relied on finding a contiguous area that can then be used for various purposes including user mappings. Here, you take an existing mapping and make it contiguous. [RFC PATCH 04/31] mm: add
mem_defragfunctionality talks about creating a (VPN, PFN) anchor pair for each vma and then using this pair as the base for creating a contiguous area.Quoted 6 lines
I'm curious, how 'fixed' is the anchor? As you know, there could be a non-movable page in the PFN range. As a result, you will not be able to create a contiguous area starting at that PFN. In such a case, do we try another PFN? I know this could result in much page shuffling. I'm just trying to figure out how we satisfy a user who really wants a contiguous area. Is there some method to keep trying?
Good question. The anchor is determined on a per-VMA basis, which can be changed easily,
but in this patchiest, I used a very simple strategy — making all VMAs not overlapping
in the physical address space to get maximum overall contiguity and not changing anchors
even if non-moveable pages are encountered when generating physically contiguous pages.Basically, first VMA1 in the virtual address space has its anchor as (
VMA1_start_VPN,ZONE_start_PFN),
second VMA1 has its anchor as (VMA2_start_VPN,ZONE_start_PFN+VMA1_size), and so on.
This makes all VMA not overlapping in physical address space during contiguous memory
generation. When there is a non-moveable page, the anchor will not be changed, because
no matter whether we assign a new anchor or not, the contiguous pages stops at the non-moveable page. If we are trying to get a new anchor, more effort is needed to
avoid overlapping new anchor with existing contiguous pages. Any overlapping will nullify the existing contiguous pages.To satisfy a user who wants a contiguous area with N pages, the minimal distance between
any two non-moveable pages should be bigger than N pages in the system memory. Otherwise,
nothing would work. If there is such an area (PFN1, PFN1+N) in the physical address space,
you can set the anchor to (VPN_USER, PFN1) and useexchange_pages()to generate a contiguous
area with N pages. Instead,alloc_contig_pages(PFN1, PFN1+N, …)could also work, but
only at page allocation time. It also requires the system has N free pages whenalloc_contig_pages()are migrating the pages in (PFN1, PFN1+N) away, or you need to swap
pages to make the space.Quoted 2 lines
Let me know if this makes sense to you.
Yes, that is how I expected the implementation would work. Thank you.
Quoted 9 lines
Another high level question. One of the benefits of this approach is that exchanging pages does not require N free pages as you describe above. This assumes that the vma which we are trying to make contiguous is already populated. If it is not populated, then you also need to have N free pages. Correct? If this is true, then is the expected use case to first populate a vma, and then try to make contiguous? I would assume that if it is not populated and a request to make contiguous is given, we should try to allocate/populate the vma with contiguous pages at that time?
Yes, I assume the pages within the VMA are already populated but not contiguous yet.
My approach considers memory contiguity as an on-demand resource. In some phases of an application, accelerators or RDMA controllers would process/transfer data in one or more VMAs, at which time contiguous memory can help reduce address translation overheads or lift certain constraints. And different VMAs could be processed at different program phases, thus it might be hard to get contiguous memory for all these VMAs at the allocation time using alloc_contig_pages(). My approach can help get contiguous memory later, when the demand comes.
For some cases, you definitely can use alloc_contig_pages() to give users a contiguous area at page allocation time, if you know the user is going to use this area for accelerator data processing or as a RDMA buffer and the area size is fixed.
In addition, we can also use khugepaged approach, having a daemon periodically scan VMAs and use alloc_contig_pages() to convert non-contiguous pages in a VMA to contiguous pages, but it would require N free pages during the conversion.
In sum, my approach complements alloc_contig_pages() and provides more flexibility. It is not trying to replaces alloc_contig_pages().
--
Best Regards,
Yan Zi
Quoted 29 lines · Zi Yan
Quoted 9 lines · Mike Kravetz
Another high level question. One of the benefits of this approach is that exchanging pages does not require N free pages as you describe above. This assumes that the vma which we are trying to make contiguous is already populated. If it is not populated, then you also need to have N free pages. Correct? If this is true, then is the expected use case to first populate a vma, and then try to make contiguous? I would assume that if it is not populated and a request to make contiguous is given, we should try to allocate/populate the vma with contiguous pages at that time?
Yes, I assume the pages within the VMA are already populated but not contiguous yet.
My approach considers memory contiguity as an on-demand resource. In some phases of an application, accelerators or RDMA controllers would process/transfer data in one
or more VMAs, at which time contiguous memory can help reduce address translation overheads or lift certain constraints. And different VMAs could be processed at different program phases, thus it might be hard to get contiguous memory for all these VMAs at the allocation time usingalloc_contig_pages(). My approach can help get contiguous memory later, when the demand comes.For some cases, you definitely can use
alloc_contig_pages()to give users a contiguous area at page allocation time, if you know the user is going to use this
area for accelerator data processing or as a RDMA buffer and the area size is fixed.
Quoted 6 lines
In addition, we can also use khugepaged approach, having a daemon periodically scan VMAs and use
alloc_contig_pages()to convert non-contiguous pages in a VMA to contiguous pages, but it would require N free pages during the conversion.In sum, my approach complements
alloc_contig_pages()and provides more flexibility. It is not trying to replacesalloc_contig_pages().
Thank you for the explanation. That makes sense. I have mostly been thinking about contiguous memory from an allocation perspective and did not really consider other use cases.
Signature · 1 line
Mike Kravetz