thp/khugepaged improvements and CoW semantics
Subsystems (6)
v3 cover thp/khugepaged improvements and CoW semantics
The patchset adds khugepaged selftest (anon-THP only for now), expands cases khugepaged can handle and switches anon-THP copy-on-write handling to 4k.
Please review and consider applying.
v3:
- Fix handling compound pages in swap cache;
- Rework swaped vs. referenced check;
- Fix refcounting for compound pages;
- Drop Reviewed-by/Acked-by as patchset changed non-trivially;
- Typos;
v2:
- Fix race in compound page handling;
- Add one more test-case for compound page case;
- Rework LRU add cache draining;
- Typos;
Kirill A. Shutemov (8):
khugepaged: Add self test
khugepaged: Do not stop collapse if less than half PTEs are referenced
khugepaged: Drain all LRU caches before scanning pages
khugepaged: Drain LRU add pagevec after swapin
khugepaged: Allow to collapse a page shared across fork
khugepaged: Allow to collapse PTE-mapped compound pages
thp: Change CoW semantics for anon-THP
khugepaged: Introduce 'max_ptes_shared' tunable
Documentation/admin-guide/mm/transhuge.rst | 7 + include/trace/events/huge_memory.h | 3 +- mm/huge_memory.c | 249 +----- mm/khugepaged.c | 218 +++-- tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/khugepaged.c | 982 +++++++++++++++++++++ 6 files changed, 1174 insertions(+), 286 deletions(-) create mode 100644 tools/testing/selftests/vm/khugepaged.c
Signature · 1 line
2.26.0
PATCH RESEND v3 1/8 khugepaged: Add self test
The test checks if khugepaged is able to recover huge page where we expect to do so. It only covers anon-THP for now.
Currently the test shows few failures. They are going to be addressed by the following patches.
--- tools/testing/selftests/vm/Makefile | 1 + tools/testing/selftests/vm/khugepaged.c | 899 ++++++++++++++++++++++++ 2 files changed, 900 insertions(+) create mode 100644 tools/testing/selftests/vm/khugepaged.c
tools/testing/selftests/vm/Makefile +1 -0
@@ -18,6 +18,7 @@ TEST_GEN_FILES += on-fault-limit TEST_GEN_FILES += thuge-gen TEST_GEN_FILES += transhuge-stress TEST_GEN_FILES += userfaultfd +TEST_GEN_FILES += khugepaged ifneq (,$(filter $(ARCH),arm64 ia64 mips64 parisc64 ppc64 riscv64 s390x sh64 sparc64 x86_64)) TEST_GEN_FILES += va_128TBswitch
tools/testing/selftests/vm/khugepaged.c new file +899 -0
899 lines not on this page · raw message
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 1/8] khugepaged: Add self test
Quoted 10 lines · Kirill A. Shutemov
External email: Use caution opening links or attachments
The test checks if khugepaged is able to recover huge page where we expect to do so. It only covers anon-THP for now.
Currently the test shows few failures. They are going to be addressed by the following patches.
Signed-off-by Kirill A. Shutemov <[email protected]>
All tests passed with the whole series applied on v5.6 and Linus’ tree (commit: 8632e9b5).
Thanks.
—
Best Regards,
Yan Zi
Yang Shi
Acked-byRe: [PATCHv3, RESEND 1/8] khugepaged: Add self test
Quoted 8 lines · Zi Yan
Quoted 6 lines · Kirill A. Shutemov
External email: Use caution opening links or attachments
The test checks if khugepaged is able to recover huge page where we expect to do so. It only covers anon-THP for now.
Quoted 8 lines
Quoted 4 lines
Currently the test shows few failures. They are going to be addressed by the following patches.
Signed-off-by Kirill A. Shutemov <[email protected]>Reviewed-by Zi Yan <[email protected]>Tested-by Zi Yan <[email protected]>All tests passed with the whole series applied on v5.6 and Linus’ tree (commit:
8632e9b5).
I didn't run the tests by myself, I choose to trust the result from Zi Yan.
Quoted 6 lines
Thanks.
—
Best Regards,
Yan Zi
PATCH RESEND v3 2/8 khugepaged: Do not stop collapse if less than half PTEs are referenced
In reply to Kirill A. Shutemov__collapse_huge_page_swapin() checks the number of referenced PTE to decide if the memory range is hot enough to justify swapin.
We have few problems with the approach:
- It is way too late: we can do the check much earlier and safe time. khugepaged_scan_pmd() already knows if we have any pages to swap in and number of referenced page.
- It stops collapse altogether if there's not enough referenced pages, not only swappingin.
Fix it by making the right check early. We also can avoid additional page table scanning if khugepaged_scan_pmd() haven't found any swap entries.
--- mm/khugepaged.c | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-)
mm/khugepaged.c +10 -15
@@ -902,11 +902,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, .pgoff = linear_page_index(vma, address), }; -/* we only decide to swapin, if there is enough young ptes */ -if (referenced <HPAGE_PMD_NR /2 ) { -trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 ); -return false; - } vmf.pte =pte_offset_map (pmd, address);for (; vmf.address < address +HPAGE_PMD_NR *PAGE_SIZE ; vmf.pte++, vmf.address +=PAGE_SIZE ) {
@@ -946,7 +941,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,static void collapse_huge_page (struct mm_struct *mm,unsigned long address,struct page **hpage, -int node,int referenced) +int node,int referenced,int unmapped) {pmd_t *pmd, _pmd;pte_t *pte;
@@ -1003,7 +998,8 @@ static void collapse_huge_page(struct mm_struct *mm,* If it fails, we release mmap_sem and jump out_nolock. * Continuing to collapse causes inconsistency. */ -if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { +if (unmapped && !__collapse_huge_page_swapin (mm, vma, address, + pmd, referenced)) {mem_cgroup_cancel_charge (new_page, memcg, true);up_read (&mm->mmap_sem); goto out_nolock;
@@ -1214,22 +1210,21 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, mmu_notifier_test_young(vma->vm_mm, address)) referenced++; } -if (writable) { -if (referenced) { +if (!writable) { + result =SCAN_PAGE_RO ; + }else if (!referenced || (unmapped && referenced <HPAGE_PMD_NR /2 )) { + result =SCAN_LACK_REFERENCED_PAGE ; + }else { result =SCAN_SUCCEED ; ret =1 ; - }else { - result =SCAN_LACK_REFERENCED_PAGE ; - } - }else { - result =SCAN_PAGE_RO ; } out_unmap:pte_unmap_unlock (pte, ptl);if (ret) { node =khugepaged_find_target_node ();/* collapse_huge_page will return with the mmap_sem released */ -collapse_huge_page (mm, address, hpage, node, referenced); +collapse_huge_page (mm, address, hpage, node, + referenced, unmapped); } out: trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 2/8] khugepaged: Do not stop collapse if less than half PTEs are referenced
Quoted 16 lines · Kirill A. Shutemov
__collapse_huge_page_swapin()checks the number of referenced PTE to decide if the memory range is hot enough to justify swapin.We have few problems with the approach:
- It is way too late: we can do the check much earlier and safe time.
khugepaged_scan_pmd()already knows if we have any pages to swap in and number of referenced page.- It stops collapse altogether if there's not enough referenced pages, not only swappingin.
Fix it by making the right check early. We also can avoid additional page table scanning if
khugepaged_scan_pmd()haven't found any swap entries.
Quoted 5 lines
Signed-off-by Kirill A. Shutemov <[email protected]>Fixes 0db501f7a34c ("mm, thp: convert from optimistic swapin collapsing to conservative")--- mm/khugepaged.c | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-)
Just a nit below.
Quoted 39 lines
mm/khugepaged.c +3
-7@@ -902,11 +902,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, .pgoff = linear_page_index(vma, address), }; -/* we only decide to swapin, if there is enough young ptes */ -if (referenced <HPAGE_PMD_NR /2 ) { -trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,0 ); -return false; - } vmf.pte =pte_offset_map (pmd, address);for (; vmf.address < address +HPAGE_PMD_NR *PAGE_SIZE ; vmf.pte++, vmf.address +=PAGE_SIZE ) {@@ -946,7 +941,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,static void collapse_huge_page (struct mm_struct *mm,unsigned long address,struct page **hpage, -int node,int referenced) +int node,int referenced,int unmapped) {pmd_t *pmd, _pmd;pte_t *pte;@@ -1003,7 +998,8 @@ static void collapse_huge_page(struct mm_struct *mm,* If it fails, we release mmap_sem and jump out_nolock. * Continuing to collapse causes inconsistency. */ -if (!__collapse_huge_page_swapin (mm, vma, address, pmd, referenced)) { +if (unmapped && !__collapse_huge_page_swapin (mm, vma, address, + pmd, referenced)) {mem_cgroup_cancel_charge (new_page, memcg, true);up_read (&mm->mmap_sem); goto out_nolock;@@ -1214,22 +1210,21 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, mmu_notifier_test_young(vma->vm_mm, address)) referenced++;
mm/khugepaged.c static int khugepaged_scan_pmd(struct mm_struct *mm, · line 1219
Quoted 10 lines
mm/khugepaged.c +5
-2} -if (writable) { -if (referenced) { +if (!writable) { + result =SCAN_PAGE_RO ; + }else if (!referenced || (unmapped && referenced <HPAGE_PMD_NR /2 )) { + result =SCAN_LACK_REFERENCED_PAGE ; + }else { result =SCAN_SUCCEED ; ret =1 ;
Shall fix the indentation for the above two statements?
Quoted 17 lines
mm/khugepaged.c +2
-6- }else { - result =SCAN_LACK_REFERENCED_PAGE ; - } - }else { - result =SCAN_PAGE_RO ; } out_unmap:pte_unmap_unlock (pte, ptl);if (ret) { node =khugepaged_find_target_node ();/* collapse_huge_page will return with the mmap_sem released */ -collapse_huge_page (mm, address, hpage, node, referenced); +collapse_huge_page (mm, address, hpage, node, + referenced, unmapped); } out: trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
PATCH RESEND v3 3/8 khugepaged: Drain all LRU caches before scanning pages
In reply to Kirill A. ShutemovHaving a page in LRU add cache offsets page refcount and gives false-negative on PageLRU(). It reduces collapse success rate.
Drain all LRU add caches before scanning. It happens relatively rare and should not disturb the system too much.
--- mm/khugepaged.c | 2 ++ 1 file changed, 2 insertions(+)
mm/khugepaged.c +2 -0
@@ -2059,6 +2059,8 @@ static void khugepaged_do_scan(void)barrier ();/* write khugepaged_pages_to_scan to local stack */ +lru_add_drain_all (); +while (progress < pages) {if (!khugepaged_prealloc_page (&hpage, &wait))break ;
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 3/8] khugepaged: Drain all LRU caches before scanning pages
Quoted 10 lines · Kirill A. Shutemov
Having a page in LRU add cache offsets page refcount and gives false-negative on
PageLRU(). It reduces collapse success rate.Drain all LRU add caches before scanning. It happens relatively rare and should not disturb the system too much.
Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 14 lines
mm/khugepaged.c +2
-0@@ -2059,6 +2059,8 @@ static void khugepaged_do_scan(void)barrier ();/* write khugepaged_pages_to_scan to local stack */ +lru_add_drain_all (); +while (progress < pages) {if (!khugepaged_prealloc_page (&hpage, &wait))break ;
PATCH RESEND v3 4/8 khugepaged: Drain LRU add pagevec after swapin
In reply to Kirill A. Shutemov__collapse_huge_page_isolate() may fail due to extra pin in the LRU add pagevec. It's pretty common for swapin case: we swap in pages just to fail due to the extra pin.
Drain LRU add pagevec on successful swapin.
--- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/khugepaged.c +5 -0
@@ -934,6 +934,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, } vmf.pte--;pte_unmap (vmf.pte); + +/* Drain LRU add pagevec to remove extra pin on the swapped in pages */ +if (swapped_in) +lru_add_drain (); +trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,1 );return true; }
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 4/8] khugepaged: Drain LRU add pagevec after swapin
Quoted 10 lines · Kirill A. Shutemov
__collapse_huge_page_isolate()may fail due to extra pin in the LRU add pagevec. It's pretty common for swapin case: we swap in pages just to fail due to the extra pin.Drain LRU add pagevec on successful swapin.
Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+)
Quoted 17 lines
mm/khugepaged.c +5
-0@@ -934,6 +934,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm, } vmf.pte--;pte_unmap (vmf.pte); + +/* Drain LRU add pagevec to remove extra pin on the swapped in pages */ +if (swapped_in) +lru_add_drain (); +trace_mm_collapse_huge_page_swapin (mm, swapped_in, referenced,1 );return true; }
PATCH RESEND v3 5/8 khugepaged: Allow to collapse a page shared across fork
In reply to Kirill A. ShutemovThe page can be included into collapse as long as it doesn't have extra pins (from GUP or otherwise).
Logic to check the refcound is moved to a separate function. Note that the function is ready to deal with compound pages. It's preparation for the following patch.
VM_BUG_ON_PAGE() was removed from __collapse_huge_page_copy() as the invariant it checks is no longer valid: the source can be mapped multiple times now.
--- mm/khugepaged.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
mm/khugepaged.c +30 -11
@@ -529,6 +529,24 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte) } } +static bool is_refcount_suitable (struct page *page) +{ +int expected_refcount, refcount; + + refcount =page_count (page); + expected_refcount =total_mapcount (page); +if (PageSwapCache (page)) + expected_refcount +=compound_nr (page); + +if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" ); + } + +return page_count (page) == expected_refcount; +} +static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address,pte_t *pte)
@@ -581,11 +599,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* The page table that maps the page has been already unlinked +* from the page table tree and this process cannot get +* an additinal pin on the page. +* +* New pins can come later if the page is shared across fork, +* but not for the this process. It is fine. The other process +* cannot write to the page, only trigger CoW. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) {unlock_page (page); result =SCAN_PAGE_COUNT ; goto out;
@@ -672,7 +696,6 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to
@@ -1201,12 +1224,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. -*/ -if (page_count (page) !=1 +PageSwapCache (page)) { +/* Check if the page has any GUP (or other external) pins */ +if (!is_refcount_suitable (page)) { result =SCAN_PAGE_COUNT ; goto out_unmap; }
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 5/8] khugepaged: Allow to collapse a page shared across fork
Quoted 4 lines · Kirill A. Shutemov
The page can be included into collapse as long as it doesn't have extra pins (from GUP or otherwise).
Logic to check the refcound is moved to a separate function.
"refcount"
Quoted 2 lines
Note that the function is ready to deal with compound pages. It's preparation for the following patch.
Maybe:
"Added compound_nr(page) to the expected refcount, in order to handle the compound page case. This is in preparation for the following patch."
(Just to make it clear that this is not just refactoring, but also a change.)
Quoted 15 lines
VM_BUG_ON_PAGE()was removed from__collapse_huge_page_copy()as the invariant it checks is no longer valid: the source can be mapped multiple times now.Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)mm/khugepaged.c +0
-0
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte) · line 533
Quoted 5 lines
mm/khugepaged.c +2
-0@@ -529,6 +529,24 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte) } } +static bool is_refcount_suitable (struct page *page) +{
This looks nice.
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte) · line 544
Quoted 11 lines
mm/khugepaged.c +11
-0+int expected_refcount, refcount; + + refcount =page_count (page); + expected_refcount =total_mapcount (page); +if (PageSwapCache (page)) + expected_refcount +=compound_nr (page); + +if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" );
I see two issues with the pr_err() and the dump_page() call:
1. You probably want to rate limit this, otherwise you'll have a big problem if lots of pages are pinned!
2. Actually, I don't think you'd want to print anything at all here, even with rate limiting, because doing so presumes that "unexpected" means "wrong". And I think this patch doesn't expect to have GUP pins (or pin_user_pages() pins, ha), but that doesn't mean that they're wrong to have.
Quoted 13 lines
mm/khugepaged.c +5
-1+ } + +return page_count (page) == expected_refcount; +} +static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address,pte_t *pte)@@ -581,11 +599,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }/* -* cannot use mapcount: can't collapse if there's a gup pin.
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 609
Quoted 10 lines
mm/khugepaged.c +8
-2-* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* The page table that maps the page has been already unlinked +* from the page table tree and this process cannot get +* an additinal pin on the page. +* +* New pins can come later if the page is shared across fork, +* but not for the this process. It is fine. The other process
"but not from this process. The other process"
(It's very hard to figure out what "it is fine" means in this context, so better to leave it out, imho.)
Quoted 21 lines
mm/khugepaged.c +2
-3+* cannot write to the page, only trigger CoW. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) {unlock_page (page); result =SCAN_PAGE_COUNT ; goto out;@@ -672,7 +696,6 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to @@ -1201,12 +1224,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/*
mm/khugepaged.c static int khugepaged_scan_pmd(struct mm_struct *mm, · line 1231
Quoted 10 lines
mm/khugepaged.c +2
-5-* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. -*/ -if (page_count (page) !=1 +PageSwapCache (page)) { +/* Check if the page has any GUP (or other external) pins */ +if (!is_refcount_suitable (page)) { result =SCAN_PAGE_COUNT ; goto out_unmap; }
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCHv3, RESEND 5/8] khugepaged: Allow to collapse a page shared across fork
On Mon, Apr 13, 2020 at 01:48:22PM -0700, John Hubbard wrote:
[Thanks for all your suggestions and corrections]
Quoted 10 lines
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte) · line 544
Quoted 4 lines
mm/khugepaged.c +4
-0+if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" );I see two issues with the
pr_err()and thedump_page()call:1. You probably want to rate limit this, otherwise you'll have a big problem if lots of pages are pinned!
Nope. Only if kernel is buggy. See below.
Quoted 4 lines
2. Actually, I don't think you'd want to print anything at all here, even with rate limiting, because doing so presumes that "unexpected" means "wrong". And I think this patch doesn't expect to have GUP pins (or
pin_user_pages()pins, ha), but that doesn't mean that they're wrong to have.
See condition. We only do it if refcount is *below* expected refcount. It should never happen. Pinned page would have refcount above expected.
Signature · 1 line
Kirill A. Shutemov
Re: [PATCHv3, RESEND 5/8] khugepaged: Allow to collapse a page shared across fork
Quoted 17 lines · Kirill A. Shutemov
On Mon, Apr 13, 2020 at 01:48:22PM -0700, John Hubbard wrote:
[Thanks for all your suggestions and corrections]
Quoted 10 lines
mm/khugepaged.c static void release_pte_pages(pte_t *pte, pte_t *_pte) · line 544
Quoted 4 lines
mm/khugepaged.c +4
-0+if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" );I see two issues with the
pr_err()and thedump_page()call:1. You probably want to rate limit this, otherwise you'll have a big problem if lots of pages are pinned!
Nope. Only if kernel is buggy. See below.
Quoted 8 lines
Quoted 4 lines
2. Actually, I don't think you'd want to print anything at all here, even with rate limiting, because doing so presumes that "unexpected" means "wrong". And I think this patch doesn't expect to have GUP pins (or
pin_user_pages()pins, ha), but that doesn't mean that they're wrong to have.See condition. We only do it if refcount is *below* expected refcount. It should never happen. Pinned page would have refcount above expected.
Yes, you are right. I misread the condition. This actually is just right. :)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Re: [PATCHv3, RESEND 5/8] khugepaged: Allow to collapse a page shared across fork
In reply to Kirill A. ShutemovQuoted 7 lines · Kirill A. Shutemov
The page can be included into collapse as long as it doesn't have extra pins (from GUP or otherwise).
Logic to check the refcound is moved to a separate function. Note that the function is ready to deal with compound pages. It's preparation for the following patch.
Quoted 8 lines
VM_BUG_ON_PAGE()was removed from__collapse_huge_page_copy()as the invariant it checks is no longer valid: the source can be mapped multiple times now.Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
thanks,
Signature · 2 lines
John Hubbard NVIDIA
Quoted 76 lines
mm/khugepaged.c +30
-11@@ -529,6 +529,24 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte) } } +static bool is_refcount_suitable (struct page *page) +{ +int expected_refcount, refcount; + + refcount =page_count (page); + expected_refcount =total_mapcount (page); +if (PageSwapCache (page)) + expected_refcount +=compound_nr (page); + +if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" ); + } + +return page_count (page) == expected_refcount; +} +static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address,pte_t *pte)@@ -581,11 +599,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* The page table that maps the page has been already unlinked +* from the page table tree and this process cannot get +* an additinal pin on the page. +* +* New pins can come later if the page is shared across fork, +* but not for the this process. It is fine. The other process +* cannot write to the page, only trigger CoW. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) {unlock_page (page); result =SCAN_PAGE_COUNT ; goto out;@@ -672,7 +696,6 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to @@ -1201,12 +1224,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. -*/ -if (page_count (page) !=1 +PageSwapCache (page)) { +/* Check if the page has any GUP (or other external) pins */ +if (!is_refcount_suitable (page)) { result =SCAN_PAGE_COUNT ; goto out_unmap; }
Re: [PATCHv3, RESEND 5/8] khugepaged: Allow to collapse a page shared across fork
In reply to Kirill A. ShutemovQuoted 4 lines · Kirill A. Shutemov
The page can be included into collapse as long as it doesn't have extra pins (from GUP or otherwise).
Logic to check the refcound is moved to a separate function.
s/refcound/refcount
Quoted 11 lines
Note that the function is ready to deal with compound pages. It's preparation for the following patch.
VM_BUG_ON_PAGE()was removed from__collapse_huge_page_copy()as the invariant it checks is no longer valid: the source can be mapped multiple times now.Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-)
Just a minor typo problem.
Quoted 75 lines
mm/khugepaged.c +30
-11@@ -529,6 +529,24 @@ static void release_pte_pages(pte_t *pte, pte_t *_pte) } } +static bool is_refcount_suitable (struct page *page) +{ +int expected_refcount, refcount; + + refcount =page_count (page); + expected_refcount =total_mapcount (page); +if (PageSwapCache (page)) + expected_refcount +=compound_nr (page); + +if (IS_ENABLED (CONFIG_DEBUG_VM ) && expected_refcount > refcount) { +pr_err ("expected_refcount: %d, refcount: %d\n" , + expected_refcount, refcount); +dump_page (page,"Unexpected refcount" ); + } + +return page_count (page) == expected_refcount; +} +static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address,pte_t *pte)@@ -581,11 +599,17 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, }/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. +* Check if the page has any GUP (or other external) pins. +* +* The page table that maps the page has been already unlinked +* from the page table tree and this process cannot get +* an additinal pin on the page. +* +* New pins can come later if the page is shared across fork, +* but not for the this process. It is fine. The other process +* cannot write to the page, only trigger CoW. */ -if (page_count (page) !=1 +PageSwapCache (page)) { +if (!is_refcount_suitable (page)) {unlock_page (page); result =SCAN_PAGE_COUNT ; goto out;@@ -672,7 +696,6 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -VM_BUG_ON_PAGE (page_mapcount (src_page) !=1 , src_page);release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to @@ -1201,12 +1224,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/* -* cannot use mapcount: can't collapse if there's a gup pin. -* The page must only be referenced by the scanned process -* and page swap cache. -*/ -if (page_count (page) !=1 +PageSwapCache (page)) { +/* Check if the page has any GUP (or other external) pins */ +if (!is_refcount_suitable (page)) { result =SCAN_PAGE_COUNT ; goto out_unmap; }
PATCH RESEND v3 6/8 khugepaged: Allow to collapse PTE-mapped compound pages
In reply to Kirill A. ShutemovWe can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
--- mm/khugepaged.c | 99 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 65 insertions(+), 34 deletions(-)
mm/khugepaged.c +65 -34
@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_cache (page), + -compound_nr (page));unlock_page (page);putback_lru_page (page); } -static void release_pte_pages (pte_t *pte,pte_t *_pte) +static void release_pte_pages (pte_t *pte,pte_t *_pte, +struct list_head *compound_pagelist) { +struct page *page, *tmp; +while (--_pte >= pte) {pte_t pteval = *_pte; -if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval))) -release_pte_page (pte_page (pteval)); + + page =pte_page (pteval); +if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval)) && + !PageCompound (page)) +release_pte_page (page); + } + +list_for_each_entry_safe (page, tmp, compound_pagelist, lru) { +list_del (&page->lru); +release_pte_page (page); } }
@@ -549,7 +562,8 @@ static bool is_refcount_suitable(struct page *page)static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address, -pte_t *pte) +pte_t *pte, +struct list_head *compound_pagelist) {struct page *page =NULL ;pte_t *_pte;
@@ -579,13 +593,21 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ +VM_BUG_ON_PAGE (!PageAnon (page), page); +if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out; - } +struct page *p; + page =compound_head (page); -VM_BUG_ON_PAGE (!PageAnon (page), page); +/* +* Check if we have dealt with the compound page +* already +*/ +list_for_each_entry (p, compound_pagelist, lru) { +if (page == p) + goto next; + } + }/* * We can do it before isolate_lru_page because the
@@ -614,19 +636,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result =SCAN_PAGE_COUNT ; goto out; } -if (pte_write (pteval)) { - writable = true; - }else { -if (PageSwapCache (page) && - !reuse_swap_page (page,NULL )) { -unlock_page (page); - result =SCAN_SWAP_CACHE_PAGE ; - goto out; - } +if (!pte_write (pteval) &&PageSwapCache (page) && + !reuse_swap_page (page,NULL )) {/* -* Page is not in the swap cache. It can be collapsed -* into a THP. +* Page is in the swap cache and cannot be re-used. +* It cannot be collapsed into a THP. */ +unlock_page (page); + result =SCAN_SWAP_CACHE_PAGE ; + goto out; }/*
@@ -638,16 +656,23 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result =SCAN_DEL_PAGE_LRU ; goto out; } -inc_node_page_state (page, -NR_ISOLATED_ANON +page_is_file_cache (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_cache (page), +compound_nr (page));VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE (PageLRU (page), page); +if (PageCompound (page)) +list_add_tail (&page->lru, compound_pagelist); +next:/* There should be enough young pte to collapse the page */ if (pte_young (pteval) ||page_is_young (page) ||PageReferenced (page) ||mmu_notifier_test_young (vma->vm_mm, address)) referenced++; + +if (pte_write (pteval)) + writable = true; }if (likely (writable)) {if (likely (referenced)) {
@@ -661,7 +686,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } out: -release_pte_pages (pte, _pte); +release_pte_pages (pte, _pte, compound_pagelist);trace_mm_collapse_huge_page_isolate (page, none_or_zero, referenced, writable, result);return 0 ;
@@ -670,13 +695,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,static void __collapse_huge_page_copy (pte_t *pte,struct page *page,struct vm_area_struct *vma,unsigned long address, -spinlock_t *ptl) +spinlock_t *ptl, +struct list_head *compound_pagelist) { +struct page *src_page, *tmp;pte_t *_pte;for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, page++, address +=PAGE_SIZE ) {pte_t pteval = *_pte; -struct page *src_page;if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {clear_user_highpage (page, address);
@@ -696,7 +722,8 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -release_pte_page (src_page); +if (!PageCompound (src_page)) +release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats
@@ -713,6 +740,11 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page,free_page_and_swap_cache (src_page); } } + +list_for_each_entry_safe (src_page, tmp, compound_pagelist, lru) { +list_del (&src_page->lru); +release_pte_page (src_page); + } }static void khugepaged_alloc_sleep (void )
@@ -971,6 +1003,7 @@ static void collapse_huge_page(struct mm_struct *mm,struct page **hpage, int node, int referenced, int unmapped) { +LIST_HEAD (compound_pagelist );pmd_t *pmd, _pmd;pte_t *pte;pgtable_t pgtable;
@@ -1071,7 +1104,8 @@ static void collapse_huge_page(struct mm_struct *mm,mmu_notifier_invalidate_range_end (&range);spin_lock (pte_ptl); - isolated =__collapse_huge_page_isolate (vma, address, pte); + isolated =__collapse_huge_page_isolate (vma, address, pte, + &compound_pagelist);spin_unlock (pte_ptl);if (unlikely (!isolated)) {
@@ -1096,7 +1130,8 @@ static void collapse_huge_page(struct mm_struct *mm,*/ anon_vma_unlock_write (vma->anon_vma); -__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl); +__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl, + &compound_pagelist);pte_unmap (pte);__SetPageUptodate (new_page); pgtable =pmd_pgtable (_pmd);
@@ -1193,11 +1228,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ -if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out_unmap; - } + page =compound_head (page);/* * Record which node the original page is from and save this
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 6/8] khugepaged: Allow to collapse PTE-mapped compound pages
Quoted 9 lines · Kirill A. Shutemov
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/khugepaged.c | 99 ++++++++++++++++++++++++++++++++----------------- 1 file changed, 65 insertions(+), 34 deletions(-)
Quoted 219 lines
mm/khugepaged.c +65
-34@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_cache (page), + -compound_nr (page));unlock_page (page);putback_lru_page (page); } -static void release_pte_pages (pte_t *pte,pte_t *_pte) +static void release_pte_pages (pte_t *pte,pte_t *_pte, +struct list_head *compound_pagelist) { +struct page *page, *tmp; +while (--_pte >= pte) {pte_t pteval = *_pte; -if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval))) -release_pte_page (pte_page (pteval)); + + page =pte_page (pteval); +if (!pte_none (pteval) && !is_zero_pfn (pte_pfn (pteval)) && + !PageCompound (page)) +release_pte_page (page); + } + +list_for_each_entry_safe (page, tmp, compound_pagelist, lru) { +list_del (&page->lru); +release_pte_page (page); } }@@ -549,7 +562,8 @@ static bool is_refcount_suitable(struct page *page)static int __collapse_huge_page_isolate (struct vm_area_struct *vma,unsigned long address, -pte_t *pte) +pte_t *pte, +struct list_head *compound_pagelist) {struct page *page =NULL ;pte_t *_pte;@@ -579,13 +593,21 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, goto out; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ +VM_BUG_ON_PAGE (!PageAnon (page), page); +if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out; - } +struct page *p; + page =compound_head (page); -VM_BUG_ON_PAGE (!PageAnon (page), page); +/* +* Check if we have dealt with the compound page +* already +*/ +list_for_each_entry (p, compound_pagelist, lru) { +if (page == p) + goto next; + } + }/* * We can do it before isolate_lru_page because the @@ -614,19 +636,15 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result =SCAN_PAGE_COUNT ; goto out; } -if (pte_write (pteval)) { - writable = true; - }else { -if (PageSwapCache (page) && - !reuse_swap_page (page,NULL )) { -unlock_page (page); - result =SCAN_SWAP_CACHE_PAGE ; - goto out; - } +if (!pte_write (pteval) &&PageSwapCache (page) && + !reuse_swap_page (page,NULL )) {/* -* Page is not in the swap cache. It can be collapsed -* into a THP. +* Page is in the swap cache and cannot be re-used. +* It cannot be collapsed into a THP. */ +unlock_page (page); + result =SCAN_SWAP_CACHE_PAGE ; + goto out; }/* @@ -638,16 +656,23 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, result =SCAN_DEL_PAGE_LRU ; goto out; } -inc_node_page_state (page, -NR_ISOLATED_ANON +page_is_file_cache (page)); +mod_node_page_state (page_pgdat (page), +NR_ISOLATED_ANON +page_is_file_cache (page), +compound_nr (page));VM_BUG_ON_PAGE (!PageLocked (page), page);VM_BUG_ON_PAGE (PageLRU (page), page); +if (PageCompound (page)) +list_add_tail (&page->lru, compound_pagelist); +next:/* There should be enough young pte to collapse the page */ if (pte_young (pteval) ||page_is_young (page) ||PageReferenced (page) ||mmu_notifier_test_young (vma->vm_mm, address)) referenced++; + +if (pte_write (pteval)) + writable = true; }if (likely (writable)) {if (likely (referenced)) {@@ -661,7 +686,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, } out: -release_pte_pages (pte, _pte); +release_pte_pages (pte, _pte, compound_pagelist);trace_mm_collapse_huge_page_isolate (page, none_or_zero, referenced, writable, result);return 0 ;@@ -670,13 +695,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,static void __collapse_huge_page_copy (pte_t *pte,struct page *page,struct vm_area_struct *vma,unsigned long address, -spinlock_t *ptl) +spinlock_t *ptl, +struct list_head *compound_pagelist) { +struct page *src_page, *tmp;pte_t *_pte;for (_pte = pte; _pte < pte +HPAGE_PMD_NR ; _pte++, page++, address +=PAGE_SIZE ) {pte_t pteval = *_pte; -struct page *src_page;if (pte_none (pteval) ||is_zero_pfn (pte_pfn (pteval))) {clear_user_highpage (page, address);@@ -696,7 +722,8 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page, }else { src_page =pte_page (pteval);copy_user_highpage (page, src_page, address, vma); -release_pte_page (src_page); +if (!PageCompound (src_page)) +release_pte_page (src_page);/* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats @@ -713,6 +740,11 @@ static void __collapse_huge_page_copy(pte_t *pte, struct page *page,free_page_and_swap_cache (src_page); } } + +list_for_each_entry_safe (src_page, tmp, compound_pagelist, lru) { +list_del (&src_page->lru); +release_pte_page (src_page); + } }static void khugepaged_alloc_sleep (void )@@ -971,6 +1003,7 @@ static void collapse_huge_page(struct mm_struct *mm,struct page **hpage, int node, int referenced, int unmapped) { +LIST_HEAD (compound_pagelist );pmd_t *pmd, _pmd;pte_t *pte;pgtable_t pgtable;@@ -1071,7 +1104,8 @@ static void collapse_huge_page(struct mm_struct *mm,mmu_notifier_invalidate_range_end (&range);spin_lock (pte_ptl); - isolated =__collapse_huge_page_isolate (vma, address, pte); + isolated =__collapse_huge_page_isolate (vma, address, pte, + &compound_pagelist);spin_unlock (pte_ptl);if (unlikely (!isolated)) {@@ -1096,7 +1130,8 @@ static void collapse_huge_page(struct mm_struct *mm,*/ anon_vma_unlock_write (vma->anon_vma); -__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl); +__collapse_huge_page_copy (pte, new_page, vma, address, pte_ptl, + &compound_pagelist);pte_unmap (pte);__SetPageUptodate (new_page); pgtable =pmd_pgtable (_pmd);@@ -1193,11 +1228,7 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } -/* TODO: teach khugepaged to collapse THP mapped with pte */ -if (PageCompound (page)) { - result =SCAN_PAGE_COMPOUND ; - goto out_unmap; - } + page =compound_head (page);/* * Record which node the original page is from and save this
Andrew Morton
maintainerRe: [PATCHv3, RESEND 6/8] khugepaged: Allow to collapse PTE-mapped compound pages
In reply to Kirill A. ShutemovQuoted 10 lines · On Mon, 13 Apr 2020 15:52:18 +0300 "Kirill A. Shutemov"…
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
...
mm/khugepaged.c void __khugepaged_exit(struct mm_struct *mm) · line 518
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page));
I have
dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_lru(page));
here. Is there some prerequisite which I wasn't able to find?
Re: [PATCHv3, RESEND 6/8] khugepaged: Allow to collapse PTE-mapped compound pages
Quoted 19 lines · Andrew Morton
Quoted 10 lines · On Mon, 13 Apr 2020 15:52:18 +0300 "Kirill A. Shutemov"…
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
...
mm/khugepaged.c void __khugepaged_exit(struct mm_struct *mm) · line 518
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page));I have
Quoted 3 lines
dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_lru(page));here. Is there some prerequisite which I wasn't able to find?
The patchset is on top of v5.6. It has been changed since. See 9de4f22a60f7 ("mm: code cleanup for MADV_FREE").
Look like a trivial fixup is required.
Signature · 1 line
Kirill A. Shutemov
Andrew Morton
maintainerRe: [PATCHv3, RESEND 6/8] khugepaged: Allow to collapse PTE-mapped compound pages
Quoted 20 lines · On Thu, 16 Apr 2020 00:52:05 +0300 "Kirill A. Shutemov" <[email protected]>…
Quoted 19 lines · Andrew Morton
Quoted 10 lines · On Mon, 13 Apr 2020 15:52:18 +0300 "Kirill A. Shutemov"…
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
...
mm/khugepaged.c void __khugepaged_exit(struct mm_struct *mm) · line 518
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page));I have
Quoted 8 lines
Quoted 3 lines
dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_lru(page));here. Is there some prerequisite which I wasn't able to find?
The patchset is on top of v5.6. It has been changed since. See
9de4f22a60f7("mm: code cleanup forMADV_FREE").Look like a trivial fixup is required.
[7/8] makes a big mess. Can we please have a v4?
Re: [PATCHv3, RESEND 6/8] khugepaged: Allow to collapse PTE-mapped compound pages
Quoted 24 lines · Andrew Morton
Quoted 22 lines · On Thu, 16 Apr 2020 00:52:05 +0300 "Kirill A. Shutemov" <[email protected]>…
Quoted 21 lines · Andrew Morton
Quoted 10 lines · On Mon, 13 Apr 2020 15:52:18 +0300 "Kirill A. Shutemov"…
We can collapse PTE-mapped compound pages. We only need to avoid handling them more than once: lock/unlock page only once if it's present in the PMD range multiple times as it handled on compound level. The same goes for LRU isolation and putback.
...
mm/khugepaged.c void __khugepaged_exit(struct mm_struct *mm) · line 518
Quoted 4 lines
mm/khugepaged.c +0
-1@@ -515,17 +515,30 @@ void __khugepaged_exit(struct mm_struct *mm)static void release_pte_page (struct page *page) { -dec_node_page_state (page,NR_ISOLATED_ANON +page_is_file_cache (page));I have
dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_lru(page));
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
here. Is there some prerequisite which I wasn't able to find?
The patchset is on top of v5.6. It has been changed since. See
9de4f22a60f7("mm: code cleanup forMADV_FREE").Look like a trivial fixup is required.
[7/8] makes a big mess. Can we please have a v4?
Sure. Give me a day.
Signature · 1 line
Kirill A. Shutemov
PATCH RESEND v3 7/8 thp: Change CoW semantics for anon-THP
In reply to Kirill A. ShutemovCurrently we have different copy-on-write semantics for anon- and file-THP. For anon-THP we try to allocate huge page on the write fault, but on file-THP we split PMD and allocate 4k page.
Arguably, file-THP semantics is more desirable: we don't necessary want to unshare full PMD range from the parent on the first access. This is the primary reason THP is unusable for some workloads, like Redis.
The original THP refcounting didn't allow to have PTE-mapped compound pages, so we had no options, but to allocate huge page on CoW (with fallback to 512 4k pages).
The current refcounting doesn't have such limitations and we can cut a lot of complex code out of fault path.
khugepaged is now able to recover THP from such ranges if the configuration allows.
--- mm/huge_memory.c | 249 +++++------------------------------------------ 1 file changed, 25 insertions(+), 224 deletions(-)
mm/huge_memory.c +25 -224
@@ -1206,262 +1206,63 @@ void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)spin_unlock (vmf->ptl); } -static vm_fault_t do_huge_pmd_wp_page_fallback (struct vm_fault *vmf, -pmd_t orig_pmd,struct page *page) -{ -struct vm_area_struct *vma = vmf->vma; -unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mem_cgroup *memcg; -pgtable_t pgtable; -pmd_t _pmd; -int i; -vm_fault_t ret =0 ; -struct page **pages; -struct mmu_notifier_range range; - - pages =kmalloc_array (HPAGE_PMD_NR ,sizeof (struct page *), -GFP_KERNEL ); -if (unlikely (!pages)) { - ret |=VM_FAULT_OOM ; - goto out; - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - pages[i] =alloc_page_vma_node (GFP_HIGHUSER_MOVABLE , vma, - vmf->address,page_to_nid (page)); -if (unlikely (!pages[i] || -mem_cgroup_try_charge_delay (pages[i], vma->vm_mm, -GFP_KERNEL , &memcg, false))) { -if (pages[i]) -put_page (pages[i]); -while (--i >=0 ) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, - false); -put_page (pages[i]); - } -kfree (pages); - ret |=VM_FAULT_OOM ; - goto out; - } -set_page_private (pages[i], (unsigned long )memcg); - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { -copy_user_highpage (pages[i], page + i, - haddr +PAGE_SIZE * i, vma); -__SetPageUptodate (pages[i]); -cond_resched (); - } - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_free_pages; -VM_BUG_ON_PAGE (!PageHead (page), page); - -/* -* Leave pmd empty until pte is filled note we must notify here as -* concurrent CPU thread might write to new page before the call to -* mmu_notifier_invalidate_range_end() happens which can lead to a -* device seeing memory write in different order than CPU. -* -* See Documentation/vm/mmu_notifier.rst -*/ -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); - - pgtable =pgtable_trans_huge_withdraw (vma->vm_mm, vmf->pmd); -pmd_populate (vma->vm_mm, &_pmd, pgtable); - -for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) { -pte_t entry; - entry =mk_pte (pages[i], vma->vm_page_prot); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -page_add_new_anon_rmap (pages[i], vmf->vma, haddr, false); -mem_cgroup_commit_charge (pages[i], memcg, false, false); -lru_cache_add_active_or_unevictable (pages[i], vma); - vmf->pte =pte_offset_map (&_pmd, haddr); -VM_BUG_ON (!pte_none (*vmf->pte)); -set_pte_at (vma->vm_mm, haddr, vmf->pte, entry); -pte_unmap (vmf->pte); - } -kfree (pages); - -smp_wmb ();/* make pte visible before pmd */ -pmd_populate (vma->vm_mm, vmf->pmd, pgtable); -page_remove_rmap (page, true); -spin_unlock (vmf->ptl); - -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); - - ret |=VM_FAULT_WRITE ; -put_page (page); - -out: -return ret; - -out_free_pages: -spin_unlock (vmf->ptl); -mmu_notifier_invalidate_range_end (&range); -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, false); -put_page (pages[i]); - } -kfree (pages); - goto out; -} -vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd) {struct vm_area_struct *vma = vmf->vma; -struct page *page =NULL , *new_page; -struct mem_cgroup *memcg; +struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mmu_notifier_range range; -gfp_t huge_gfp;/* for allocation and charge */ -vm_fault_t ret =0 ; vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd);VM_BUG_ON_VMA (!vma->anon_vma, vma); +if (is_huge_zero_pmd (orig_pmd)) - goto alloc; + goto fallback; +spin_lock (vmf->ptl); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_unlock; + +if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl); +return 0 ; + } page =pmd_page (orig_pmd);VM_BUG_ON_PAGE (!PageCompound (page) || !PageHead (page), page); -/* -* We can only reuse the page if nobody else maps the huge page or it's -* part. -*/ + +/* Lock page for reuse_swap_page() */ if (!trylock_page (page)) {get_page (page);spin_unlock (vmf->ptl);lock_page (page);spin_lock (vmf->ptl);if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl);unlock_page (page);put_page (page); - goto out_unlock; +return 0 ; }put_page (page); } + +/* +* We can only reuse the page if nobody else maps the huge page or it's +* part. +*/ if (reuse_swap_page (page,NULL )) {pmd_t entry; entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 )) +if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); - ret |=VM_FAULT_WRITE ;unlock_page (page); - goto out_unlock; - } -unlock_page (page); -get_page (page); -spin_unlock (vmf->ptl); -alloc: -if (__transparent_hugepage_enabled (vma) && - !transparent_hugepage_debug_cow ()) { - huge_gfp =alloc_hugepage_direct_gfpmask (vma); - new_page =alloc_hugepage_vma (huge_gfp, vma, haddr,HPAGE_PMD_ORDER ); - }else - new_page =NULL ; - -if (likely (new_page)) { -prep_transhuge_page (new_page); - }else { -if (!page) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - }else { - ret =do_huge_pmd_wp_page_fallback (vmf, orig_pmd, page); -if (ret &VM_FAULT_OOM ) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - } -put_page (page); - } -count_vm_event (THP_FAULT_FALLBACK ); - goto out; - } - -if (unlikely (mem_cgroup_try_charge_delay (new_page, vma->vm_mm, - huge_gfp, &memcg, true))) { -put_page (new_page); -split_huge_pmd (vma, vmf->pmd, vmf->address); -if (page) -put_page (page); - ret |=VM_FAULT_FALLBACK ; -count_vm_event (THP_FAULT_FALLBACK ); - goto out; - } - -count_vm_event (THP_FAULT_ALLOC ); -count_memcg_events (memcg,THP_FAULT_ALLOC ,1 ); - -if (!page) -clear_huge_page (new_page, vmf->address,HPAGE_PMD_NR ); -else -copy_user_huge_page (new_page, page, vmf->address, - vma,HPAGE_PMD_NR ); -__SetPageUptodate (new_page); - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - -spin_lock (vmf->ptl); -if (page) -put_page (page); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) {spin_unlock (vmf->ptl); -mem_cgroup_cancel_charge (new_page, memcg, true); -put_page (new_page); - goto out_mn; - }else { -pmd_t entry; - entry =mk_huge_pmd (new_page, vma->vm_page_prot); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); -page_add_new_anon_rmap (new_page, vma, haddr, true); -mem_cgroup_commit_charge (new_page, memcg, false, true); -lru_cache_add_active_or_unevictable (new_page, vma); -set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry); -update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); -if (!page) { -add_mm_counter (vma->vm_mm,MM_ANONPAGES ,HPAGE_PMD_NR ); - }else { -VM_BUG_ON_PAGE (!PageHead (page), page); -page_remove_rmap (page, true); -put_page (page); - } - ret |=VM_FAULT_WRITE ; +return VM_FAULT_WRITE ; } + +unlock_page (page);spin_unlock (vmf->ptl); -out_mn: -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); -out: -return ret; -out_unlock: -spin_unlock (vmf->ptl); -return ret; +fallback: +__split_huge_pmd (vma, vmf->pmd, vmf->address, false,NULL ); +return VM_FAULT_FALLBACK ; }/*
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 7/8] thp: Change CoW semantics for anon-THP
Quoted 15 lines · Kirill A. Shutemov
Currently we have different copy-on-write semantics for anon- and file-THP. For anon-THP we try to allocate huge page on the write fault, but on file-THP we split PMD and allocate 4k page.
Arguably, file-THP semantics is more desirable: we don't necessary want to unshare full PMD range from the parent on the first access. This is the primary reason THP is unusable for some workloads, like Redis.
The original THP refcounting didn't allow to have PTE-mapped compound pages, so we had no options, but to allocate huge page on CoW (with fallback to 512 4k pages).
The current refcounting doesn't have such limitations and we can cut a lot of complex code out of fault path.
Quoted 7 lines
khugepaged is now able to recover THP from such ranges if the configuration allows.
Signed-off-by Kirill A. Shutemov <[email protected]>--- mm/huge_memory.c | 249 +++++------------------------------------------ 1 file changed, 25 insertions(+), 224 deletions(-)
The change looks reasonable and the complexity is reduced significantly. Basically I'm fine to this change although the potential increasing priority inversion might be concerned for some corner cases, however it should be unusual.
Quoted 293 lines
mm/huge_memory.c +25
-224@@ -1206,262 +1206,63 @@ void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)spin_unlock (vmf->ptl); } -static vm_fault_t do_huge_pmd_wp_page_fallback (struct vm_fault *vmf, -pmd_t orig_pmd,struct page *page) -{ -struct vm_area_struct *vma = vmf->vma; -unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mem_cgroup *memcg; -pgtable_t pgtable; -pmd_t _pmd; -int i; -vm_fault_t ret =0 ; -struct page **pages; -struct mmu_notifier_range range; - - pages =kmalloc_array (HPAGE_PMD_NR ,sizeof (struct page *), -GFP_KERNEL ); -if (unlikely (!pages)) { - ret |=VM_FAULT_OOM ; - goto out; - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - pages[i] =alloc_page_vma_node (GFP_HIGHUSER_MOVABLE , vma, - vmf->address,page_to_nid (page)); -if (unlikely (!pages[i] || -mem_cgroup_try_charge_delay (pages[i], vma->vm_mm, -GFP_KERNEL , &memcg, false))) { -if (pages[i]) -put_page (pages[i]); -while (--i >=0 ) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, - false); -put_page (pages[i]); - } -kfree (pages); - ret |=VM_FAULT_OOM ; - goto out; - } -set_page_private (pages[i], (unsigned long )memcg); - } - -for (i =0 ; i <HPAGE_PMD_NR ; i++) { -copy_user_highpage (pages[i], page + i, - haddr +PAGE_SIZE * i, vma); -__SetPageUptodate (pages[i]); -cond_resched (); - } - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_free_pages; -VM_BUG_ON_PAGE (!PageHead (page), page); - -/* -* Leave pmd empty until pte is filled note we must notify here as -* concurrent CPU thread might write to new page before the call to -* mmu_notifier_invalidate_range_end() happens which can lead to a -* device seeing memory write in different order than CPU. -* -* See Documentation/vm/mmu_notifier.rst -*/ -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); - - pgtable =pgtable_trans_huge_withdraw (vma->vm_mm, vmf->pmd); -pmd_populate (vma->vm_mm, &_pmd, pgtable); - -for (i =0 ; i <HPAGE_PMD_NR ; i++, haddr +=PAGE_SIZE ) { -pte_t entry; - entry =mk_pte (pages[i], vma->vm_page_prot); - entry =maybe_mkwrite (pte_mkdirty (entry), vma); - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -page_add_new_anon_rmap (pages[i], vmf->vma, haddr, false); -mem_cgroup_commit_charge (pages[i], memcg, false, false); -lru_cache_add_active_or_unevictable (pages[i], vma); - vmf->pte =pte_offset_map (&_pmd, haddr); -VM_BUG_ON (!pte_none (*vmf->pte)); -set_pte_at (vma->vm_mm, haddr, vmf->pte, entry); -pte_unmap (vmf->pte); - } -kfree (pages); - -smp_wmb ();/* make pte visible before pmd */ -pmd_populate (vma->vm_mm, vmf->pmd, pgtable); -page_remove_rmap (page, true); -spin_unlock (vmf->ptl); - -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); - - ret |=VM_FAULT_WRITE ; -put_page (page); - -out: -return ret; - -out_free_pages: -spin_unlock (vmf->ptl); -mmu_notifier_invalidate_range_end (&range); -for (i =0 ; i <HPAGE_PMD_NR ; i++) { - memcg = (void *)page_private (pages[i]); -set_page_private (pages[i],0 ); -mem_cgroup_cancel_charge (pages[i], memcg, false); -put_page (pages[i]); - } -kfree (pages); - goto out; -} -vm_fault_t do_huge_pmd_wp_page (struct vm_fault *vmf,pmd_t orig_pmd) {struct vm_area_struct *vma = vmf->vma; -struct page *page =NULL , *new_page; -struct mem_cgroup *memcg; +struct page *page;unsigned long haddr = vmf->address &HPAGE_PMD_MASK ; -struct mmu_notifier_range range; -gfp_t huge_gfp;/* for allocation and charge */ -vm_fault_t ret =0 ; vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd);VM_BUG_ON_VMA (!vma->anon_vma, vma); +if (is_huge_zero_pmd (orig_pmd)) - goto alloc; + goto fallback; +spin_lock (vmf->ptl); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) - goto out_unlock; + +if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl); +return 0 ; + } page =pmd_page (orig_pmd);VM_BUG_ON_PAGE (!PageCompound (page) || !PageHead (page), page); -/* -* We can only reuse the page if nobody else maps the huge page or it's -* part. -*/ + +/* Lock page for reuse_swap_page() */ if (!trylock_page (page)) {get_page (page);spin_unlock (vmf->ptl);lock_page (page);spin_lock (vmf->ptl);if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) { +spin_unlock (vmf->ptl);unlock_page (page);put_page (page); - goto out_unlock; +return 0 ; }put_page (page); } + +/* +* We can only reuse the page if nobody else maps the huge page or it's +* part. +*/ if (reuse_swap_page (page,NULL )) {pmd_t entry; entry =pmd_mkyoung (orig_pmd); entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 )) +if (pmdp_set_access_flags (vma, haddr, vmf->pmd, entry,1 ))update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); - ret |=VM_FAULT_WRITE ;unlock_page (page); - goto out_unlock; - } -unlock_page (page); -get_page (page); -spin_unlock (vmf->ptl); -alloc: -if (__transparent_hugepage_enabled (vma) && - !transparent_hugepage_debug_cow ()) { - huge_gfp =alloc_hugepage_direct_gfpmask (vma); - new_page =alloc_hugepage_vma (huge_gfp, vma, haddr,HPAGE_PMD_ORDER ); - }else - new_page =NULL ; - -if (likely (new_page)) { -prep_transhuge_page (new_page); - }else { -if (!page) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - }else { - ret =do_huge_pmd_wp_page_fallback (vmf, orig_pmd, page); -if (ret &VM_FAULT_OOM ) { -split_huge_pmd (vma, vmf->pmd, vmf->address); - ret |=VM_FAULT_FALLBACK ; - } -put_page (page); - } -count_vm_event (THP_FAULT_FALLBACK ); - goto out; - } - -if (unlikely (mem_cgroup_try_charge_delay (new_page, vma->vm_mm, - huge_gfp, &memcg, true))) { -put_page (new_page); -split_huge_pmd (vma, vmf->pmd, vmf->address); -if (page) -put_page (page); - ret |=VM_FAULT_FALLBACK ; -count_vm_event (THP_FAULT_FALLBACK ); - goto out; - } - -count_vm_event (THP_FAULT_ALLOC ); -count_memcg_events (memcg,THP_FAULT_ALLOC ,1 ); - -if (!page) -clear_huge_page (new_page, vmf->address,HPAGE_PMD_NR ); -else -copy_user_huge_page (new_page, page, vmf->address, - vma,HPAGE_PMD_NR ); -__SetPageUptodate (new_page); - -mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, vma->vm_mm, - haddr, haddr +HPAGE_PMD_SIZE ); -mmu_notifier_invalidate_range_start (&range); - -spin_lock (vmf->ptl); -if (page) -put_page (page); -if (unlikely (!pmd_same (*vmf->pmd, orig_pmd))) {spin_unlock (vmf->ptl); -mem_cgroup_cancel_charge (new_page, memcg, true); -put_page (new_page); - goto out_mn; - }else { -pmd_t entry; - entry =mk_huge_pmd (new_page, vma->vm_page_prot); - entry =maybe_pmd_mkwrite (pmd_mkdirty (entry), vma); -pmdp_huge_clear_flush_notify (vma, haddr, vmf->pmd); -page_add_new_anon_rmap (new_page, vma, haddr, true); -mem_cgroup_commit_charge (new_page, memcg, false, true); -lru_cache_add_active_or_unevictable (new_page, vma); -set_pmd_at (vma->vm_mm, haddr, vmf->pmd, entry); -update_mmu_cache_pmd (vma, vmf->address, vmf->pmd); -if (!page) { -add_mm_counter (vma->vm_mm,MM_ANONPAGES ,HPAGE_PMD_NR ); - }else { -VM_BUG_ON_PAGE (!PageHead (page), page); -page_remove_rmap (page, true); -put_page (page); - } - ret |=VM_FAULT_WRITE ; +return VM_FAULT_WRITE ; } + +unlock_page (page);spin_unlock (vmf->ptl); -out_mn: -/* -* No need to double call mmu_notifier->invalidate_range() callback as -* the above pmdp_huge_clear_flush_notify() did already call it. -*/ -mmu_notifier_invalidate_range_only_end (&range); -out: -return ret; -out_unlock: -spin_unlock (vmf->ptl); -return ret; +fallback: +__split_huge_pmd (vma, vmf->pmd, vmf->address, false,NULL ); +return VM_FAULT_FALLBACK ; }/*
PATCH RESEND v3 8/8 khugepaged: Introduce 'max_ptes_shared' tunable
In reply to Kirill A. Shutemov'max_ptes_shared' specifies how many pages can be shared across multiple processes. Exceeding the number would block the collapse::
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared
A higher value may increase memory footprint for some workloads.
By default, at least half of pages has to be not shared.
--- Documentation/admin-guide/mm/transhuge.rst | 7 ++ include/trace/events/huge_memory.h | 3 +- mm/khugepaged.c | 52 ++++++++++++-- tools/testing/selftests/vm/khugepaged.c | 83 ++++++++++++++++++++++ 4 files changed, 140 insertions(+), 5 deletions(-)
Documentation/admin-guide/mm/transhuge.rst +7 -0
@@ -220,6 +220,13 @@ memory. A lower value can prevent THPs from being collapsed, resulting fewer pages being collapsed into THPs, and lower memory access performance. +``max_ptes_shared`` specifies how many pages can be shared across multiple +processes. Exceeding the number would block the collapse:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared + +A higher value may increase memory footprint for some workloads. + Boot parameter ==============
include/trace/events/huge_memory.h +2 -1
@@ -12,6 +12,8 @@ EM( SCAN_SUCCEED, "succeeded") \ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ + EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ + EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ EM( SCAN_PAGE_RO, "no_writable_page") \EM (SCAN_LACK_REFERENCED_PAGE ,"lack_referenced_page" ) \
@@ -30,7 +32,6 @@ EM( SCAN_DEL_PAGE_LRU, "could_not_delete_page_from_lru")\ EM( SCAN_ALLOC_HUGE_PAGE_FAIL, "alloc_huge_page_failed") \ EM( SCAN_CGROUP_CHARGE_FAIL, "ccgroup_charge_failed") \ - EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ EM( SCAN_TRUNCATED, "truncated") \EMe (SCAN_PAGE_HAS_PRIVATE ,"page_has_private" ) \
mm/khugepaged.c +48 -4
114 lines not on this page · raw message
tools/testing/selftests/vm/khugepaged.c +83 -0
125 lines not on this page · raw message
Signature · 1 line
2.26.0
Re: [PATCHv3, RESEND 8/8] khugepaged: Introduce 'max_ptes_shared' tunable
Quoted 10 lines · Kirill A. Shutemov
'
max_ptes_shared' specifies how many pages can be shared across multiple processes. Exceeding the number would block the collapse::
/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_sharedA higher value may increase memory footprint for some workloads.
By default, at least half of pages has to be not shared.
Signed-off-by Kirill A. Shutemov <[email protected]>
One nit below.
Quoted 138 lines
--- Documentation/admin-guide/mm/transhuge.rst | 7 ++ include/trace/events/huge_memory.h | 3 +- mm/khugepaged.c | 52 ++++++++++++-- tools/testing/selftests/vm/khugepaged.c | 83 ++++++++++++++++++++++ 4 files changed, 140 insertions(+), 5 deletions(-)Documentation/admin-guide/mm/transhuge.rst +7
-0@@ -220,6 +220,13 @@ memory. A lower value can prevent THPs from being collapsed, resulting fewer pages being collapsed into THPs, and lower memory access performance. +``max_ptes_shared`` specifies how many pages can be shared across multiple +processes. Exceeding the number would block the collapse:: + + /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_shared + +A higher value may increase memory footprint for some workloads. + Boot parameter ==============include/trace/events/huge_memory.h +2
-1@@ -12,6 +12,8 @@ EM( SCAN_SUCCEED, "succeeded") \ EM( SCAN_PMD_NULL, "pmd_null") \ EM( SCAN_EXCEED_NONE_PTE, "exceed_none_pte") \ + EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ + EM( SCAN_EXCEED_SHARED_PTE, "exceed_shared_pte") \ EM( SCAN_PTE_NON_PRESENT, "pte_non_present") \ EM( SCAN_PAGE_RO, "no_writable_page") \EM (SCAN_LACK_REFERENCED_PAGE ,"lack_referenced_page" ) \@@ -30,7 +32,6 @@ EM( SCAN_DEL_PAGE_LRU, "could_not_delete_page_from_lru")\ EM( SCAN_ALLOC_HUGE_PAGE_FAIL, "alloc_huge_page_failed") \ EM( SCAN_CGROUP_CHARGE_FAIL, "ccgroup_charge_failed") \ - EM( SCAN_EXCEED_SWAP_PTE, "exceed_swap_pte") \ EM( SCAN_TRUNCATED, "truncated") \EMe (SCAN_PAGE_HAS_PRIVATE ,"page_has_private" ) \mm/khugepaged.c +34
-3@@ -28,6 +28,8 @@ enum scan_result {SCAN_SUCCEED ,SCAN_PMD_NULL ,SCAN_EXCEED_NONE_PTE , +SCAN_EXCEED_SWAP_PTE , +SCAN_EXCEED_SHARED_PTE ,SCAN_PTE_NON_PRESENT ,SCAN_PAGE_RO , SCAN_LACK_REFERENCED_PAGE,@@ -46,7 +48,6 @@ enum scan_result {SCAN_DEL_PAGE_LRU ,SCAN_ALLOC_HUGE_PAGE_FAIL ,SCAN_CGROUP_CHARGE_FAIL , -SCAN_EXCEED_SWAP_PTE ,SCAN_TRUNCATED , SCAN_PAGE_HAS_PRIVATE, };@@ -71,6 +72,7 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);*/ static unsigned int khugepaged_max_ptes_none__read_mostly ;static unsigned int khugepaged_max_ptes_swap__read_mostly ; +static unsigned int khugepaged_max_ptes_shared__read_mostly ;#define MM_SLOTS_HASH_BITS 10 static __read_mostly DEFINE_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);@@ -290,15 +292,43 @@ static struct kobj_attribute khugepaged_max_ptes_swap_attr =__ATTR (max_ptes_swap,0644 , khugepaged_max_ptes_swap_show, khugepaged_max_ptes_swap_store); +static ssize_t khugepaged_max_ptes_shared_show (struct kobject *kobj, +struct kobj_attribute *attr, +char *buf) +{ +return sprintf (buf,"%u\n" , khugepaged_max_ptes_shared); +} + +static ssize_t khugepaged_max_ptes_shared_store (struct kobject *kobj, +struct kobj_attribute *attr, +const char *buf,size_t count) +{ +int err; +unsigned long max_ptes_shared; + + err =kstrtoul (buf,10 , &max_ptes_shared); +if (err || max_ptes_shared >HPAGE_PMD_NR -1 ) +return -EINVAL ; + + khugepaged_max_ptes_shared = max_ptes_shared; + +return count; +} + +static struct kobj_attribute khugepaged_max_ptes_shared_attr = +__ATTR (max_ptes_shared,0644 , khugepaged_max_ptes_shared_show, + khugepaged_max_ptes_shared_store); +static struct attribute *khugepaged_attr[] = { &khugepaged_defrag_attr.attr, &khugepaged_max_ptes_none_attr.attr, + &khugepaged_max_ptes_swap_attr.attr, + &khugepaged_max_ptes_shared_attr.attr, &pages_to_scan_attr.attr, &pages_collapsed_attr.attr, &full_scans_attr.attr, &scan_sleep_millisecs_attr.attr, &alloc_sleep_millisecs_attr.attr, - &khugepaged_max_ptes_swap_attr.attr,NULL , };@@ -360,6 +390,7 @@ int __init khugepaged_init(void) khugepaged_pages_to_scan =HPAGE_PMD_NR *8 ; khugepaged_max_ptes_none =HPAGE_PMD_NR -1 ; khugepaged_max_ptes_swap =HPAGE_PMD_NR /8 ; + khugepaged_max_ptes_shared =HPAGE_PMD_NR /2 ;return 0 ; }@@ -567,7 +598,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, {struct page *page =NULL ;pte_t *_pte; -int none_or_zero =0 , result =0 , referenced =0 ; +int none_or_zero =0 , shared =0 , result =0 , referenced =0 ;bool writable = false; for (_pte = pte; _pte < pte+HPAGE_PMD_NR;
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 631
Quoted 6 lines
mm/khugepaged.c +3
-0@@ -595,6 +626,12 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,VM_BUG_ON_PAGE (!PageAnon (page), page); +if (page_mapcount (page) >1 && + ++shared > khugepaged_max_ptes_shared) { + result =SCAN_EXCEED_SHARED_PTE ;
It may be better to extract the check into a helper? For example, bool exceed_max_ptes_shared(struct page)?
Quoted 164 lines
mm/khugepaged.c +11
-1+ goto out; + } +if (PageCompound (page)) {struct page *p; page =compound_head (page);@@ -1178,7 +1215,8 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, {pmd_t *pmd;pte_t *pte, *_pte; -int ret =0 , none_or_zero =0 , result =0 , referenced =0 ; +int ret =0 , result =0 , referenced =0 ; +int none_or_zero =0 , shared =0 ;struct page *page =NULL ;unsigned long _address;spinlock_t *ptl;@@ -1228,6 +1266,12 @@ static int khugepaged_scan_pmd(struct mm_struct *mm, goto out_unmap; } +if (page_mapcount (page) >1 && + ++shared > khugepaged_max_ptes_shared) { + result =SCAN_EXCEED_SHARED_PTE ; + goto out_unmap; + } + page =compound_head (page);/* tools/testing/selftests/vm/khugepaged.c +83
-0@@ -77,6 +77,7 @@ struct khugepaged_settings {unsigned int scan_sleep_millisecs;unsigned int max_ptes_none;unsigned int max_ptes_swap; +unsigned int max_ptes_shared;unsigned long pages_to_scan; };@@ -276,6 +277,7 @@ static void write_settings(struct settings *settings) khugepaged->scan_sleep_millisecs);write_num ("khugepaged/max_ptes_none" , khugepaged->max_ptes_none);write_num ("khugepaged/max_ptes_swap" , khugepaged->max_ptes_swap); +write_num ("khugepaged/max_ptes_shared" , khugepaged->max_ptes_shared);write_num ("khugepaged/pages_to_scan" , khugepaged->pages_to_scan); }@@ -312,6 +314,7 @@ static void save_settings(void) read_num("khugepaged/scan_sleep_millisecs"), .max_ptes_none = read_num("khugepaged/max_ptes_none"), .max_ptes_swap = read_num("khugepaged/max_ptes_swap"), + .max_ptes_shared = read_num("khugepaged/max_ptes_shared"), .pages_to_scan =read_num ("khugepaged/pages_to_scan" ), };success ("OK" );@@ -843,12 +846,90 @@ static void collapse_fork_compound(void)fail ("Fail" );fill_memory (p,0 , page_size); +write_num ("khugepaged/max_ptes_shared" , hpage_pmd_nr -1 );if (wait_for_scan ("Collapse PTE table full of compound pages in child" , p))fail ("Timeout" );else if (check_huge (p))success ("OK" );else fail ("Fail" ); +write_num ("khugepaged/max_ptes_shared" , + default_settings.khugepaged.max_ptes_shared); + +validate_memory (p,0 , hpage_pmd_size); +munmap (p, hpage_pmd_size); +exit (exit_status); + } + +wait (&wstatus); + exit_status +=WEXITSTATUS (wstatus); + +printf ("Check if parent still has huge page..." ); +if (check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); +validate_memory (p,0 , hpage_pmd_size); +munmap (p, hpage_pmd_size); +} + +static void collapse_max_ptes_shared () +{ +int max_ptes_shared =read_num ("khugepaged/max_ptes_shared" ); +int wstatus; +void *p; + + p =alloc_mapping (); + +printf ("Allocate huge page..." ); +madvise (p, hpage_pmd_size,MADV_HUGEPAGE ); +fill_memory (p,0 , hpage_pmd_size); +if (check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); + +printf ("Share huge page over fork()..." ); +if (!fork ()) { +/* Do not touch settings on child exit */ + skip_settings_restore = true; + exit_status =0 ; + +if (check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); + +printf ("Trigger CoW in %d of %d..." , + hpage_pmd_nr - max_ptes_shared -1 , hpage_pmd_nr); +fill_memory (p,0 , (hpage_pmd_nr - max_ptes_shared -1 ) * page_size); +if (!check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); + +if (wait_for_scan ("Do not collapse with max_ptes_shared exeeded" , p)) +fail ("Timeout" ); +else if (!check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); + +printf ("Trigger CoW in %d of %d..." , + hpage_pmd_nr - max_ptes_shared, hpage_pmd_nr); +fill_memory (p,0 , (hpage_pmd_nr - max_ptes_shared) * page_size); +if (!check_huge (p)) +success ("OK" ); +else +fail ("Fail" ); + + +if (wait_for_scan ("Collapse with max_ptes_shared PTEs shared" , p)) +fail ("Timeout" ); +else if (check_huge (p)) +success ("OK" ); +else +fail ("Fail" );validate_memory (p,0 , hpage_pmd_size);munmap (p, hpage_pmd_size);@@ -877,6 +958,7 @@ int main(void) default_settings.khugepaged.max_ptes_none = hpage_pmd_nr -1 ; default_settings.khugepaged.max_ptes_swap = hpage_pmd_nr /8 ; + default_settings.khugepaged.max_ptes_shared = hpage_pmd_nr /2 ; default_settings.khugepaged.pages_to_scan = hpage_pmd_nr *8 ;save_settings ();@@ -894,6 +976,7 @@ int main(void)collapse_compound_extreme ();collapse_fork ();collapse_fork_compound (); +collapse_max_ptes_shared ();restore_settings (0 ); }
Re: [PATCHv3, RESEND 8/8] khugepaged: Introduce 'max_ptes_shared' tunable
Quoted 8 lines · Yang Shi
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 629
Quoted 5 lines
mm/khugepaged.c +3
-0@@ -595,6 +626,12 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,VM_BUG_ON_PAGE (!PageAnon (page), page); +if (page_mapcount (page) >1 && + ++shared > khugepaged_max_ptes_shared) { + result =SCAN_EXCEED_SHARED_PTE ;It may be better to extract the check into a helper? For example, bool
exceed_max_ptes_shared(struct page)?
It might be reasonable as part of wider cleanup: the same has to be done to khugepaged_max_ptes_none and khugepaged_max_ptes_swap.
Volunteers? :P
Signature · 1 line
Kirill A. Shutemov
Re: [PATCHv3, RESEND 8/8] khugepaged: Introduce 'max_ptes_shared' tunable
Quoted 8 lines · Kirill A. Shutemov
Quoted 7 lines · Yang Shi
mm/khugepaged.c static int __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 629
Quoted 5 lines
mm/khugepaged.c +3
-0@@ -595,6 +626,12 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,VM_BUG_ON_PAGE (!PageAnon (page), page); +if (page_mapcount (page) >1 && + ++shared > khugepaged_max_ptes_shared) { + result =SCAN_EXCEED_SHARED_PTE ;It may be better to extract the check into a helper? For example, bool
exceed_max_ptes_shared(struct page)?
Quoted 4 lines
It might be reasonable as part of wider cleanup: the same has to be done to
khugepaged_max_ptes_noneandkhugepaged_max_ptes_swap.Volunteers? :P
Fine, I will take it. :-)