mm/khugepaged: skip copying lazyfree pages on collapse
Subsystems (2)
PATCH 1/1 mm/khugepaged: skip copying lazyfree pages on collapse
The collapsing behavior of khugepaged with pages marked using MADV_FREE might cause confusion among users.
For instance, allocate a 2MB chunk using mmap and later release it by MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages as pte_none. However, for some pages marked as lazyfree with MADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.
After a successful MADV_FREE operation, if there is no subsequent write, the kernel can free the pages at any time. Therefore, in my opinion, counting lazyfree pages in max_pte_none seems reasonable.
Perhaps treating MADV_FREE like MADV_DONTNEED, not copying lazyfree pages when khugepaged collapses huge pages in the background better aligns with user expectations.
--- mm/khugepaged.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
mm/khugepaged.c +9 -1
@@ -777,6 +777,7 @@ static int __collapse_huge_page_copy(pte_t *pte,pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, + struct collapse_control *cc, spinlock_t *ptl, struct list_head *compound_pagelist) {
@@ -797,6 +798,13 @@ static int __collapse_huge_page_copy(pte_t *pte,continue ; } src_page =pte_page (pteval); + +if (cc->is_khugepaged + && !folio_test_swapbacked (page_folio (src_page))) { +clear_user_highpage (page, _address); +continue ; + } +if (copy_mc_user_highpage (page, src_page, _address, vma) >0 ) { result =SCAN_COPY_MC ;break ;
@@ -1205,7 +1213,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_unlock_write (vma->anon_vma); result =__collapse_huge_page_copy (pte, hpage, pmd, _pmd, - vma, address, pte_ptl, + vma, address, cc, pte_ptl, &compound_pagelist);pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED ))
Signature · 1 line
2.33.1
THP:
enable=madivse
defrag=defer
max_ptes_none=511
scan_sleep_millisecs=1000
alloc_sleep_millisecs=1000
Test code:
// allocate a 2MB chunk using mmap and later
// release it by MADV_FRE
root@x:/tmp# ./a.out | grep -B 23 hg 7f762a200000-7f762a400000 rw-p 00000000 00:00 0 Size: 2048 kB Anonymous: 2048 kB LazyFree: 2048 kB AnonHugePages: 0 kB THPeligible: 1 VmFlags: rd wr mr mw me ac sd hg
// allocate a 2MB chunk using mmap and later
// some pages marked as lazyfree with MADV_FREE
root@x:/tmp# ./a.out | grep -B 23 hg 7f762a200000-7f762a400000 rw-p 00000000 00:00 0 Size: 2048 kB Anonymous: 2048 kB LazyFree: 0 kB AnonHugePages: 2048 kB THPeligible: 1 VmFlags: rd wr mr mw me ac sd hg
root@x:/tmp# ./a.out [...] root@x:/tmp# echo $? 2
Quoted 67 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.After a successful
MADV_FREEoperation, if there is no subsequent write, the kernel can free the pages at any time. Therefore, in my opinion, counting lazyfree pages inmax_pte_noneseems reasonable.Perhaps treating
MADV_FREElikeMADV_DONTNEED, not copying lazyfree pages when khugepaged collapses huge pages in the background better aligns with user expectations.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/khugepaged.c +9
-2@@ -777,6 +777,7 @@ static int __collapse_huge_page_copy(pte_t *pte,pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, + struct collapse_control *cc, spinlock_t *ptl, struct list_head *compound_pagelist) {@@ -797,6 +798,13 @@ static int __collapse_huge_page_copy(pte_t *pte,continue ; } src_page =pte_page (pteval); + +if (cc->is_khugepaged + && !folio_test_swapbacked (page_folio (src_page))) { +clear_user_highpage (page, _address); +continue ; + } +if (copy_mc_user_highpage (page, src_page, _address, vma) >0 ) { result =SCAN_COPY_MC ;break ;@@ -1205,7 +1213,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_unlock_write (vma->anon_vma); result =__collapse_huge_page_copy (pte, hpage, pmd, _pmd, - vma, address, pte_ptl, + vma, address, cc, pte_ptl, &compound_pagelist);pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) --2.33.1
Quoted 42 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.After a successful
MADV_FREEoperation, if there is no subsequent write, the kernel can free the pages at any time. Therefore, in my opinion, counting lazyfree pages inmax_pte_noneseems reasonable.Perhaps treating
MADV_FREElikeMADV_DONTNEED, not copying lazyfree pages when khugepaged collapses huge pages in the background better aligns with user expectations.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/khugepaged.c +1
-0@@ -777,6 +777,7 @@ static int __collapse_huge_page_copy(pte_t *pte,pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, + struct collapse_control *cc, spinlock_t *ptl, struct list_head *compound_pagelist) {
mm/khugepaged.c static int __collapse_huge_page_copy(pte_t *pte, · line 805
Quoted 8 lines
mm/khugepaged.c +5
-0@@ -797,6 +798,13 @@ static int __collapse_huge_page_copy(pte_t *pte,continue ; } src_page =pte_page (pteval); + +if (cc->is_khugepaged + && !folio_test_swapbacked (page_folio (src_page))) { +clear_user_highpage (page, _address); +continue ;
If the page was written before khugepaged collapsed it, and khugepaged collapsed the page before memory reclaim kicked in, didn't this somehow cause data corruption?
Quoted 17 lines
mm/khugepaged.c +3
-2+ } +if (copy_mc_user_highpage (page, src_page, _address, vma) >0 ) { result =SCAN_COPY_MC ;break ;@@ -1205,7 +1213,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_unlock_write (vma->anon_vma); result =__collapse_huge_page_copy (pte, hpage, pmd, _pmd, - vma, address, pte_ptl, + vma, address, cc, pte_ptl, &compound_pagelist);pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) --2.33.1
Quoted 48 lines · Yang Shi
Quoted 46 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.After a successful
MADV_FREEoperation, if there is no subsequent write, the kernel can free the pages at any time. Therefore, in my opinion, counting lazyfree pages inmax_pte_noneseems reasonable.Perhaps treating
MADV_FREElikeMADV_DONTNEED, not copying lazyfree pages when khugepaged collapses huge pages in the background better aligns with user expectations.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/khugepaged.c +2
-0@@ -777,6 +777,7 @@ static int __collapse_huge_page_copy(pte_t *pte,pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, + struct collapse_control *cc, spinlock_t *ptl, struct list_head *compound_pagelist) {@@ -797,6 +798,13 @@ static int __collapse_huge_page_copy(pte_t *pte,continue ; } src_page =pte_page (pteval); +
Quoted 9 lines
mm/khugepaged.c static int __collapse_huge_page_copy(pte_t *pte, · line 805
Quoted 4 lines
mm/khugepaged.c +4
-0+if (cc->is_khugepaged + && !folio_test_swapbacked (page_folio (src_page))) { +clear_user_highpage (page, _address); +continue ;If the page was written before khugepaged collapsed it, and khugepaged collapsed the page before memory reclaim kicked in, didn't this somehow cause data corruption?
Thanks a lot! Yang, you're correct; indeed, there is a potential issue with data corruption.
I took a look at the check for lazyfree pages insmaps_pte_entry.
Here's the modification:
if (cc->is_khugepaged && !PageSwapBacked(src_page)
&& !pte_dirty(pteval) && !PageDirty(src_page)) {
clear_user_highpage(page, _address);
continue;
}Could you please take a look?
Thanks,
Lance
Quoted 17 lines
Quoted 17 lines
mm/khugepaged.c +3
-2+ } +if (copy_mc_user_highpage (page, src_page, _address, vma) >0 ) { result =SCAN_COPY_MC ;break ;@@ -1205,7 +1213,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_unlock_write (vma->anon_vma); result =__collapse_huge_page_copy (pte, hpage, pmd, _pmd, - vma, address, pte_ptl, + vma, address, cc, pte_ptl, &compound_pagelist);pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) --2.33.1
Quoted 66 lines · Lance Yang
Quoted 48 lines · Yang Shi
Quoted 46 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.After a successful
MADV_FREEoperation, if there is no subsequent write, the kernel can free the pages at any time. Therefore, in my opinion, counting lazyfree pages inmax_pte_noneseems reasonable.Perhaps treating
MADV_FREElikeMADV_DONTNEED, not copying lazyfree pages when khugepaged collapses huge pages in the background better aligns with user expectations.Signed-off-by Lance Yang <[email protected]>--- mm/khugepaged.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)mm/khugepaged.c +2
-0@@ -777,6 +777,7 @@ static int __collapse_huge_page_copy(pte_t *pte,pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, + struct collapse_control *cc, spinlock_t *ptl, struct list_head *compound_pagelist) {@@ -797,6 +798,13 @@ static int __collapse_huge_page_copy(pte_t *pte,continue ; } src_page =pte_page (pteval); +Quoted 9 lines
mm/khugepaged.c static int __collapse_huge_page_copy(pte_t *pte, · line 805
Quoted 4 lines
mm/khugepaged.c +4
-0+if (cc->is_khugepaged + && !folio_test_swapbacked (page_folio (src_page))) { +clear_user_highpage (page, _address); +continue ;If the page was written before khugepaged collapsed it, and khugepaged collapsed the page before memory reclaim kicked in, didn't this somehow cause data corruption?
Thanks a lot! Yang, you're correct; indeed, there is a potential issue with data corruption.
I took a look at the check for lazyfree pages in
smaps_pte_entry.
Quoted 6 lines
Here's the modification:
if (cc->is_khugepaged&& !PageSwapBacked(src_page)&& !pte_dirty(pteval) && !PageDirty(src_page)) { clear_user_highpage(page, _address); continue; }
This may be ok. But as I said in another reply, this may still incur data corruption.
Quoted 23 lines
Could you please take a look?
Thanks,
LanceQuoted 17 lines
Quoted 17 lines
mm/khugepaged.c +3
-2+ } +if (copy_mc_user_highpage (page, src_page, _address, vma) >0 ) { result =SCAN_COPY_MC ;break ;@@ -1205,7 +1213,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,anon_vma_unlock_write (vma->anon_vma); result =__collapse_huge_page_copy (pte, hpage, pmd, _pmd, - vma, address, pte_ptl, + vma, address, cc, pte_ptl, &compound_pagelist);pte_unmap (pte);if (unlikely (result !=SCAN_SUCCEED )) --2.33.1
Quoted 4 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.
Quoted 8 lines
For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.
Is that any more confusing than collapsing pte_none pages?
TBH I do not really see why this is a problem. MADV_FREE are correctly recognized same as pte_none so the user defined trashold applies.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 15 lines · Michal Hocko
Quoted 4 lines · Lance Yang
The collapsing behavior of khugepaged with pages marked using
MADV_FREEmight cause confusion among users.Quoted 8 lines
For instance, allocate a 2MB chunk using mmap and later release it by
MADV_FREE. Khugepaged will not collapse this chunk. From the user's perspective, it treats lazyfree pages aspte_none. However, for some pages marked as lazyfree withMADV_FREE, khugepaged might collapse this chunk and copy these pages to a new huge page. This inconsistency in behavior could be confusing for users.
Quoted 6 lines
Is that any more confusing than collapsing
pte_nonepages?TBH I do not really see why this is a problem.
MADV_FREEare correctly recognized same aspte_noneso the user defined trashold applies.
Sorry, I might not have expressed it clearly. MADV_FREE is correctly recognized, just like pte_none. This is also reasonable and not an issue.
IMO, since it's treated the same as pte_none, perhaps lazyfree pages shouldn't be copied to the new huge page.
Thanks,
Lance
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 3 lines · Lance Yang
IMO, since it's treated the same as
pte_none,
perhaps lazyfree pages shouldn't be copied to
the new huge page.
Why? The content of MADV_FREE page is valid until it is reclaimed.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 7 lines · Michal Hocko
Quoted 3 lines · Lance Yang
IMO, since it's treated the same as
pte_none,
perhaps lazyfree pages shouldn't be copied to
the new huge page.Why? The content of
MADV_FREEpage is valid until it is reclaimed.
IMO, if MADV_FREE pages are considered valid until reclaimed, treating them the same as pte_none might pose a conflict.
Thanks,
Lance
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 7 lines · Lance Yang
Quoted 6 lines · Michal Hocko
Quoted 3 lines · Lance Yang
IMO, since it's treated the same as
pte_none,
perhaps lazyfree pages shouldn't be copied to
the new huge page.
Quoted 5 lines
Quoted 1 line
Why? The content of
MADV_FREEpage is valid until it is reclaimed.IMO, if
MADV_FREEpages are considered valid until reclaimed, treating them the same aspte_nonemight pose a conflict.
What kind of conflict?
Signature · 2 lines
Michal Hocko SUSE Labs
Here is a part from the man page explaining the MADV_FREE semantics:
The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed. khugepaged treats lazyfree pages the same as pte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.
However, IMO, if the content of MADV_FREE pages remains valid during collapse, then khugepaged treating lazyfree pages the same as pte_none might not be suitable.
Thanks,
Lance
Quoted 19 lines · Michal Hocko
Quoted 7 lines · Lance Yang
Quoted 6 lines · Michal Hocko
Quoted 3 lines · Lance Yang
IMO, since it's treated the same as
pte_none,
perhaps lazyfree pages shouldn't be copied to
the new huge page.Quoted 5 lines
Quoted 1 line
Why? The content of
MADV_FREEpage is valid until it is reclaimed.IMO, if
MADV_FREEpages are considered valid until reclaimed, treating them the same aspte_nonemight pose a conflict.What kind of conflict?
--
Michal Hocko
SUSE Labs
I'd like to add one more point.
Users mark pages as lazyfree with MADV_FREE, expecting these pages to be reclaimed until memory pressure occurs. Currently, khugepaged treats lazyfree pages the same as pte_none, which seems reasonable. IMO, avoiding the copying of these pages to the new huge page during khugepaged collapse can better keep the semantics of MADV_FREE. It appears that lazyfree pages are reclaimed before khugepaged collapses them.
Thanks,
Lance
Quoted 48 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed. khugepaged treats lazyfree pages the same as
pte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.Thanks,
LanceQuoted 19 lines · Michal Hocko
Quoted 7 lines · Lance Yang
Quoted 6 lines · Michal Hocko
Quoted 3 lines · Lance Yang
IMO, since it's treated the same as
pte_none,
perhaps lazyfree pages shouldn't be copied to
the new huge page.Quoted 5 lines
Quoted 1 line
Why? The content of
MADV_FREEpage is valid until it is reclaimed.IMO, if
MADV_FREEpages are considered valid until reclaimed, treating them the same aspte_nonemight pose a conflict.What kind of conflict?
--
Michal Hocko
SUSE Labs
Quoted 11 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
Quoted 2 lines
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed.
If there is no memory pressure then this might not ever happen. User cannot make any assumption about their content once madvise call has been done. The content has to be considered lost. Sure the userspace might have means to tell those pages from zero pages and recheck after the write but that is about it.
Quoted 11 lines
khugepaged
treats lazyfree pages the same aspte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.
Why?
Unless I am missing something (which is possible of course) I do not really see why dropping the content of those pages and replacing them with a THP is any difference from reclaiming those pages and then faulting in a non-THP zero page.
Now, if khugepaged reused the original content of MADV_FREE pages that would be a slightly different story. I can see why users would expect zero pages to back madvised area.
Signature · 2 lines
Michal Hocko SUSE Labs
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
of MADV_FREE?
What do you think?
Thanks,
Lance
Quoted 50 lines · Michal Hocko
Quoted 11 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
Quoted 2 lines
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed.
If there is no memory pressure then this might not ever happen. User cannot make any assumption about their content once madvise call has been done. The content has to be considered lost. Sure the userspace might have means to tell those pages from zero pages and recheck after the write but that is about it.
Quoted 11 lines
khugepaged
treats lazyfree pages the same aspte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.Why?
Unless I am missing something (which is possible of course) I do not really see why dropping the content of those pages and replacing them with a THP is any difference from reclaiming those pages and then faulting in a non-THP zero page.
Now, if khugepaged reused the original content of
MADV_FREEpages that would be a slightly different story. I can see why users would expect zero pages to back madvised area.
--
Michal Hocko
SUSE Labs
Quoted 6 lines · Lance Yang
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
ofMADV_FREE?What do you think?
Why do we even want to change any behavior here? A lot of things "might cause confusion among users". Even worse, a lot of things do cause confusion ;)
Signature · 3 lines
Cheers, David / dhildenb
Quoted 4 lines · Lance Yang
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
ofMADV_FREE?
I do not see any reason why. And you are not providing any actual reasoning either. Unless you have a very specific example in mind I do not think we want to change the current behavior.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 7 lines · Lance Yang
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
ofMADV_FREE?What do you think?
First of all, khugepaged doesn't treat MADV_FREE pages as pte_none IIUC. The khugepaged does skip the 2M block if all the pages are old and unreferenced pages in the range in hpage_collapse_scan_pmd(), then repeat the check in collapse_huge_page() again.
And MADV_FREE pages are just old and unreferenced. This is actually what your first test case does. The whole 2M range is MADV_FREE range, so they are skipped by khugepaged.
But if the partial range is MADV_FREE, khugepaged won't skip them. This is what your second test case does.
Secondly, I think it depends on the semantics of MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".
I'm supposed it should be "abcd1234" since MADV_FREE pages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copy MADV_FREE pages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?
The current behavior is copying the page.
Quoted 55 lines
Thanks,
LanceQuoted 50 lines · Michal Hocko
Quoted 11 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
Quoted 2 lines
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed.
If there is no memory pressure then this might not ever happen. User cannot make any assumption about their content once madvise call has been done. The content has to be considered lost. Sure the userspace might have means to tell those pages from zero pages and recheck after the write but that is about it.
Quoted 11 lines
khugepaged
treats lazyfree pages the same aspte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.Why?
Unless I am missing something (which is possible of course) I do not really see why dropping the content of those pages and replacing them with a THP is any difference from reclaiming those pages and then faulting in a non-THP zero page.
Now, if khugepaged reused the original content of
MADV_FREEpages that would be a slightly different story. I can see why users would expect zero pages to back madvised area.
--
Michal Hocko
SUSE Labs
Hey Michal, David, Yang,
I sincerely appreciate your time!
I still have two questions that are perplexing me.
First question:
Given that khugepaged doesn't treat MADV_FREE pages as pte_none, why skip the 2M block when all the pages within the range are old and unreferenced, but won't skip if the partial range is MADV_FREE, even if it's not redirtied? Why make this distinction? Would it not be more straightforward to maintain if either all were skipped or not?
Second question:
Does copying lazyfree pages (not redirtied) to the new huge page during khugepaged collapse undermine the semantics of MADV_FREE? Users mark pages as lazyfree with MADV_FREE, expecting these pages to be eventually reclaimed. Even without subsequent writes, these pages will no longer be reclaimed, even if memory pressure occurs.
BR,
Lance
Quoted 92 lines · Yang Shi
Quoted 7 lines · Lance Yang
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
ofMADV_FREE?What do you think?
First of all, khugepaged doesn't treat
MADV_FREEpages aspte_noneIIUC. The khugepaged does skip the 2M block if all the pages are old and unreferenced pages in the range inhpage_collapse_scan_pmd(), then repeat the check incollapse_huge_page()again.And
MADV_FREEpages are just old and unreferenced. This is actually what your first test case does. The whole 2M range isMADV_FREErange, so they are skipped by khugepaged.But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?The current behavior is copying the page.
Quoted 55 lines
Thanks,
LanceQuoted 50 lines · Michal Hocko
Quoted 11 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
Quoted 2 lines
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed.
If there is no memory pressure then this might not ever happen. User cannot make any assumption about their content once madvise call has been done. The content has to be considered lost. Sure the userspace might have means to tell those pages from zero pages and recheck after the write but that is about it.
Quoted 11 lines
khugepaged
treats lazyfree pages the same aspte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.Why?
Unless I am missing something (which is possible of course) I do not really see why dropping the content of those pages and replacing them with a THP is any difference from reclaiming those pages and then faulting in a non-THP zero page.
Now, if khugepaged reused the original content of
MADV_FREEpages that would be a slightly different story. I can see why users would expect zero pages to back madvised area.
--
Michal Hocko
SUSE Labs
Quoted 7 lines · Lance Yang
Hey Michal, David, Yang,
I sincerely appreciate your time!
I still have two questions that are perplexing me.
Quoted 8 lines
First question:
Given that khugepaged doesn't treatMADV_FREEpages aspte_none, why skip the 2M block when all the pages within the range are old and unreferenced, but won't skip if the partial range isMADV_FREE, even if it's not redirtied? Why make this distinction? Would it not be more straightforward to maintain if either all were skipped or not?
It is just some heuristic in the code and may be some arbitrary choice. It could controlled in a more fine-grained way if we really see some workloads get benefit.
Quoted 10 lines
Second question:
Does copying lazyfree pages (not redirtied) to the new huge page during khugepaged collapse undermine the semantics ofMADV_FREE? Users mark pages as lazyfree withMADV_FREE, expecting these pages to be eventually reclaimed. Even without subsequent writes, these pages will no longer be reclaimed, even if memory pressure occurs.
Yeah, it just means khugepaged wins the race against page reclaim. I'm supposed the delayed free is one of the design goals of MADV_FREE, and the risk is the pages may not be freed eventually. If you want immediate free or more deterministic behavior, you should use MADV_DONTNEED or munmap IIUC.
Quoted 97 lines
BR,
LanceQuoted 92 lines · Yang Shi
Quoted 7 lines · Lance Yang
How about blocking khugepaged from
collapsing lazyfree pages? This way,
is it not better to keep the semantics
ofMADV_FREE?What do you think?
First of all, khugepaged doesn't treat
MADV_FREEpages aspte_noneIIUC. The khugepaged does skip the 2M block if all the pages are old and unreferenced pages in the range inhpage_collapse_scan_pmd(), then repeat the check incollapse_huge_page()again.And
MADV_FREEpages are just old and unreferenced. This is actually what your first test case does. The whole 2M range isMADV_FREErange, so they are skipped by khugepaged.But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?The current behavior is copying the page.
Quoted 55 lines
Thanks,
LanceQuoted 50 lines · Michal Hocko
Quoted 11 lines · Lance Yang
Here is a part from the man page explaining the
MADV_FREEsemantics:The kernel can thus free thesepages, but the freeing could be delayed until memory pressure occurs. For each of the pages that has been marked to be freed but has not yet been freed, the free operation will be canceled if the caller writes into the page. If there is no subsequent write, the kernel can free the pages at any time.
Quoted 2 lines
IIUC, if there is no subsequent write, lazyfree pages will eventually be reclaimed.
If there is no memory pressure then this might not ever happen. User cannot make any assumption about their content once madvise call has been done. The content has to be considered lost. Sure the userspace might have means to tell those pages from zero pages and recheck after the write but that is about it.
Quoted 11 lines
khugepaged
treats lazyfree pages the same aspte_none, avoiding copying them to the new huge page during collapse. It seems that lazyfree pages are reclaimed before khugepaged collapses them. This aligns with user expectations.However, IMO, if the content of
MADV_FREEpages remains valid during collapse, then khugepaged treating lazyfree pages the same aspte_nonemight not be suitable.Why?
Unless I am missing something (which is possible of course) I do not really see why dropping the content of those pages and replacing them with a THP is any difference from reclaiming those pages and then faulting in a non-THP zero page.
Now, if khugepaged reused the original content of
MADV_FREEpages that would be a slightly different story. I can see why users would expect zero pages to back madvised area.
--
Michal Hocko
SUSE Labs
Quoted 8 lines · Yang Shi
But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".
Correct. You cannot assume the content of the first page as it could have been reclaimed at any time.
Quoted 6 lines
I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?
Right. As pointed above this is a valid outcome if the page has been dropped. User has means to tell that from /proc/vmstat though. Not in a great precision but I think it would be really surprising to not see any pglazyfreed yet the content is gone. I think it would be legit to call it a bug. One could argue the bug would be in the accounting rather than the khugepaged implementation because madvised pages could be dropped at any time. But I think it makes more sense to copy the existing content.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · Michal Hocko
Quoted 8 lines · Yang Shi
But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".Correct. You cannot assume the content of the first page as it could have been reclaimed at any time.
Quoted 6 lines
I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?
Quoted 7 lines
Right. As pointed above this is a valid outcome if the page has been dropped. User has means to tell that from
/proc/vmstatthough. Not in a great precision but I think it would be really surprising to not see any pglazyfreed yet the content is gone. I think it would be legit to call it a bug. One could argue the bug would be in the accounting rather than the khugepaged implementation because madvised pages could be dropped at any time. But I think it makes more sense to copy the existing content.
Yeah, as long as khugepaged sees the MADV_FREE pages, it means they have "NOT" been dropped yet. It may be dropped later if memory pressure occurs, but anyway khugepaged wins the race and khugepaged can't assume the pages will be dropped before they get redirtied. So copying the content does make sense.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Quoted 23 lines · Yang Shi
Quoted 21 lines · Michal Hocko
Quoted 8 lines · Yang Shi
But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".Correct. You cannot assume the content of the first page as it could have been reclaimed at any time.
Quoted 6 lines
I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?
Quoted 7 lines
Quoted 7 lines
Right. As pointed above this is a valid outcome if the page has been dropped. User has means to tell that from
/proc/vmstatthough. Not in a great precision but I think it would be really surprising to not see any pglazyfreed yet the content is gone. I think it would be legit to call it a bug. One could argue the bug would be in the accounting rather than the khugepaged implementation because madvised pages could be dropped at any time. But I think it makes more sense to copy the existing content.
+1. I agree that the content should be dropped iff pglazyfreed is incremented. Of course, we could do that here, but I don't think there is a good reason to, and we should just copy the contents.
Quoted 5 lines
Yeah, as long as khugepaged sees the
MADV_FREEpages, it means they have "NOT" been dropped yet. It may be dropped later if memory pressure occurs, but anyway khugepaged wins the race and khugepaged can't assume the pages will be dropped before they get redirtied. So copying the content does make sense.
Per Lance, I kinda get that this "undermines" MADV_FREE, insofar that, from the user's perspective, that memory which was intended as a buffer against OOM kill scenarios, is no longer there to reclaim trivially. I don't have a real world example where this is an issue, however. Also, not copying the contents doesn't change that fact.
The proper alternative, if you want to make the "undermining" argument, is for khugepaged to stay away from hugepage regions with any MADV_FREE pages. I think it's fair to assume MADV_FREE'd memory is likely cold memory, and therefore not a good hugepage target anyways. However, it'd be unfortunate if there were a couple MADV_FREE pages in the middle of an otherwise hot / highly-utilized hugepage region that would prevent it from being pmd-mapped via khugepaged. But.. this is exactly-ish what you get when hugepage-ware system/runtime allocators split THPs to free up internal caches.
Best,
Zach
Quoted 3 lines
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Hey Zach, Yang, Michal, and David,
Please accept my sincerest apologies for the delayed response.
Thanks for the replies; it‘s been very helpful to me! I also appreciate the valuable information you’ve shared!
I agree that it’s not a good idea to let khugepaged avoid any pages marked with MADV_FREE.
Thanks again for your time!
Best,
Lance
Quoted 66 lines · Zach O'Keefe
Quoted 23 lines · Yang Shi
Quoted 21 lines · Michal Hocko
Quoted 8 lines · Yang Shi
But if the partial range is
MADV_FREE, khugepaged won't skip them. This is what your second test case does.Secondly, I think it depends on the semantics of
MADV_FREE, particularly how to treat the redirtied pages. TBH I'm always confused by the semantics. For example, the page contained "abcd", then it was MADV_FREE'ed, then it was written again with "1234" after "abcd". So the user should expect to see "abcd1234" or "00001234".Correct. You cannot assume the content of the first page as it could have been reclaimed at any time.
Quoted 6 lines
I'm supposed it should be "
abcd1234" sinceMADV_FREEpages are still valid and available, if I'm wrong please feel free to correct me. If so we should always copyMADV_FREEpages in khugepaged regardless of whether it is redirtied or not otherwise it may incur data corruption. If we don't copy, then the follow up redirty after collapse to the hugepage may return "00001234", right?Quoted 7 lines
Quoted 7 lines
Right. As pointed above this is a valid outcome if the page has been dropped. User has means to tell that from
/proc/vmstatthough. Not in a great precision but I think it would be really surprising to not see any pglazyfreed yet the content is gone. I think it would be legit to call it a bug. One could argue the bug would be in the accounting rather than the khugepaged implementation because madvised pages could be dropped at any time. But I think it makes more sense to copy the existing content.+1. I agree that the content should be dropped iff pglazyfreed is incremented. Of course, we could do that here, but I don't think there is a good reason to, and we should just copy the contents.
Quoted 5 lines
Yeah, as long as khugepaged sees the
MADV_FREEpages, it means they have "NOT" been dropped yet. It may be dropped later if memory pressure occurs, but anyway khugepaged wins the race and khugepaged can't assume the pages will be dropped before they get redirtied. So copying the content does make sense.Per Lance, I kinda get that this "undermines"
MADV_FREE, insofar that, from the user's perspective, that memory which was intended as a buffer against OOM kill scenarios, is no longer there to reclaim trivially. I don't have a real world example where this is an issue, however. Also, not copying the contents doesn't change that fact.The proper alternative, if you want to make the "undermining" argument, is for khugepaged to stay away from hugepage regions with any
MADV_FREEpages. I think it's fair to assume MADV_FREE'd memory is likely cold memory, and therefore not a good hugepage target anyways. However, it'd be unfortunate if there were a coupleMADV_FREEpages in the middle of an otherwise hot / highly-utilized hugepage region that would prevent it from being pmd-mapped via khugepaged. But.. this is exactly-ish what you get when hugepage-ware system/runtime allocators split THPs to free up internal caches.Best,
ZachQuoted 3 lines
Quoted 3 lines
--
Michal Hocko
SUSE Labs