mm: split thp synchronously on MADV_DONTNEED and munmap
Subsystems (7)
PATCH v2 1/1 mm: split thp synchronously on MADV_DONTNEED and munmap
Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through MADV_DONTNEED to keep the memory cost low.
The kernel defers the split and release of THPs until there is memory pressure. This complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.
More specifically these applications monitor their cgroup usage to decide if they can expand the memory footprint or release some (unneeded/cold) buffer. They uses madvise(MADV_DONTNEED) to release the memory which basically puts the THP into defer list. These deferred THPs are still charged to the cgroup which leads to bloated usage read by the application and making wrong decisions. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.
Internally we added a cgroup interface to trigger the split of deferred THPs for that cgroup but this is hacky and exposing kernel internals to users. This patch solves this problem in a more general way for the users by splitting the THPS synchronously on MADV_DONTNEED. This patch does the same for munmap() too.
---
Changes since v1:
- Switched from task_struct based list passing to zap_details struct as suggested by Kirill A. Shutemov.
- Updated commit message as suggested by David Hildenbrand.
- Covers munmap() use-case as well as suggested by David Hildenbrand.
- This patch has dependency on Peter Xu's patch "mm: Don't skip swap entry even if zap_details specified".
include/linux/huge_mm.h | 2 +- include/linux/mm.h | 5 +++ include/linux/mmzone.h | 5 +++ include/linux/rmap.h | 1 + mm/huge_memory.c | 75 ++++++++++++++++++++++++++++++++------- mm/madvise.c | 2 +- mm/memory.c | 78 +++++++++++++++++++++++++++++++---------- mm/mmap.c | 2 +- mm/rmap.c | 27 ++++++++++---- 9 files changed, 158 insertions(+), 39 deletions(-)
include/linux/huge_mm.h +1 -1
@@ -191,7 +191,7 @@ static inline int split_huge_page(struct page *page) {return split_huge_page_to_list (page,NULL ); } -void deferred_split_huge_page (struct page *page); +void deferred_split_huge_page (struct page *page,struct list_head *list);void __split_huge_pmd (struct vm_area_struct *vma,pmd_t *pmd,unsigned long address,bool freeze,struct page *page);
include/linux/mm.h +5 -0
@@ -1831,6 +1831,7 @@ extern void user_shm_unlock(size_t, struct ucounts *);struct zap_details {struct address_space *zap_mapping;/* Check page->mapping if set */ struct page *single_page;/* Locked page to be unmapped */ +struct list_head *defer_list;/* List to ret THPs to split */ };/*
@@ -1856,8 +1857,12 @@ void zap_vma_ptes(struct vm_area_struct *vma, unsigned long address, unsigned long size);void zap_page_range (struct vm_area_struct *vma,unsigned long address,unsigned long size); +void zap_page_range_sync (struct vm_area_struct *vma,unsigned long address, +unsigned long size);void unmap_vmas (struct mmu_gather *tlb,struct vm_area_struct *start_vma,unsigned long start,unsigned long end); +void unmap_vmas_sync (struct mmu_gather *tlb,struct vm_area_struct *start_vma, +unsigned long start,unsigned long end);struct mmu_notifier_range ;
include/linux/mmzone.h +5 -0
@@ -795,6 +795,11 @@ struct deferred_split {struct list_head split_queue; unsigned long split_queue_len; }; +void split_local_deferred_list (struct list_head *defer_list); +#else +static inline void split_local_deferred_list (struct list_head *defer_list) +{ +}#endif /*
include/linux/rmap.h +1 -0
@@ -175,6 +175,7 @@ void do_page_add_anon_rmap(struct page *, struct vm_area_struct *,void page_add_new_anon_rmap (struct page *,struct vm_area_struct *,unsigned long ,bool );void page_add_file_rmap (struct page *,bool ); +void page_remove_rmap_list (struct page *,bool ,struct list_head *);void page_remove_rmap (struct page *,bool ); void hugepage_add_anon_rmap(struct page *, struct vm_area_struct *,
mm/huge_memory.c +63 -12
94 lines not on this page · raw message
mm/madvise.c +1 -1
@@ -795,7 +795,7 @@ static int madvise_free_single_vma(struct vm_area_struct *vma,static long madvise_dontneed_single_vma (struct vm_area_struct *vma,unsigned long start,unsigned long end) { -zap_page_range (vma, start, end - start); +zap_page_range_sync (vma, start, end - start);return 0 ; }
mm/memory.c +60 -18
120 lines not on this page · raw message
mm/mmap.c +1 -1
@@ -2652,7 +2652,7 @@ static void unmap_region(struct mm_struct *mm,lru_add_drain ();tlb_gather_mmu (&tlb, mm);update_hiwater_rss (mm); -unmap_vmas (&tlb, vma, start, end); +unmap_vmas_sync (&tlb, vma, start, end);free_pgtables (&tlb, vma, prev ? prev->vm_end :FIRST_USER_ADDRESS , next ? next->vm_start :USER_PGTABLES_CEILING );tlb_finish_mmu (&tlb);
mm/rmap.c +21 -6
@@ -1292,7 +1292,8 @@ static void page_remove_file_rmap(struct page *page, bool compound)clear_page_mlock (page); } -static void page_remove_anon_compound_rmap (struct page *page) +static void page_remove_anon_compound_rmap (struct page *page, +struct list_head *list) {int i, nr;
@@ -1324,7 +1325,7 @@ static void page_remove_anon_compound_rmap(struct page *page)* small page is still mapped. */ if (nr && nr <thp_nr_pages (page)) -deferred_split_huge_page (page); +deferred_split_huge_page (page, list); }else { nr =thp_nr_pages (page); }
@@ -1337,13 +1338,15 @@ static void page_remove_anon_compound_rmap(struct page *page) }/** -* page_remove_rmap - take down pte mapping from a page +* page_remove_rmap_list - take down pte mapping from a page * @page: page to remove mapping from * @compound: uncharge the page as compound or small page +* @list: list to return THPs to split * * The caller needs to hold the pte lock. */ -void page_remove_rmap (struct page *page,bool compound) +void page_remove_rmap_list (struct page *page,bool compound, +struct list_head *list) {lock_page_memcg (page);
@@ -1353,7 +1356,7 @@ void page_remove_rmap(struct page *page, bool compound) }if (compound) { -page_remove_anon_compound_rmap (page); +page_remove_anon_compound_rmap (page, list); goto out; }
@@ -1372,7 +1375,7 @@ void page_remove_rmap(struct page *page, bool compound)clear_page_mlock (page);if (PageTransCompound (page)) -deferred_split_huge_page (compound_head (page)); +deferred_split_huge_page (compound_head (page), list);/* * It would be tidy to reset the PageAnon mapping here,
@@ -1387,6 +1390,18 @@ void page_remove_rmap(struct page *page, bool compound)unlock_page_memcg (page); } +/** +* page_remove_rmap - take down pte mapping from a page +* @page: page to remove mapping from +* @compound: uncharge the page as compound or small page +* +* The caller needs to hold the pte lock. +*/ +void page_remove_rmap (struct page *page,bool compound) +{ +page_remove_rmap_list (page, compound,NULL ); +} +/* * @arg: enum ttu_flags will be passed to this argument */
Signature · 1 line
2.34.0.rc2.393.gf8c9666880-goog
Quoted 21 lines · Shakeel Butt
Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through
MADV_DONTNEEDto keep the memory cost low.The kernel defers the split and release of THPs until there is memory pressure. This complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.
More specifically these applications monitor their cgroup usage to decide if they can expand the memory footprint or release some (unneeded/cold) buffer. They uses
madvise(MADV_DONTNEED)to release the memory which basically puts the THP into defer list. These deferred THPs are still charged to the cgroup which leads to bloated usage read by the application and making wrong decisions. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.
Quoted 6 lines
Internally we added a cgroup interface to trigger the split of deferred THPs for that cgroup but this is hacky and exposing kernel internals to users. This patch solves this problem in a more general way for the users by splitting the THPS synchronously on
MADV_DONTNEED. This patch does the same formunmap()too.
I'll have to defer diving into the code.
Just a comment: It might be good to add that there are still cases where splitting the compound page can fail -- for example, if the page is still pinned/referenced.
So if you have a THP and intended to only pin/reference e.g., the first 4k of it (e.g., O_DIRECT, io_uring fixed buffers), MADV_DONTNEED/unmap e.g., the last 4k of it will not split synchronously.
In addition to explicit user action on a compound page; I remember there might be other kernel-internal temporary references that could theoretically block splitting, but maybe most of them are at least for now limited to !compound pages.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 36 lines · David Hildenbrand
Quoted 21 lines · Shakeel Butt
Many applications do sophisticated management of their heap memory for better performance but with low cost. We have a bunch of such applications running on our production and examples include caching and data storage services. These applications keep their hot data on the THPs for better performance and release the cold data through
MADV_DONTNEEDto keep the memory cost low.The kernel defers the split and release of THPs until there is memory pressure. This complicates the memory management of these sophisticated applications which then needs to look into low level kernel handling of THPs to better gauge their headroom for expansion.
More specifically these applications monitor their cgroup usage to decide if they can expand the memory footprint or release some (unneeded/cold) buffer. They uses
madvise(MADV_DONTNEED)to release the memory which basically puts the THP into defer list. These deferred THPs are still charged to the cgroup which leads to bloated usage read by the application and making wrong decisions. In addition these applications are very latency sensitive and would prefer to not face memory reclaim due to non-deterministic nature of reclaim.Quoted 6 lines
Internally we added a cgroup interface to trigger the split of deferred THPs for that cgroup but this is hacky and exposing kernel internals to users. This patch solves this problem in a more general way for the users by splitting the THPS synchronously on
MADV_DONTNEED. This patch does the same formunmap()too.I'll have to defer diving into the code.
Just a comment: It might be good to add that there are still cases where splitting the compound page can fail -- for example, if the page is still pinned/referenced.
Quoted 9 lines
So if you have a THP and intended to only pin/reference e.g., the first 4k of it (e.g.,
O_DIRECT,io_uringfixed buffers), MADV_DONTNEED/unmap e.g., the last 4k of it will not split synchronously.In addition to explicit user action on a compound page; I remember there might be other kernel-internal temporary references that could theoretically block splitting, but maybe most of them are at least for now limited to !compound pages.
Hi David,
Thanks for your time (and apologies) but I have to rescind this patch for now due to mistaken performance impact. Let's move the discussion to the other thread and decide next steps there.
thanks,
Shakeel