mm: make try_to_unmap_one support batched unmap for anon large folios
Subsystems (4)
RFC PATCH 1/1 mm: make try_to_unmap_one support batched unmap for anon large folios
From: Barry Song <[email protected]>
My commit 354dffd29575c ("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.
Handling __folio_try_share_anon_rmap() and swap_duplicate() becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.
SPARC is currently the only architecture that implements arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.
Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <errno.h>
#define SIZE_MB 256
#define SIZE_BYTES (SIZE_MB * 1024 * 1024)
int main() {
void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (addr == MAP_FAILED) {
perror("mmap failed");
return 1;
}
memset(addr, 0, SIZE_BYTES);
struct timespec start, end;
clock_gettime(CLOCK_MONOTONIC, &start);
if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) {
perror("madvise(MADV_PAGEOUT) failed");
munmap(addr, SIZE_BYTES);
return 1;
}
clock_gettime(CLOCK_MONOTONIC, &end);
long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 +
(end.tv_nsec - start.tv_nsec);
printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n",
duration_ns, duration_ns / 1e6);
munmap(addr, SIZE_BYTES);
return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT) took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 1347834992 ns (1347.835 ms)
w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT) took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 690790000 ns (690.790 ms)
We found that the time to reclaim this memory was reduced by half.
--- include/linux/swap.h | 4 +-- mm/memory.c | 2 +- mm/rmap.c | 79 +++++++++++++++++++++++++++++--------------- mm/swapfile.c | 10 ++++-- 4 files changed, 62 insertions(+), 33 deletions(-)
include/linux/swap.h +2 -2
@@ -479,7 +479,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int add_swap_count_continuation (swp_entry_t ,gfp_t );extern void swap_shmem_alloc (swp_entry_t ,int ); -extern int swap_duplicate (swp_entry_t ); +extern int swap_duplicate (swp_entry_t ,int nr);extern int swapcache_prepare (swp_entry_t entry,int nr);extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);
@@ -546,7 +546,7 @@ static inline void swap_shmem_alloc(swp_entry_t swp, int nr) { } -static inline int swap_duplicate (swp_entry_t swp) +static inline int swap_duplicate (swp_entry_t swp,int nr) {return 0 ; }
mm/memory.c +1 -1
@@ -803,7 +803,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,swp_entry_t entry =pte_to_swp_entry (orig_pte);if (likely (!non_swap_entry (entry))) { -if (swap_duplicate (entry) <0 ) +if (swap_duplicate (entry,1 ) <0 )return -EIO ;/* make sure dst_mm is on swapoff's mmlist. */
mm/rmap.c +52 -27
122 lines not on this page · raw message
mm/swapfile.c +7 -3
@@ -3550,13 +3550,17 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); -VM_WARN_ON (usage ==1 && nr >1 ); ci =lock_cluster (si, offset); err =0 ;for (i =0 ; i < nr; i++) { count = si->swap_map[offset + i]; +/* +* We only support batched swap_duplicate() for unmapping +* exclusive large folios where count should be zero +*/ +VM_WARN_ON (usage ==1 && nr >1 &&swap_count (count));/* * swapin_readahead() doesn't check if a swap entry is valid, so the * swap entry could be SWAP_MAP_BAD. Check here with lock held.
@@ -3626,11 +3630,11 @@ void swap_shmem_alloc(swp_entry_t entry, int nr)* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which * might occur if a page table entry has got corrupted. */ -int swap_duplicate (swp_entry_t entry) +int swap_duplicate (swp_entry_t entry,int nr) {int err =0 ; -while (!err &&__swap_duplicate (entry,1 ,1 ) == -ENOMEM ) +while (!err &&__swap_duplicate (entry,1 , nr) == -ENOMEM ) err =add_swap_count_continuation (entry,GFP_ATOMIC );return err; }
Signature · 1 line
2.39.3 (Apple Git-146)
Quoted 122 lines · Barry Song
From: Barry Song <[email protected]>
My commit
354dffd29575c("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.Handling
__folio_try_share_anon_rmap()andswap_duplicate()becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.SPARC is currently the only architecture that implements
arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <string.h> #include <time.h> #include <unistd.h> #include <errno.h> #define SIZE_MB 256 #define SIZE_BYTES (SIZE_MB * 1024 * 1024) int main() { void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed"); return 1; }memset(addr, 0, SIZE_BYTES); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) { perror("madvise(MADV_PAGEOUT) failed"); munmap(addr, SIZE_BYTES); return 1; } clock_gettime(CLOCK_MONOTONIC, &end); long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec); printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n", duration_ns, duration_ns / 1e6); munmap(addr, SIZE_BYTES); return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1347834992 ns (1347.835 ms)w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 690790000 ns (690.790 ms)We found that the time to reclaim this memory was reduced by half.
Cc David Hildenbrand <[email protected]>Cc Baolin Wang <[email protected]>Cc Ryan Roberts <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Liam R. Howlett <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Harry Yoo <[email protected]>Cc Kairui Song <[email protected]>Cc Chris Li <[email protected]>Cc Baoquan He <[email protected]>Cc Dan Schatzberg <[email protected]>Cc Kaixiong Yu <[email protected]>Cc Fan Ni <[email protected]>Cc Tangquan Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 4 +-- mm/memory.c | 2 +- mm/rmap.c | 79 +++++++++++++++++++++++++++++--------------- mm/swapfile.c | 10 ++++-- 4 files changed, 62 insertions(+), 33 deletions(-)include/linux/swap.h +0
-0
include/linux/swap.h void put_swap_folio(struct folio *folio, swp_entry_t entry); · line 482
Quoted 5 lines
include/linux/swap.h +1
-1@@ -479,7 +479,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int add_swap_count_continuation (swp_entry_t ,gfp_t );extern void swap_shmem_alloc (swp_entry_t ,int ); -extern int swap_duplicate (swp_entry_t ); +extern int swap_duplicate (swp_entry_t ,int nr);
Missed writing the parameter name here.
Quoted 193 lines
include/linux/swap.h +1
-1extern int swapcache_prepare (swp_entry_t entry,int nr);extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -546,7 +546,7 @@ static inline void swap_shmem_alloc(swp_entry_t swp, int nr) { } -static inline int swap_duplicate (swp_entry_t swp) +static inline int swap_duplicate (swp_entry_t swp,int nr) {return 0 ; }mm/memory.c +1
-1@@ -803,7 +803,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,swp_entry_t entry =pte_to_swp_entry (orig_pte);if (likely (!non_swap_entry (entry))) { -if (swap_duplicate (entry) <0 ) +if (swap_duplicate (entry,1 ) <0 )return -EIO ;/* make sure dst_mm is on swapoff's mmlist. */ mm/rmap.c +52
-27@@ -1845,23 +1845,42 @@ void folio_remove_rmap_pud(struct folio *folio, struct page *page,#endif } -/* We support batch unmapping of PTEs for lazyfree large folios */ +/* +* We support batch unmapping of PTEs for lazyfree or exclusive anon large +* folios +*/ static inline bool can_batch_unmap_folio_ptes (unsigned long addr, -struct folio *folio,pte_t *ptep) +struct folio *folio,pte_t *ptep,bool exclusive) {const fpb_t fpb_flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;int max_nr =folio_nr_pages (folio); +#ifndef __HAVE_ARCH_UNMAP_ONE +bool no_arch_unmap = true; +#else +bool no_arch_unmap = false; +#endif pte_t pte =ptep_get (ptep); +int mapped_nr; -if (!folio_test_anon (folio) ||folio_test_swapbacked (folio)) +if (!folio_test_anon (folio))return false;if (pte_unused (pte))return false;if (pte_pfn (pte) !=folio_pfn (folio))return false; -return folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , -NULL ,NULL ) == max_nr; + mapped_nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , +NULL ,NULL ); +if (mapped_nr != max_nr) +return false; +if (!folio_test_swapbacked (folio)) +return true; + +/* +* The large folio is fully mapped and its mapcount is the same as its +* number of pages, it must be exclusive. +*/ +return no_arch_unmap && exclusive &&folio_mapcount (folio) == max_nr; }/* @@ -2025,7 +2044,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,folio_mark_dirty (folio); }else if (likely (pte_present (pteval))) {if (folio_test_large (folio) && !(flags &TTU_HWPOISON ) && -can_batch_unmap_folio_ptes (address, folio, pvmw.pte)) +can_batch_unmap_folio_ptes (address, folio, pvmw.pte, + anon_exclusive)) nr_pages =folio_nr_pages (folio); end_addr = address + nr_pages *PAGE_SIZE ;flush_cache_range (vma, address, end_addr);@@ -2141,8 +2161,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto discard; } -if (swap_duplicate (entry) <0 ) { -set_pte_at (mm, address, pvmw.pte, pteval); +if (swap_duplicate (entry, nr_pages) <0 ) { +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }@@ -2159,9 +2179,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && -folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); -set_pte_at (mm, address, pvmw.pte, pteval); +__folio_try_share_anon_rmap (folio, subpage, nr_pages, +RMAP_LEVEL_PTE )) { +swap_free_nr (entry, nr_pages); +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }if (list_empty (&mm->mmlist)) {@@ -2170,23 +2191,27 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,list_add (&mm->mmlist, &init_mm.mmlist);spin_unlock (&mmlist_lock); } -dec_mm_counter (mm,MM_ANONPAGES ); -inc_mm_counter (mm,MM_SWAPENTS ); - swp_pte =swp_entry_to_pte (entry); -if (anon_exclusive) - swp_pte =pte_swp_mkexclusive (swp_pte); -if (likely (pte_present (pteval))) { -if (pte_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); - }else { -if (pte_swp_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_swp_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); +add_mm_counter (mm,MM_ANONPAGES , -nr_pages); +add_mm_counter (mm,MM_SWAPENTS , nr_pages); +/* TODO: let set_ptes() support swp_offset advance */ +for (pte_t *ptep = pvmw.pte; address < end_addr; + entry.val++, address +=PAGE_SIZE , ptep++) { + swp_pte =swp_entry_to_pte (entry); +if (anon_exclusive) + swp_pte =pte_swp_mkexclusive (swp_pte); +if (likely (pte_present (pteval))) { +if (pte_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + }else { +if (pte_swp_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_swp_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + } +set_pte_at (mm, address, ptep, swp_pte); } -set_pte_at (mm, address, pvmw.pte, swp_pte); }else {/* * This is a locked file-backed folio, mm/swapfile.c +7
-3@@ -3550,13 +3550,17 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); -VM_WARN_ON (usage ==1 && nr >1 ); ci =lock_cluster (si, offset); err =0 ;for (i =0 ; i < nr; i++) { count = si->swap_map[offset + i]; +/* +* We only support batched swap_duplicate() for unmapping +* exclusive large folios where count should be zero +*/ +VM_WARN_ON (usage ==1 && nr >1 &&swap_count (count));/* * swapin_readahead() doesn't check if a swap entry is valid, so the * swap entry could be SWAP_MAP_BAD. Check here with lock held. @@ -3626,11 +3630,11 @@ void swap_shmem_alloc(swp_entry_t entry, int nr)* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which * might occur if a page table entry has got corrupted. */ -int swap_duplicate (swp_entry_t entry) +int swap_duplicate (swp_entry_t entry,int nr) {int err =0 ; -while (!err &&__swap_duplicate (entry,1 ,1 ) == -ENOMEM ) +while (!err &&__swap_duplicate (entry,1 , nr) == -ENOMEM ) err =add_swap_count_continuation (entry,GFP_ATOMIC );return err; }
Quoted 121 lines · Dev Jain
Quoted 117 lines · Barry Song
From: Barry Song <[email protected]>
My commit
354dffd29575c("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.Handling
__folio_try_share_anon_rmap()andswap_duplicate()becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.SPARC is currently the only architecture that implements
arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <string.h> #include <time.h> #include <unistd.h> #include <errno.h> #define SIZE_MB 256 #define SIZE_BYTES (SIZE_MB * 1024 * 1024) int main() { void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed"); return 1; }memset(addr, 0, SIZE_BYTES); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) { perror("madvise(MADV_PAGEOUT) failed"); munmap(addr, SIZE_BYTES); return 1; } clock_gettime(CLOCK_MONOTONIC, &end); long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec); printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n", duration_ns, duration_ns / 1e6); munmap(addr, SIZE_BYTES); return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1347834992 ns (1347.835 ms)w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 690790000 ns (690.790 ms)We found that the time to reclaim this memory was reduced by half.
Cc David Hildenbrand <[email protected]>Cc Baolin Wang <[email protected]>Cc Ryan Roberts <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Liam R. Howlett <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Harry Yoo <[email protected]>Cc Kairui Song <[email protected]>Cc Chris Li <[email protected]>Cc Baoquan He <[email protected]>Cc Dan Schatzberg <[email protected]>Cc Kaixiong Yu <[email protected]>Cc Fan Ni <[email protected]>Cc Tangquan Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 4 +-- mm/memory.c | 2 +- mm/rmap.c | 79 +++++++++++++++++++++++++++++--------------- mm/swapfile.c | 10 ++++-- 4 files changed, 62 insertions(+), 33 deletions(-)
Quoted 12 lines
include/linux/swap.h void put_swap_folio(struct folio *folio, swp_entry_t entry); · line 482
Quoted 10 lines
include/linux/swap.h +1
-1@@ -479,7 +479,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int add_swap_count_continuation (swp_entry_t ,gfp_t );extern void swap_shmem_alloc (swp_entry_t ,int ); -extern int swap_duplicate (swp_entry_t ); +extern int swap_duplicate (swp_entry_t ,int nr);Missed writing the parameter name here.
Thanks — I noticed this as well after sending. But compilation went through without
any issues.
Quoted 189 lines
Quoted 189 lines
include/linux/swap.h +1
-1extern int swapcache_prepare (swp_entry_t entry,int nr);extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -546,7 +546,7 @@ static inline void swap_shmem_alloc(swp_entry_t swp, int nr) { } -static inline int swap_duplicate (swp_entry_t swp) +static inline int swap_duplicate (swp_entry_t swp,int nr) {return 0 ; }mm/memory.c +1
-1@@ -803,7 +803,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,swp_entry_t entry =pte_to_swp_entry (orig_pte);if (likely (!non_swap_entry (entry))) { -if (swap_duplicate (entry) <0 ) +if (swap_duplicate (entry,1 ) <0 )return -EIO ;/* make sure dst_mm is on swapoff's mmlist. */ mm/rmap.c +52
-27@@ -1845,23 +1845,42 @@ void folio_remove_rmap_pud(struct folio *folio, struct page *page,#endif } -/* We support batch unmapping of PTEs for lazyfree large folios */ +/* +* We support batch unmapping of PTEs for lazyfree or exclusive anon large +* folios +*/ static inline bool can_batch_unmap_folio_ptes (unsigned long addr, -struct folio *folio,pte_t *ptep) +struct folio *folio,pte_t *ptep,bool exclusive) {const fpb_t fpb_flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;int max_nr =folio_nr_pages (folio); +#ifndef __HAVE_ARCH_UNMAP_ONE +bool no_arch_unmap = true; +#else +bool no_arch_unmap = false; +#endif pte_t pte =ptep_get (ptep); +int mapped_nr; -if (!folio_test_anon (folio) ||folio_test_swapbacked (folio)) +if (!folio_test_anon (folio))return false;if (pte_unused (pte))return false;if (pte_pfn (pte) !=folio_pfn (folio))return false; -return folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , -NULL ,NULL ) == max_nr; + mapped_nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , +NULL ,NULL ); +if (mapped_nr != max_nr) +return false; +if (!folio_test_swapbacked (folio)) +return true; + +/* +* The large folio is fully mapped and its mapcount is the same as its +* number of pages, it must be exclusive. +*/ +return no_arch_unmap && exclusive &&folio_mapcount (folio) == max_nr; }/* @@ -2025,7 +2044,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,folio_mark_dirty (folio); }else if (likely (pte_present (pteval))) {if (folio_test_large (folio) && !(flags &TTU_HWPOISON ) && -can_batch_unmap_folio_ptes (address, folio, pvmw.pte)) +can_batch_unmap_folio_ptes (address, folio, pvmw.pte, + anon_exclusive)) nr_pages =folio_nr_pages (folio); end_addr = address + nr_pages *PAGE_SIZE ;flush_cache_range (vma, address, end_addr);@@ -2141,8 +2161,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto discard; } -if (swap_duplicate (entry) <0 ) { -set_pte_at (mm, address, pvmw.pte, pteval); +if (swap_duplicate (entry, nr_pages) <0 ) { +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }@@ -2159,9 +2179,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && -folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); -set_pte_at (mm, address, pvmw.pte, pteval); +__folio_try_share_anon_rmap (folio, subpage, nr_pages, +RMAP_LEVEL_PTE )) { +swap_free_nr (entry, nr_pages); +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }if (list_empty (&mm->mmlist)) {@@ -2170,23 +2191,27 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,list_add (&mm->mmlist, &init_mm.mmlist);spin_unlock (&mmlist_lock); } -dec_mm_counter (mm,MM_ANONPAGES ); -inc_mm_counter (mm,MM_SWAPENTS ); - swp_pte =swp_entry_to_pte (entry); -if (anon_exclusive) - swp_pte =pte_swp_mkexclusive (swp_pte); -if (likely (pte_present (pteval))) { -if (pte_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); - }else { -if (pte_swp_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_swp_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); +add_mm_counter (mm,MM_ANONPAGES , -nr_pages); +add_mm_counter (mm,MM_SWAPENTS , nr_pages); +/* TODO: let set_ptes() support swp_offset advance */ +for (pte_t *ptep = pvmw.pte; address < end_addr; + entry.val++, address +=PAGE_SIZE , ptep++) { + swp_pte =swp_entry_to_pte (entry); +if (anon_exclusive) + swp_pte =pte_swp_mkexclusive (swp_pte); +if (likely (pte_present (pteval))) { +if (pte_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + }else { +if (pte_swp_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_swp_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + } +set_pte_at (mm, address, ptep, swp_pte); } -set_pte_at (mm, address, pvmw.pte, swp_pte); }else {/* * This is a locked file-backed folio, mm/swapfile.c +6
-2@@ -3550,13 +3550,17 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); -VM_WARN_ON (usage ==1 && nr >1 ); ci =lock_cluster (si, offset); err =0 ;for (i =0 ; i < nr; i++) { count = si->swap_map[offset + i]; +/* +* We only support batched swap_duplicate() for unmapping +* exclusive large folios where count should be zero +*/ +VM_WARN_ON (usage ==1 && nr >1 &&swap_count (count));/* * swapin_readahead() doesn't check if a swap entry is valid, so the * swap entry could be SWAP_MAP_BAD. Check here with lock held. @@ -3626,11 +3630,11 @@ void swap_shmem_alloc(swp_entry_t entry, int nr)* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which * might occur if a page table entry has got corrupted. */ -int swap_duplicate (swp_entry_t entry) +int swap_duplicate (swp_entry_t entry,int nr) {int err =0 ;
Quoted 6 lines
mm/swapfile.c void swap_shmem_alloc(swp_entry_t entry, int nr) · line 3640
Quoted 5 lines
mm/swapfile.c +1
-1-while (!err &&__swap_duplicate (entry,1 ,1 ) == -ENOMEM ) +while (!err &&__swap_duplicate (entry,1 , nr) == -ENOMEM ) err =add_swap_count_continuation (entry,GFP_ATOMIC );return err; }
Best Regards
Barry
Quoted 89 lines · Barry Song
From: Barry Song <[email protected]>
My commit
354dffd29575c("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.Handling
__folio_try_share_anon_rmap()andswap_duplicate()becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.SPARC is currently the only architecture that implements
arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <string.h> #include <time.h> #include <unistd.h> #include <errno.h> #define SIZE_MB 256 #define SIZE_BYTES (SIZE_MB * 1024 * 1024) int main() { void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed"); return 1; }memset(addr, 0, SIZE_BYTES); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) { perror("madvise(MADV_PAGEOUT) failed"); munmap(addr, SIZE_BYTES); return 1; } clock_gettime(CLOCK_MONOTONIC, &end); long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec); printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n", duration_ns, duration_ns / 1e6); munmap(addr, SIZE_BYTES); return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1347834992 ns (1347.835 ms)w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 690790000 ns (690.790 ms)
Quoted 1 line
We found that the time to reclaim this memory was reduced by half.
Do you have some performance numbers for the base page?
Quoted 202 lines
Cc David Hildenbrand <[email protected]>Cc Baolin Wang <[email protected]>Cc Ryan Roberts <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Liam R. Howlett <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Harry Yoo <[email protected]>Cc Kairui Song <[email protected]>Cc Chris Li <[email protected]>Cc Baoquan He <[email protected]>Cc Dan Schatzberg <[email protected]>Cc Kaixiong Yu <[email protected]>Cc Fan Ni <[email protected]>Cc Tangquan Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 4 +-- mm/memory.c | 2 +- mm/rmap.c | 79 +++++++++++++++++++++++++++++--------------- mm/swapfile.c | 10 ++++-- 4 files changed, 62 insertions(+), 33 deletions(-)include/linux/swap.h +2
-2@@ -479,7 +479,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int add_swap_count_continuation (swp_entry_t ,gfp_t );extern void swap_shmem_alloc (swp_entry_t ,int ); -extern int swap_duplicate (swp_entry_t ); +extern int swap_duplicate (swp_entry_t ,int nr);extern int swapcache_prepare (swp_entry_t entry,int nr);extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -546,7 +546,7 @@ static inline void swap_shmem_alloc(swp_entry_t swp, int nr) { } -static inline int swap_duplicate (swp_entry_t swp) +static inline int swap_duplicate (swp_entry_t swp,int nr) {return 0 ; }mm/memory.c +1
-1@@ -803,7 +803,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,swp_entry_t entry =pte_to_swp_entry (orig_pte);if (likely (!non_swap_entry (entry))) { -if (swap_duplicate (entry) <0 ) +if (swap_duplicate (entry,1 ) <0 )return -EIO ;/* make sure dst_mm is on swapoff's mmlist. */ mm/rmap.c +52
-27@@ -1845,23 +1845,42 @@ void folio_remove_rmap_pud(struct folio *folio, struct page *page,#endif } -/* We support batch unmapping of PTEs for lazyfree large folios */ +/* +* We support batch unmapping of PTEs for lazyfree or exclusive anon large +* folios +*/ static inline bool can_batch_unmap_folio_ptes (unsigned long addr, -struct folio *folio,pte_t *ptep) +struct folio *folio,pte_t *ptep,bool exclusive) {const fpb_t fpb_flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;int max_nr =folio_nr_pages (folio); +#ifndef __HAVE_ARCH_UNMAP_ONE +bool no_arch_unmap = true; +#else +bool no_arch_unmap = false; +#endif pte_t pte =ptep_get (ptep); +int mapped_nr; -if (!folio_test_anon (folio) ||folio_test_swapbacked (folio)) +if (!folio_test_anon (folio))return false;if (pte_unused (pte))return false;if (pte_pfn (pte) !=folio_pfn (folio))return false; -return folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , -NULL ,NULL ) == max_nr; + mapped_nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , +NULL ,NULL ); +if (mapped_nr != max_nr) +return false; +if (!folio_test_swapbacked (folio)) +return true; + +/* +* The large folio is fully mapped and its mapcount is the same as its +* number of pages, it must be exclusive. +*/ +return no_arch_unmap && exclusive &&folio_mapcount (folio) == max_nr; }/* @@ -2025,7 +2044,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,folio_mark_dirty (folio); }else if (likely (pte_present (pteval))) {if (folio_test_large (folio) && !(flags &TTU_HWPOISON ) && -can_batch_unmap_folio_ptes (address, folio, pvmw.pte)) +can_batch_unmap_folio_ptes (address, folio, pvmw.pte, + anon_exclusive)) nr_pages =folio_nr_pages (folio); end_addr = address + nr_pages *PAGE_SIZE ;flush_cache_range (vma, address, end_addr);@@ -2141,8 +2161,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto discard; } -if (swap_duplicate (entry) <0 ) { -set_pte_at (mm, address, pvmw.pte, pteval); +if (swap_duplicate (entry, nr_pages) <0 ) { +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }@@ -2159,9 +2179,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && -folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); -set_pte_at (mm, address, pvmw.pte, pteval); +__folio_try_share_anon_rmap (folio, subpage, nr_pages, +RMAP_LEVEL_PTE )) { +swap_free_nr (entry, nr_pages); +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }if (list_empty (&mm->mmlist)) {@@ -2170,23 +2191,27 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,list_add (&mm->mmlist, &init_mm.mmlist);spin_unlock (&mmlist_lock); } -dec_mm_counter (mm,MM_ANONPAGES ); -inc_mm_counter (mm,MM_SWAPENTS ); - swp_pte =swp_entry_to_pte (entry); -if (anon_exclusive) - swp_pte =pte_swp_mkexclusive (swp_pte); -if (likely (pte_present (pteval))) { -if (pte_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); - }else { -if (pte_swp_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_swp_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); +add_mm_counter (mm,MM_ANONPAGES , -nr_pages); +add_mm_counter (mm,MM_SWAPENTS , nr_pages); +/* TODO: let set_ptes() support swp_offset advance */ +for (pte_t *ptep = pvmw.pte; address < end_addr; + entry.val++, address +=PAGE_SIZE , ptep++) { + swp_pte =swp_entry_to_pte (entry); +if (anon_exclusive) + swp_pte =pte_swp_mkexclusive (swp_pte); +if (likely (pte_present (pteval))) { +if (pte_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + }else { +if (pte_swp_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_swp_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + } +set_pte_at (mm, address, ptep, swp_pte); } -set_pte_at (mm, address, pvmw.pte, swp_pte); }else {/* * This is a locked file-backed folio, mm/swapfile.c +0
-1@@ -3550,13 +3550,17 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); -VM_WARN_ON (usage ==1 && nr >1 ); ci =lock_cluster (si, offset);
mm/swapfile.c static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) · line 3563
Quoted 10 lines
mm/swapfile.c +5
-0err =0 ;for (i =0 ; i < nr; i++) { count = si->swap_map[offset + i]; +/* +* We only support batched swap_duplicate() for unmapping +* exclusive large folios where count should be zero +*/ +VM_WARN_ON (usage ==1 && nr >1 &&swap_count (count));
I think we should return an error instead of a warning. Although we triggered this warning (which might not be reported in an actual product), the program will continue to run and potentially cause more serious consequences, because the add_swap_count_continuation() needs to know which offset in the large swap entry to perform recovery.
Quoted 15 lines
mm/swapfile.c +2
-2/* * swapin_readahead() doesn't check if a swap entry is valid, so the * swap entry could be SWAP_MAP_BAD. Check here with lock held. @@ -3626,11 +3630,11 @@ void swap_shmem_alloc(swp_entry_t entry, int nr)* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which * might occur if a page table entry has got corrupted. */ -int swap_duplicate (swp_entry_t entry) +int swap_duplicate (swp_entry_t entry,int nr) {int err =0 ; -while (!err &&__swap_duplicate (entry,1 ,1 ) == -ENOMEM ) +while (!err &&__swap_duplicate (entry,1 , nr) == -ENOMEM ) err =add_swap_count_continuation (entry,GFP_ATOMIC );
Quoted 93 lines · Baolin Wang
Quoted 89 lines · Barry Song
From: Barry Song <[email protected]>
My commit
354dffd29575c("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.Handling
__folio_try_share_anon_rmap()andswap_duplicate()becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.SPARC is currently the only architecture that implements
arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <string.h> #include <time.h> #include <unistd.h> #include <errno.h> #define SIZE_MB 256 #define SIZE_BYTES (SIZE_MB * 1024 * 1024) int main() { void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed"); return 1; }memset(addr, 0, SIZE_BYTES); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) { perror("madvise(MADV_PAGEOUT) failed"); munmap(addr, SIZE_BYTES); return 1; } clock_gettime(CLOCK_MONOTONIC, &end); long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec); printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n", duration_ns, duration_ns / 1e6); munmap(addr, SIZE_BYTES); return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1347834992 ns (1347.835 ms)w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 690790000 ns (690.790 ms)
Quoted 3 lines
Quoted 1 line
We found that the time to reclaim this memory was reduced by half.
Do you have some performance numbers for the base page?
We verified that folio_test_large(folio) needs to run in a batched context; otherwise, nr_pages remains 1 for each folio.
if (folio_test_large(folio) && !(flags &
TTU_HWPOISON) &&
can_batch_unmap_folio_ptes(address, folio, pvmw.pte, anon_exclusive))
nr_pages = folio_nr_pages(folio);
I didn't expect any noticeable performance change for base pages, but testing shows the patch appears to make them slightly faster—likely due to test noise or jitter.
W/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT) took 5686488000 ns (5686.488 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5628330992 ns (5628.331 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5771742992 ns (5771.743 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5672108000 ns (5672.108 ms)
W/ patch:
~ # ./a.outmadvise(MADV_PAGEOUT) took 5481578000 ns (5481.578 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5425394992 ns (5425.395 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5522109008 ns (5522.109 ms) ~ # ./a.outmadvise(MADV_PAGEOUT) took 5506832000 ns (5506.832 ms)
Quoted 208 lines
Quoted 208 lines
Cc David Hildenbrand <[email protected]>Cc Baolin Wang <[email protected]>Cc Ryan Roberts <[email protected]>Cc Lorenzo Stoakes <[email protected]>Cc Liam R. Howlett <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Mike Rapoport <[email protected]>Cc Suren Baghdasaryan <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Harry Yoo <[email protected]>Cc Kairui Song <[email protected]>Cc Chris Li <[email protected]>Cc Baoquan He <[email protected]>Cc Dan Schatzberg <[email protected]>Cc Kaixiong Yu <[email protected]>Cc Fan Ni <[email protected]>Cc Tangquan Zheng <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 4 +-- mm/memory.c | 2 +- mm/rmap.c | 79 +++++++++++++++++++++++++++++--------------- mm/swapfile.c | 10 ++++-- 4 files changed, 62 insertions(+), 33 deletions(-)include/linux/swap.h +2
-2@@ -479,7 +479,7 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry);extern swp_entry_t get_swap_page_of_type (int );extern int add_swap_count_continuation (swp_entry_t ,gfp_t );extern void swap_shmem_alloc (swp_entry_t ,int ); -extern int swap_duplicate (swp_entry_t ); +extern int swap_duplicate (swp_entry_t ,int nr);extern int swapcache_prepare (swp_entry_t entry,int nr);extern void swap_free_nr (swp_entry_t entry,int nr_pages);extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);@@ -546,7 +546,7 @@ static inline void swap_shmem_alloc(swp_entry_t swp, int nr) { } -static inline int swap_duplicate (swp_entry_t swp) +static inline int swap_duplicate (swp_entry_t swp,int nr) {return 0 ; }mm/memory.c +1
-1@@ -803,7 +803,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,swp_entry_t entry =pte_to_swp_entry (orig_pte);if (likely (!non_swap_entry (entry))) { -if (swap_duplicate (entry) <0 ) +if (swap_duplicate (entry,1 ) <0 )return -EIO ;/* make sure dst_mm is on swapoff's mmlist. */ mm/rmap.c +52
-27@@ -1845,23 +1845,42 @@ void folio_remove_rmap_pud(struct folio *folio, struct page *page,#endif } -/* We support batch unmapping of PTEs for lazyfree large folios */ +/* +* We support batch unmapping of PTEs for lazyfree or exclusive anon large +* folios +*/ static inline bool can_batch_unmap_folio_ptes (unsigned long addr, -struct folio *folio,pte_t *ptep) +struct folio *folio,pte_t *ptep,bool exclusive) {const fpb_t fpb_flags =FPB_IGNORE_DIRTY |FPB_IGNORE_SOFT_DIRTY ;int max_nr =folio_nr_pages (folio); +#ifndef __HAVE_ARCH_UNMAP_ONE +bool no_arch_unmap = true; +#else +bool no_arch_unmap = false; +#endif pte_t pte =ptep_get (ptep); +int mapped_nr; -if (!folio_test_anon (folio) ||folio_test_swapbacked (folio)) +if (!folio_test_anon (folio))return false;if (pte_unused (pte))return false;if (pte_pfn (pte) !=folio_pfn (folio))return false; -return folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , -NULL ,NULL ) == max_nr; + mapped_nr =folio_pte_batch (folio, addr, ptep, pte, max_nr, fpb_flags,NULL , +NULL ,NULL ); +if (mapped_nr != max_nr) +return false; +if (!folio_test_swapbacked (folio)) +return true; + +/* +* The large folio is fully mapped and its mapcount is the same as its +* number of pages, it must be exclusive. +*/ +return no_arch_unmap && exclusive &&folio_mapcount (folio) == max_nr; }/* @@ -2025,7 +2044,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,folio_mark_dirty (folio); }else if (likely (pte_present (pteval))) {if (folio_test_large (folio) && !(flags &TTU_HWPOISON ) && -can_batch_unmap_folio_ptes (address, folio, pvmw.pte)) +can_batch_unmap_folio_ptes (address, folio, pvmw.pte, + anon_exclusive)) nr_pages =folio_nr_pages (folio); end_addr = address + nr_pages *PAGE_SIZE ;flush_cache_range (vma, address, end_addr);@@ -2141,8 +2161,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, goto discard; } -if (swap_duplicate (entry) <0 ) { -set_pte_at (mm, address, pvmw.pte, pteval); +if (swap_duplicate (entry, nr_pages) <0 ) { +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }@@ -2159,9 +2179,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && -folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); -set_pte_at (mm, address, pvmw.pte, pteval); +__folio_try_share_anon_rmap (folio, subpage, nr_pages, +RMAP_LEVEL_PTE )) { +swap_free_nr (entry, nr_pages); +set_ptes (mm, address, pvmw.pte, pteval, nr_pages); goto walk_abort; }if (list_empty (&mm->mmlist)) {@@ -2170,23 +2191,27 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,list_add (&mm->mmlist, &init_mm.mmlist);spin_unlock (&mmlist_lock); } -dec_mm_counter (mm,MM_ANONPAGES ); -inc_mm_counter (mm,MM_SWAPENTS ); - swp_pte =swp_entry_to_pte (entry); -if (anon_exclusive) - swp_pte =pte_swp_mkexclusive (swp_pte); -if (likely (pte_present (pteval))) { -if (pte_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); - }else { -if (pte_swp_soft_dirty (pteval)) - swp_pte =pte_swp_mksoft_dirty (swp_pte); -if (pte_swp_uffd_wp (pteval)) - swp_pte =pte_swp_mkuffd_wp (swp_pte); +add_mm_counter (mm,MM_ANONPAGES , -nr_pages); +add_mm_counter (mm,MM_SWAPENTS , nr_pages); +/* TODO: let set_ptes() support swp_offset advance */ +for (pte_t *ptep = pvmw.pte; address < end_addr; + entry.val++, address +=PAGE_SIZE , ptep++) { + swp_pte =swp_entry_to_pte (entry); +if (anon_exclusive) + swp_pte =pte_swp_mkexclusive (swp_pte); +if (likely (pte_present (pteval))) { +if (pte_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + }else { +if (pte_swp_soft_dirty (pteval)) + swp_pte =pte_swp_mksoft_dirty (swp_pte); +if (pte_swp_uffd_wp (pteval)) + swp_pte =pte_swp_mkuffd_wp (swp_pte); + } +set_pte_at (mm, address, ptep, swp_pte); } -set_pte_at (mm, address, pvmw.pte, swp_pte); }else {/* * This is a locked file-backed folio, mm/swapfile.c +0
-1@@ -3550,13 +3550,17 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset =swp_offset (entry);VM_WARN_ON (nr >SWAPFILE_CLUSTER - offset %SWAPFILE_CLUSTER ); -VM_WARN_ON (usage ==1 && nr >1 ); ci =lock_cluster (si, offset); err =0 ;for (i =0 ; i < nr; i++) { count = si->swap_map[offset + i];
Quoted 11 lines
mm/swapfile.c static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) · line 3563
Quoted 5 lines
mm/swapfile.c +5
-0+/* +* We only support batched swap_duplicate() for unmapping +* exclusive large folios where count should be zero +*/ +VM_WARN_ON (usage ==1 && nr >1 &&swap_count (count));I think we should return an error instead of a warning. Although we triggered this warning (which might not be reported in an actual product), the program will continue to run and potentially cause more serious consequences, because the
add_swap_count_continuation()needs to know which offset in the large swap entry to perform recovery.
Sounds like a good idea to me. We're currently doing for (i = 0; i < nr; i++) twice:
- First pass: check if it's safe to proceed;
- Second pass: actually perform the work.
As long as we return an error code other than ENOMEM, this approach should be fine.
Quoted 13 lines
Quoted 13 lines
mm/swapfile.c +1
-1/* * swapin_readahead() doesn't check if a swap entry is valid, so the * swap entry could be SWAP_MAP_BAD. Check here with lock held. @@ -3626,11 +3630,11 @@ void swap_shmem_alloc(swp_entry_t entry, int nr)* if __swap_duplicate() fails for another reason (-EINVAL or -ENOENT), which * might occur if a page table entry has got corrupted. */ -int swap_duplicate (swp_entry_t entry) +int swap_duplicate (swp_entry_t entry,int nr) {int err =0 ;
Quoted 3 lines
mm/swapfile.c void swap_shmem_alloc(swp_entry_t entry, int nr) · line 3638
Quoted 3 lines
mm/swapfile.c +1
-1-while (!err &&__swap_duplicate (entry,1 ,1 ) == -ENOMEM ) +while (!err &&__swap_duplicate (entry,1 , nr) == -ENOMEM ) err =add_swap_count_continuation (entry,GFP_ATOMIC );
thanks
Barry
Quoted 126 lines · Barry Song
Quoted 93 lines · Baolin Wang
Quoted 89 lines · Barry Song
From: Barry Song <[email protected]>
My commit
354dffd29575c("mm: support batched unmap for lazyfree large folios during reclamation") introduced support for unmapping entire lazyfree anonymous large folios at once, instead of one page at a time. This patch extends that support to generic (non-lazyfree) anonymous large folios.Handling
__folio_try_share_anon_rmap()andswap_duplicate()becomes extremely complex—if not outright impractical—for non-exclusive anonymous folios. As a result, this patch limits support to exclusive large folios. Fortunately, most anonymous folios are exclusive in practice, so this restriction should be acceptable in the majority of cases.SPARC is currently the only architecture that implements
arch_unmap_one(), which also needs to be batched for consistency. However, this is not yet supported, so the platform is excluded for now.Using the following micro-benchmark to measure the time taken to perform PAGEOUT on 256MB of 64KiB anonymous large folios.
#define _GNU_SOURCE #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <string.h> #include <time.h> #include <unistd.h> #include <errno.h> #define SIZE_MB 256 #define SIZE_BYTES (SIZE_MB * 1024 * 1024) int main() { void *addr = mmap(NULL, SIZE_BYTES, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap failed"); return 1; }memset(addr, 0, SIZE_BYTES); struct timespec start, end; clock_gettime(CLOCK_MONOTONIC, &start); if (madvise(addr, SIZE_BYTES, MADV_PAGEOUT) != 0) { perror("madvise(MADV_PAGEOUT) failed"); munmap(addr, SIZE_BYTES); return 1; } clock_gettime(CLOCK_MONOTONIC, &end); long duration_ns = (end.tv_sec - start.tv_sec) * 1e9 + (end.tv_nsec - start.tv_nsec); printf("madvise(MADV_PAGEOUT) took %ld ns (%.3f ms)\n", duration_ns, duration_ns / 1e6); munmap(addr, SIZE_BYTES); return 0;}
w/o patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 1337334000 ns (1337.334 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1340471008 ns (1340.471 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1385718992 ns (1385.719 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1366070000 ns (1366.070 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 1347834992 ns (1347.835 ms)w/patch:
~ # ./a.outmadvise(MADV_PAGEOUT)took 698178000 ns (698.178 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 708570000 ns (708.570 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693884000 ns (693.884 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 693366000 ns (693.366 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 690790000 ns (690.790 ms)Quoted 3 lines
Quoted 1 line
We found that the time to reclaim this memory was reduced by half.
Do you have some performance numbers for the base page?
We verified that
folio_test_large(folio)needs to run in a batched context; otherwise,nr_pagesremains 1 for each folio.if (folio_test_large(folio) && !(flags &
TTU_HWPOISON) &&
can_batch_unmap_folio_ptes(address, folio, pvmw.pte, anon_exclusive))
nr_pages=folio_nr_pages(folio);I didn't expect any noticeable performance change for base pages, but testing shows the patch appears to make them slightly faster—likely due to test noise or jitter.
W/o patch:
~ # ./a.out
madvise(MADV_PAGEOUT)took 5686488000 ns (5686.488 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5628330992 ns (5628.331 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5771742992 ns (5771.743 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5672108000 ns (5672.108 ms)W/ patch:
Quoted 8 lines
~ # ./a.out
madvise(MADV_PAGEOUT)took 5481578000 ns (5481.578 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5425394992 ns (5425.395 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5522109008 ns (5522.109 ms) ~ # ./a.outmadvise(MADV_PAGEOUT)took 5506832000 ns (5506.832 ms)
Thanks. My expectation is also that the batch processing of large folios should not have a performance impact on the base pages, but it would be best to clearly state this in the commit message.
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2183
Quoted 9 lines
mm/rmap.c +2
-3@@ -2159,9 +2179,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,/* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && -folio_try_share_anon_rmap_pte (folio, subpage)) { -swap_free (entry); -set_pte_at (mm, address, pvmw.pte, pteval); +__folio_try_share_anon_rmap (folio, subpage, nr_pages, +RMAP_LEVEL_PTE )) {
I have a question for David regarding the __folio_try_share_anon_rmap() function. I noticed that it only clears the Exclusive flag for the first subpage, even when nr_pages > 1. Shouldn't we be clearing the Exclusive flag for all subpages in this case? Could this be a potential bug?
Thanks
Barry