Swap-out mTHP without splitting
Subsystems (7)
Ryan Roberts
reviewerv4 cover Swap-out mTHP without splitting
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.
There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================
I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run usemem from vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
This test also acts as a good stress test for swap and, more generally mm. A couple of existing bugs were found as a result [5] [6].
---
The series applies against mm-unstable (d7182786dd0a). Although I've additionally been running with a couple of extra fixes to avoid the issues at [6].
Changes since v3 [3]
====================
- Renamed SWAP_NEXT_NULL -> SWAP_NEXT_INVALID (per Huang, Ying)
- Simplified max offset calculation (per Huang, Ying)
- Reinstated struct percpu_cluster to contain per-cluster, per-order next offset (per Huang, Ying)
- Removed swap_alloc_large() and merged its functionality into scan_swap_map_slots() (per Huang, Ying)
- Avoid extra cost of folio ref and lock due to removal of CLUSTER_FLAG_HUGE by freeing swap entries in batches (see patch 2) (per DavidH)
- vmscan splits folio if its partially mapped (per Barry Song, DavidH)
- Avoid splitting in MADV_PAGEOUT path (per Barry Song)
- Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch since it's not actually a problem for THP as I first thought.
Changes since v2 [2]
====================
- Reuse scan_swap_map_try_ssd_cluster() between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug where nr_swap_pages would say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).
Changes since v1 [1]
====================
- patch 1:
- Use cluster_set_count() instead of cluster_set_count_flag() in swap_alloc_cluster() since we no longer have any flag to set. I was unable to kill cluster_set_count_flag() as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array into percpu_cluster to make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated because PMD_ORDER is not compile-time constant for powerpc (fixes build error).
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
Ryan Roberts (6):
mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache()
mm: swap: Simplify struct percpu_cluster
mm: swap: Allow storage of all mTHP orders
mm: vmscan: Avoid split during shrink_folio_list()
mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
include/linux/pgtable.h | 28 ++++ include/linux/swap.h | 33 +++-- mm/huge_memory.c | 3 - mm/internal.h | 48 +++++++ mm/madvise.c | 101 ++++++++------ mm/memory.c | 13 +- mm/swapfile.c | 298 ++++++++++++++++++++++------------------ mm/vmscan.c | 9 +- 8 files changed, 332 insertions(+), 201 deletions(-)
--
2.25.1
Ryan Roberts
reviewerPATCH v4 1/6 mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.
The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass in nr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.
The one snag is that one of the swap_page_trans_huge_swapped() call sites does not have the folio. But it was only being called there to shortcut a call __try_to_reclaim_swap() in some cases. __try_to_reclaim_swap() gets the folio and (via some other functions) calls swap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be functionally equivalent.
That said, removing the fast path means that we will take a reference and trylock a large folio much more often, which we would like to avoid. The next patch will solve this.
Removing CLUSTER_FLAG_HUGE also means we can remove split_swap_cluster() which used to be called during folio splitting, since split_swap_cluster()'s only job was to remove the flag.
--- include/linux/swap.h | 10 ---------- mm/huge_memory.c | 3 --- mm/swapfile.c | 47 ++++++++------------------------------------ 3 files changed, 8 insertions(+), 52 deletions(-)
include/linux/swap.h +0 -10
@@ -259,7 +259,6 @@ struct swap_cluster_info { };#define CLUSTER_FLAG_FREE 1/* This cluster is free */ #define CLUSTER_FLAG_NEXT_NULL 2/* This cluster has no next cluster */ -#define CLUSTER_FLAG_HUGE 4/* This cluster is backing a transparent huge page */ /* * We assign a cluster to each CPU, so each CPU can allocate swap entry from
@@ -600,15 +599,6 @@ static inline int add_swap_extent(struct swap_info_struct *sis, }#endif /* CONFIG_SWAP */ -#ifdef CONFIG_THP_SWAP -extern int split_swap_cluster (swp_entry_t entry); -#else -static inline int split_swap_cluster (swp_entry_t entry) -{ -return 0 ; -} -#endif -#ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness (struct mem_cgroup *memcg) {
mm/huge_memory.c +0 -3
@@ -2965,9 +2965,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,shmem_uncharge (folio->mapping->host, nr_dropped);remap_page (folio, nr); -if (folio_test_swapcache (folio)) -split_swap_cluster (folio->swap); -/* * set page to its compound_head when split to non order-0 pages, so * we can skip unlocking it below, since PG_locked is transferred to
mm/swapfile.c +8 -39
107 lines not on this page · raw message
Signature · 1 line
2.25.1
Ryan Roberts
reviewerPATCH v4 2/6 mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache()
In reply to Ryan RobertsNow that we no longer have a convenient flag in the cluster to determine if a folio is large, free_swap_and_cache() will take a reference and lock a large folio much more often, which could lead to contention and (e.g.) failure to split large folios, etc.
Let's solve that problem by batch freeing swap and cache with a new function, free_swap_and_cache_nr(), to free a contiguous range of swap entries together. This allows us to first drop a reference to each swap slot before we try to release the cache folio. This means we only try to release the folio once, only taking the reference and lock once - much better than the previous 512 times for the 2M THP case.
Contiguous swap entries are gathered in zap_pte_range() and madvise_free_pte_range() in a similar way to how present ptes are already gathered in zap_pte_range().
While we are at it, let's simplify by converting the return type of both functions to void. The return value was used only by zap_pte_range() to print a bad pte, and was ignored by everyone else, so the extra reporting wasn't exactly guaranteed. We will still get the warning with most of the information from get_swap_device(). With the batch version, we wouldn't know which pte was bad anyway so could print the wrong one.
--- include/linux/pgtable.h | 28 +++++++++++++++ include/linux/swap.h | 12 +++++-- mm/internal.h | 48 +++++++++++++++++++++++++ mm/madvise.c | 12 ++++--- mm/memory.c | 13 +++---- mm/swapfile.c | 78 ++++++++++++++++++++++++++++++----------- 6 files changed, 157 insertions(+), 34 deletions(-)
include/linux/pgtable.h +28 -0
@@ -708,6 +708,34 @@ static inline void pte_clear_not_present_full(struct mm_struct *mm, }#endif +#ifndef clear_not_present_full_ptes +/** +* clear_not_present_full_ptes - Clear consecutive not present PTEs. +* @mm: Address space the ptes represent. +* @addr: Address of the first pte. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries to clear. +* @full: Whether we are clearing a full mm. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over pte_clear_not_present_full(). +* +* Context: The caller holds the page table lock. The PTEs are all not present. +* The PTEs are all in the same PMD. +*/ +static inline void clear_not_present_full_ptes (struct mm_struct *mm, +unsigned long addr,pte_t *ptep,unsigned int nr,int full) +{ +for (;;) { +pte_clear_not_present_full (mm, addr, ptep, full); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + } +} +#endif +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH extern pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,
include/linux/swap.h +9 -3
@@ -481,7 +481,7 @@ extern int swap_duplicate(swp_entry_t);extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t );extern void swapcache_free_entries (swp_entry_t *entries,int n); -extern int free_swap_and_cache (swp_entry_t ); +extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);int find_first_swap (dev_t *device);extern unsigned int count_swap_pages (int ,int );
@@ -530,8 +530,9 @@ static inline void put_swap_device(struct swap_info_struct *si)#define free_pages_and_swap_cache (pages, nr) \ release_pages((pages), (nr)); -/* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ -#define free_swap_and_cache (e) is_pfn_swap_entry(e) +static inline void free_swap_and_cache_nr (swp_entry_t entry,int nr) +{ +}static inline void free_swap_cache (struct folio *folio) {
@@ -599,6 +600,11 @@ static inline int add_swap_extent(struct swap_info_struct *sis, }#endif /* CONFIG_SWAP */ +static inline void free_swap_and_cache (swp_entry_t entry) +{ +free_swap_and_cache_nr (entry,1 ); +} +#ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness (struct mem_cgroup *memcg) {
mm/internal.h +48 -0
60 lines not on this page · raw message
mm/madvise.c +8 -4
@@ -628,6 +628,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,struct folio *folio;int nr_swap =0 ;unsigned long next; +int nr, max_nr; next =pmd_addr_end (addr, end);if (pmd_trans_huge (*pmd))
@@ -640,7 +641,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr != end; pte++, addr +=PAGE_SIZE ) { +for (; addr != end; pte += nr, addr +=PAGE_SIZE * nr) { + nr =1 ; ptent =ptep_get (pte);if (pte_none (ptent))
@@ -655,9 +657,11 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, entry =pte_to_swp_entry (ptent);if (!non_swap_entry (entry)) { - nr_swap--; -free_swap_and_cache (entry); -pte_clear_not_present_full (mm, addr, pte, tlb->fullmm); + max_nr = (end - addr) /PAGE_SIZE ; + nr =swap_pte_batch (pte, max_nr, entry); + nr_swap -= nr; +free_swap_and_cache_nr (entry, nr); +clear_not_present_full_ptes (mm, addr, pte, nr, tlb->fullmm); }else if (is_hwpoison_entry (entry) ||is_poisoned_swp_entry (entry)) {pte_clear_not_present_full (mm, addr, pte, tlb->fullmm);
mm/memory.c +7 -6
@@ -1629,12 +1629,13 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,folio_remove_rmap_pte (folio, page, vma);folio_put (folio); }else if (!non_swap_entry (entry)) { -/* Genuine swap entry, hence a private anon page */ + max_nr = (end - addr) /PAGE_SIZE ; + nr =swap_pte_batch (pte, max_nr, entry); +/* Genuine swap entries, hence a private anon pages */ if (!should_zap_cows (details))continue ; - rss[MM_SWAPENTS ]--; -if (unlikely (!free_swap_and_cache (entry))) -print_bad_pte (vma, addr, ptent,NULL ); + rss[MM_SWAPENTS ] -= nr; +free_swap_and_cache_nr (entry, nr); }else if (is_migration_entry (entry)) { folio =pfn_swap_entry_folio (entry);if (!should_zap_folio (details, folio))
@@ -1657,8 +1658,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,pr_alert ("unrecognized swap entry 0x%lx\n" , entry.val);WARN_ON_ONCE (1 ); } -pte_clear_not_present_full (mm, addr, pte, tlb->fullmm); -zap_install_uffd_wp_if_needed (vma, addr, pte,1 , details, ptent); +clear_not_present_full_ptes (mm, addr, pte, nr, tlb->fullmm); +zap_install_uffd_wp_if_needed (vma, addr, pte, nr, details, ptent); }while (pte += nr, addr +=PAGE_SIZE * nr, addr != end);add_mm_rss_vec (mm, rss);
mm/swapfile.c +57 -21
109 lines not on this page · raw message
Signature · 1 line
2.25.1
Ryan Roberts
reviewerRe: [PATCH v4 2/6] mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache()
Hi David,
I hate to chase, but since you provided feedback on a couple of the other patches in the series, I wondered if you missed this one? It's the one that does the batching of free_swap_and_cache(), which you suggested in order to prevent needlessly taking folio locks and refs.
If you have any feedback, it would be appreciated, otherwise I'm planning to repost as-is next week (nobody else has posted comments against this patch either) as part of the updated series.
Thanks,
Ryan
Quoted 362 lines · Ryan Roberts
Now that we no longer have a convenient flag in the cluster to determine if a folio is large,
free_swap_and_cache()will take a reference and lock a large folio much more often, which could lead to contention and (e.g.) failure to split large folios, etc.Let's solve that problem by batch freeing swap and cache with a new function,
free_swap_and_cache_nr(), to free a contiguous range of swap entries together. This allows us to first drop a reference to each swap slot before we try to release the cache folio. This means we only try to release the folio once, only taking the reference and lock once - much better than the previous 512 times for the 2M THP case.Contiguous swap entries are gathered in
zap_pte_range()andmadvise_free_pte_range()in a similar way to how present ptes are already gathered inzap_pte_range().While we are at it, let's simplify by converting the return type of both functions to void. The return value was used only by
zap_pte_range()to print a bad pte, and was ignored by everyone else, so the extra reporting wasn't exactly guaranteed. We will still get the warning with most of the information fromget_swap_device(). With the batch version, we wouldn't know which pte was bad anyway so could print the wrong one.Signed-off-by Ryan Roberts <[email protected]>--- include/linux/pgtable.h | 28 +++++++++++++++ include/linux/swap.h | 12 +++++-- mm/internal.h | 48 +++++++++++++++++++++++++ mm/madvise.c | 12 ++++--- mm/memory.c | 13 +++---- mm/swapfile.c | 78 ++++++++++++++++++++++++++++++----------- 6 files changed, 157 insertions(+), 34 deletions(-)include/linux/pgtable.h +28
-0@@ -708,6 +708,34 @@ static inline void pte_clear_not_present_full(struct mm_struct *mm, }#endif +#ifndef clear_not_present_full_ptes +/** +* clear_not_present_full_ptes - Clear consecutive not present PTEs. +* @mm: Address space the ptes represent. +* @addr: Address of the first pte. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries to clear. +* @full: Whether we are clearing a full mm. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over pte_clear_not_present_full(). +* +* Context: The caller holds the page table lock. The PTEs are all not present. +* The PTEs are all in the same PMD. +*/ +static inline void clear_not_present_full_ptes (struct mm_struct *mm, +unsigned long addr,pte_t *ptep,unsigned int nr,int full) +{ +for (;;) { +pte_clear_not_present_full (mm, addr, ptep, full); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + } +} +#endif +#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSH extern pte_t ptep_clear_flush(struct vm_area_struct *vma, unsigned long address,include/linux/swap.h +9
-3@@ -481,7 +481,7 @@ extern int swap_duplicate(swp_entry_t);extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t );extern void swapcache_free_entries (swp_entry_t *entries,int n); -extern int free_swap_and_cache (swp_entry_t ); +extern void free_swap_and_cache_nr (swp_entry_t entry,int nr);int swap_type_of (dev_t device,sector_t offset);int find_first_swap (dev_t *device);extern unsigned int count_swap_pages (int ,int );@@ -530,8 +530,9 @@ static inline void put_swap_device(struct swap_info_struct *si)#define free_pages_and_swap_cache (pages, nr) \ release_pages((pages), (nr)); -/* used to sanity check ptes in zap_pte_range when CONFIG_SWAP=0 */ -#define free_swap_and_cache (e) is_pfn_swap_entry(e) +static inline void free_swap_and_cache_nr (swp_entry_t entry,int nr) +{ +}static inline void free_swap_cache (struct folio *folio) {@@ -599,6 +600,11 @@ static inline int add_swap_extent(struct swap_info_struct *sis, }#endif /* CONFIG_SWAP */ +static inline void free_swap_and_cache (swp_entry_t entry) +{ +free_swap_and_cache_nr (entry,1 ); +} +#ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness (struct mem_cgroup *memcg) {mm/internal.h +48
-0@@ -11,6 +11,8 @@#include <linux/mm.h> #include <linux/pagemap.h> #include <linux/rmap.h> +#include <linux/swap.h> +#include <linux/swapops.h> #include <linux/tracepoint-defs.h> struct folio_batch ;@@ -174,6 +176,52 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, return min(ptep - start_ptep, max_nr); } + +/** +* swap_pte_batch - detect a PTE batch for a set of contiguous swap entries +* @start_ptep: Page table pointer for the first entry. +* @max_nr: The maximum number of table entries to consider. +* @entry: Swap entry recovered from the first table entry. +* +* Detect a batch of contiguous swap entries: consecutive (non-present) PTEs +* containing swap entries all with consecutive offsets and targeting the same +* swap type. +* +* max_nr must be at least one and must be limited by the caller so scanning +* cannot exceed a single page table. +* +* Return: the number of table entries in the batch. +*/ +static inline int swap_pte_batch (pte_t *start_ptep,int max_nr, +swp_entry_t entry) +{ +const pte_t *end_ptep = start_ptep + max_nr; +unsigned long expected_offset =swp_offset (entry) +1 ; +unsigned int expected_type =swp_type (entry); +pte_t *ptep = start_ptep +1 ; + +VM_WARN_ON (max_nr <1 ); +VM_WARN_ON (non_swap_entry (entry)); + +while (ptep < end_ptep) { +pte_t pte =ptep_get (ptep); + +if (pte_none (pte) ||pte_present (pte)) +break ; + + entry =pte_to_swp_entry (pte); + +if (non_swap_entry (entry) || +swp_type (entry) != expected_type || +swp_offset (entry) != expected_offset) +break ; + + expected_offset++; + ptep++; + } + +return ptep - start_ptep; +}#endif /* CONFIG_MMU */ void __acct_reclaim_writeback(pg_data_t *pgdat, struct folio *folio,mm/madvise.c +8
-4@@ -628,6 +628,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,struct folio *folio;int nr_swap =0 ;unsigned long next; +int nr, max_nr; next =pmd_addr_end (addr, end);if (pmd_trans_huge (*pmd))@@ -640,7 +641,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr != end; pte++, addr +=PAGE_SIZE ) { +for (; addr != end; pte += nr, addr +=PAGE_SIZE * nr) { + nr =1 ; ptent =ptep_get (pte);if (pte_none (ptent))@@ -655,9 +657,11 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, entry =pte_to_swp_entry (ptent);if (!non_swap_entry (entry)) { - nr_swap--; -free_swap_and_cache (entry); -pte_clear_not_present_full (mm, addr, pte, tlb->fullmm); + max_nr = (end - addr) /PAGE_SIZE ; + nr =swap_pte_batch (pte, max_nr, entry); + nr_swap -= nr; +free_swap_and_cache_nr (entry, nr); +clear_not_present_full_ptes (mm, addr, pte, nr, tlb->fullmm); }else if (is_hwpoison_entry (entry) ||is_poisoned_swp_entry (entry)) {pte_clear_not_present_full (mm, addr, pte, tlb->fullmm);mm/memory.c +7
-6@@ -1629,12 +1629,13 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,folio_remove_rmap_pte (folio, page, vma);folio_put (folio); }else if (!non_swap_entry (entry)) { -/* Genuine swap entry, hence a private anon page */ + max_nr = (end - addr) /PAGE_SIZE ; + nr =swap_pte_batch (pte, max_nr, entry); +/* Genuine swap entries, hence a private anon pages */ if (!should_zap_cows (details))continue ; - rss[MM_SWAPENTS ]--; -if (unlikely (!free_swap_and_cache (entry))) -print_bad_pte (vma, addr, ptent,NULL ); + rss[MM_SWAPENTS ] -= nr; +free_swap_and_cache_nr (entry, nr); }else if (is_migration_entry (entry)) { folio =pfn_swap_entry_folio (entry);if (!should_zap_folio (details, folio))@@ -1657,8 +1658,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,pr_alert ("unrecognized swap entry 0x%lx\n" , entry.val);WARN_ON_ONCE (1 ); } -pte_clear_not_present_full (mm, addr, pte, tlb->fullmm); -zap_install_uffd_wp_if_needed (vma, addr, pte,1 , details, ptent); +clear_not_present_full_ptes (mm, addr, pte, nr, tlb->fullmm); +zap_install_uffd_wp_if_needed (vma, addr, pte, nr, details, ptent); }while (pte += nr, addr +=PAGE_SIZE * nr, addr != end);add_mm_rss_vec (mm, rss);mm/swapfile.c +57
-21@@ -130,7 +130,11 @@ static inline unsigned char swap_count(unsigned char ent)/* Reclaim the swap entry if swap is getting full*/ #define TTRS_FULL 0x4 -/* returns 1 if swap entry is freed */ +/* +* returns number of pages in the folio that backs the swap entry. If positive, +* the folio was reclaimed. If negative, the folio was not reclaimed. If 0, no +* folio was associated with the swap entry. +*/ static int __try_to_reclaim_swap (struct swap_info_struct *si,unsigned long offset,unsigned long flags) {@@ -155,6 +159,7 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, ret =folio_free_swap (folio);folio_unlock (folio); } + ret = ret ?folio_nr_pages (folio) : -folio_nr_pages (folio);folio_put (folio);return ret; }@@ -895,7 +900,7 @@ static int scan_swap_map_slots(struct swap_info_struct *si, swap_was_freed =__try_to_reclaim_swap (si, offset,TTRS_ANYWAY );spin_lock (&si->lock);/* entry was freed successfully, try to use this again */ -if (swap_was_freed) +if (swap_was_freed >0 ) goto checks; goto scan;/* check next one */ }@@ -1572,32 +1577,63 @@ bool folio_free_swap(struct folio *folio)return true; } -/* -* Free the swap entry like above, but also try to -* free the page cache entry if it is the last user. -*/ -int free_swap_and_cache (swp_entry_t entry) +void free_swap_and_cache_nr (swp_entry_t entry,int nr) { -struct swap_info_struct *p; -unsigned char count; +unsigned long end =swp_offset (entry) + nr; +unsigned int type =swp_type (entry); +struct swap_info_struct *si; +unsigned long offset;if (non_swap_entry (entry)) -return 1 ; +return ; - p =get_swap_device (entry); -if (p) { -if (WARN_ON (data_race (!p->swap_map[swp_offset (entry)]))) { -put_swap_device (p); -return 0 ; - } + si =get_swap_device (entry); +if (!si) +return ; - count =__swap_entry_free (p, entry); -if (count ==SWAP_HAS_CACHE ) -__try_to_reclaim_swap (p,swp_offset (entry), +if (WARN_ON (end > si->max)) + goto out; + +/* +* First free all entries in the range. +*/ +for (offset =swp_offset (entry); offset < end; offset++) { +if (!WARN_ON (data_race (!si->swap_map[offset]))) +__swap_entry_free (si,swp_entry (type, offset)); + } + +/* +* Now go back over the range trying to reclaim the swap cache. This is +* more efficient for large folios because we will only try to reclaim +* the swap once per folio in the common case. If we do +* __swap_entry_free() and __try_to_reclaim_swap() in the same loop, the +* latter will get a reference and lock the folio for every individual +* page but will only succeed once the swap slot for every subpage is +* zero. +*/ +for (offset =swp_offset (entry); offset < end; offset += nr) { + nr =1 ; +if (READ_ONCE (si->swap_map[offset]) ==SWAP_HAS_CACHE ) { +/* +* Folios are always naturally aligned in swap so +* advance forward to the next boundary. Zero means no +* folio was found for the swap entry, so advance by 1 +* in this case. Negative value means folio was found +* but could not be reclaimed. Here we can still advance +* to the next boundary. +*/ + nr =__try_to_reclaim_swap (si, offset,TTRS_UNMAPPED |TTRS_FULL ); -put_swap_device (p); +if (nr ==0 ) + nr =1 ; +else if (nr <0 ) + nr = -nr; + nr =ALIGN (offset +1 , nr) - offset; + } } -return p !=NULL ; + +out: +put_swap_device (si); }#ifdef CONFIG_HIBERNATION
Re: [PATCH v4 2/6] mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache()
Quoted 1 line · Ryan Roberts
Hi David,
I'm usually lazy with review during the merge window :P
Quoted 9 lines
I hate to chase, but since you provided feedback on a couple of the other patches in the series, I wondered if you missed this one? It's the one that does the batching of
free_swap_and_cache(), which you suggested in order to prevent needlessly taking folio locks and refs.If you have any feedback, it would be appreciated, otherwise I'm planning to repost as-is next week (nobody else has posted comments against this patch either) as part of the updated series.
On my TODO list!
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 2/6] mm: swap: free_swap_and_cache_nr() as batched free_swap_and_cache()
Quoted 4 lines · David Hildenbrand
Quoted 1 line · Ryan Roberts
Hi David,
I'm usually lazy with review during the merge window :P
Ahh no worries! I'll send out the latest version next week, and can go from there.
Quoted 13 lines
Quoted 10 lines
I hate to chase, but since you provided feedback on a couple of the other patches in the series, I wondered if you missed this one? It's the one that does the batching of
free_swap_and_cache(), which you suggested in order to prevent needlessly taking folio locks and refs.If you have any feedback, it would be appreciated, otherwise I'm planning to repost as-is next week (nobody else has posted comments against this patch either) as part of the updated series.
On my TODO list!
Ryan Roberts
reviewerPATCH v4 3/6 mm: swap: Simplify struct percpu_cluster
In reply to Ryan Robertsstruct percpu_cluster stores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset / SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies the spinlock_t structure, which is questionable coding practice in my view.
So let's clean this up and store only the next offset, and use a sentinal value (SWAP_NEXT_INVALID) to indicate "no cluster". SWAP_NEXT_INVALID is chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list.
This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system.
--- include/linux/swap.h | 9 ++++++++- mm/swapfile.c | 22 +++++++++++----------- 2 files changed, 19 insertions(+), 12 deletions(-)
include/linux/swap.h +8 -1
@@ -260,13 +260,20 @@ struct swap_cluster_info {#define CLUSTER_FLAG_FREE 1/* This cluster is free */ #define CLUSTER_FLAG_NEXT_NULL 2/* This cluster has no next cluster */ +/* +* The first page in the swap file is the swap header, which is always marked +* bad to prevent it from being allocated as an entry. This also prevents the +* cluster to which it belongs being marked free. Therefore 0 is safe to use as +* a sentinel to indicate next is not valid in percpu_cluster. +*/ +#define SWAP_NEXT_INVALID 0 +/* * We assign a cluster to each CPU, so each CPU can allocate swap entry from * its own cluster and swapout sequentially. The purpose is to optimize swapout * throughput. */ struct percpu_cluster { -struct swap_cluster_info index;/* Current cluster index */ unsigned int next;/* Likely next allocation offset */ };
mm/swapfile.c +11 -11
@@ -609,7 +609,7 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,return false; percpu_cluster =this_cpu_ptr (si->percpu_cluster); -cluster_set_null (&percpu_cluster->index); + percpu_cluster->next =SWAP_NEXT_INVALID ;return true; }
@@ -622,14 +622,14 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, {struct percpu_cluster *cluster;struct swap_cluster_info *ci; -unsigned long tmp, max; +unsigned int tmp, max; new_cluster: cluster =this_cpu_ptr (si->percpu_cluster); -if (cluster_is_null (&cluster->index)) { + tmp = cluster->next; +if (tmp ==SWAP_NEXT_INVALID ) {if (!cluster_list_empty (&si->free_clusters)) { - cluster->index = si->free_clusters.head; - cluster->next =cluster_next (&cluster->index) * + tmp =cluster_next (&si->free_clusters.head) *SWAPFILE_CLUSTER ; }else if (!cluster_list_empty (&si->discard_clusters)) {/*
@@ -649,9 +649,7 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* Other CPUs can use our cluster if they can't find a free cluster, * check if there is still free entry in the cluster */ - tmp = cluster->next; - max = min_t(unsigned long, si->max, - (cluster_next (&cluster->index) +1 ) *SWAPFILE_CLUSTER ); + max = min_t(unsigned long, si->max, ALIGN(tmp + 1, SWAPFILE_CLUSTER));if (tmp < max) { ci =lock_cluster (si, tmp);while (tmp < max) {
@@ -662,12 +660,13 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,unlock_cluster (ci); }if (tmp >= max) { -cluster_set_null (&cluster->index); + cluster->next =SWAP_NEXT_INVALID ; goto new_cluster; } - cluster->next = tmp +1 ; *offset = tmp; *scan_base = tmp; + tmp +=1 ; + cluster->next = tmp < max ? tmp :SWAP_NEXT_INVALID ;return true; }
@@ -3138,8 +3137,9 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) }for_each_possible_cpu (cpu) {struct percpu_cluster *cluster; + cluster =per_cpu_ptr (p->percpu_cluster, cpu); -cluster_set_null (&cluster->index); + cluster->next =SWAP_NEXT_INVALID ; } }else {atomic_inc (&nr_rotate_swap);
Signature · 1 line
2.25.1
Re: [PATCH v4 3/6] mm: swap: Simplify struct percpu_cluster
Quoted 18 lines · Ryan Roberts
struct
percpu_clusterstores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset /SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies thespinlock_tstructure, which is questionable coding practice in my view.So let's clean this up and store only the next offset, and use a sentinal value (
SWAP_NEXT_INVALID) to indicate "no cluster".SWAP_NEXT_INVALIDis chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list.
Quoted 6 lines
This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system.
Signed-off-by Ryan Roberts <[email protected]>
LGTM, Thanks!
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v4 3/6] mm: swap: Simplify struct percpu_cluster
Quoted 20 lines · Ying
Quoted 18 lines · Ryan Roberts
struct
percpu_clusterstores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset /SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies thespinlock_tstructure, which is questionable coding practice in my view.So let's clean this up and store only the next offset, and use a sentinal value (
SWAP_NEXT_INVALID) to indicate "no cluster".SWAP_NEXT_INVALIDis chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list.
Quoted 8 lines
Quoted 6 lines
This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system.
Signed-off-by Ryan Roberts <[email protected]>LGTM, Thanks!
Thanks! What's a guy got to do to get Rb or Ack? :)
Quoted 5 lines
--
Best Regards,
Huang, Ying
Huang, Ying
Reviewed-byRe: [PATCH v4 3/6] mm: swap: Simplify struct percpu_cluster
Quoted 26 lines · Ryan Roberts
Quoted 25 lines · Ying
Quoted 23 lines · Ryan Roberts
struct
percpu_clusterstores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset /SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies thespinlock_tstructure, which is questionable coding practice in my view.So let's clean this up and store only the next offset, and use a sentinal value (
SWAP_NEXT_INVALID) to indicate "no cluster".SWAP_NEXT_INVALIDis chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list.This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>LGTM, Thanks!
Thanks! What's a guy got to do to get Rb or Ack? :)
Feel free to add
in the future version.
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerPATCH v4 4/6 mm: swap: Allow storage of all mTHP orders
In reply to Ryan RobertsMulti-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.
Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 << PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).
The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
--- include/linux/swap.h | 8 ++- mm/swapfile.c | 167 +++++++++++++++++++++++++------------------ 2 files changed, 103 insertions(+), 72 deletions(-)
include/linux/swap.h +7 -1
@@ -268,13 +268,19 @@ struct swap_cluster_info {*/ #define SWAP_NEXT_INVALID 0 +#ifdef CONFIG_THP_SWAP +#define SWAP_NR_ORDERS (PMD_ORDER + 1) +#else +#define SWAP_NR_ORDERS 1 +#endif +/* * We assign a cluster to each CPU, so each CPU can allocate swap entry from * its own cluster and swapout sequentially. The purpose is to optimize swapout * throughput. */ struct percpu_cluster { -unsigned int next;/* Likely next allocation offset */ +unsigned int next[SWAP_NR_ORDERS ];/* Likely next allocation offset */ };struct swap_cluster_list {
mm/swapfile.c +96 -71
338 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 142 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 8 ++- mm/swapfile.c | 167 +++++++++++++++++++++++++------------------ 2 files changed, 103 insertions(+), 72 deletions(-)include/linux/swap.h +7
-1@@ -268,13 +268,19 @@ struct swap_cluster_info {*/ #define SWAP_NEXT_INVALID 0 +#ifdef CONFIG_THP_SWAP +#define SWAP_NR_ORDERS (PMD_ORDER + 1) +#else +#define SWAP_NR_ORDERS 1 +#endif +/* * We assign a cluster to each CPU, so each CPU can allocate swap entry from * its own cluster and swapout sequentially. The purpose is to optimize swapout * throughput. */ struct percpu_cluster { -unsigned int next;/* Likely next allocation offset */ +unsigned int next[SWAP_NR_ORDERS ];/* Likely next allocation offset */ };struct swap_cluster_list {mm/swapfile.c +29
-7@@ -551,10 +551,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -563,9 +565,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased by 1. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -595,7 +607,7 @@ static void dec_cluster_info_page(struct swap_info_struct *p,*/ static bool scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +unsigned long offset,int order) {struct percpu_cluster *percpu_cluster;bool conflict;@@ -609,24 +621,39 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,return false; percpu_cluster =this_cpu_ptr (si->percpu_cluster); - percpu_cluster->next =SWAP_NEXT_INVALID ; + percpu_cluster->next[order] =SWAP_NEXT_INVALID ; +return true; +} + +static inline bool swap_range_empty (char *swap_map,unsigned int start, +unsigned int nr_pages) +{ +unsigned int i; + +for (i =0 ; i < nr_pages; i++) { +if (swap_map[start + i])
mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 642
Quoted 10 lines
mm/swapfile.c +4
-2+return false; + } +return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* Try to get a swap entry (or size indicated by order) from current cpu's swap
IMO, it's not necessary to make mTHP a special case other than base page. So, this can be changed to
* Try to get swap entries with specified order from current cpu's swap
Quoted 54 lines
mm/swapfile.c +12
-8+* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. */ static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +unsigned long *offset,unsigned long *scan_base,int order) { +unsigned int nr_pages =1 << order;struct percpu_cluster *cluster;struct swap_cluster_info *ci;unsigned int tmp, max; new_cluster: cluster =this_cpu_ptr (si->percpu_cluster); - tmp = cluster->next; + tmp = cluster->next[order];if (tmp ==SWAP_NEXT_INVALID ) {if (!cluster_list_empty (&si->free_clusters)) { tmp = cluster_next(&si->free_clusters.head) *@@ -647,26 +674,27 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,/* * Other CPUs can use our cluster if they can't find a free cluster, -* check if there is still free entry in the cluster +* check if there is still free entry in the cluster, maintaining +* natural alignment. */ max = min_t(unsigned long, si->max, ALIGN(tmp + 1, SWAPFILE_CLUSTER));if (tmp < max) { ci =lock_cluster (si, tmp);while (tmp < max) { -if (!si->swap_map[tmp]) +if (swap_range_empty (si->swap_map, tmp, nr_pages))break ; - tmp++; + tmp += nr_pages; }unlock_cluster (ci); }if (tmp >= max) { - cluster->next =SWAP_NEXT_INVALID ; + cluster->next[order] =SWAP_NEXT_INVALID ; goto new_cluster; } *offset = tmp; *scan_base = tmp; - tmp +=1 ; - cluster->next = tmp < max ? tmp :SWAP_NEXT_INVALID ; + tmp += nr_pages; + cluster->next[order] = tmp < max ? tmp :SWAP_NEXT_INVALID ;return true; }
mm/swapfile.c static bool swap_offset_available_and_locked(struct swap_info_struct *si, · line 827
Quoted 5 lines
mm/swapfile.c +1
-1@@ -796,13 +824,14 @@ static bool swap_offset_available_and_locked(struct swap_info_struct *si,static int scan_swap_map_slots (struct swap_info_struct *si,unsigned char usage,int nr, -swp_entry_t slots[]) +swp_entry_t slots[],unsigned int nr_pages)
IMHO, it's better to use order as parameter directly. We can change the parameter of get_swap_pages() too.
Quoted 64 lines
mm/swapfile.c +29
-3{struct swap_cluster_info *ci;unsigned long offset;unsigned long scan_base;unsigned long last_in_cluster =0 ;int latency_ration =LATENCY_LIMIT ; +int order =ilog2 (nr_pages);int n_ret =0 ;bool scanned_many = false;@@ -817,6 +846,26 @@ static int scan_swap_map_slots(struct swap_info_struct *si,* And we let swap pages go all over an SSD partition. Hugh */ +if (nr_pages >1 ) { +/* +* Should not even be attempting large allocations when huge +* page swap is disabled. Warn and fail the allocation. +*/ +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) { +VM_WARN_ON_ONCE (1 ); +return 0 ; + } + +/* +* Swapfile is not block device or not using clusters so unable +* to allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info) +return 0 ; + } + si->flags +=SWP_SCANNING ;/* * Use percpu scan base for SSD to reduce lock contention on @@ -831,8 +880,11 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* SSD algorithm */ if (si->cluster_info) { -if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) { +if (order >0 ) + goto no_page; goto scan; + } }else if (unlikely (!si->cluster_nr--)) {if (si->pages - si->inuse_pages <SWAPFILE_CLUSTER ) { si->cluster_nr =SWAPFILE_CLUSTER -1 ;@@ -874,26 +926,30 @@ static int scan_swap_map_slots(struct swap_info_struct *si, checks:if (si->cluster_info) { -while (scan_swap_map_ssd_cluster_conflict (si, offset)) { +while (scan_swap_map_ssd_cluster_conflict (si, offset, order)) {/* take a break if we already got some slots */ if (n_ret) goto done;if (!scan_swap_map_try_ssd_cluster (si, &offset, - &scan_base)) + &scan_base, order)) { +if (order >0 ) + goto no_page;
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 945
Quoted 10 lines
mm/swapfile.c +2
-1goto scan; + } } }if (!(si->flags &SWP_WRITEOK )) goto no_page;if (!si->highest_bit) goto no_page; -if (offset > si->highest_bit) +if (order ==0 && offset > si->highest_bit)
I don't think that we need to check "order == 0" here. The original condition will always be false for "order != 0".
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 952
Quoted 7 lines
mm/swapfile.c +1
-0scan_base = offset = si->lowest_bit; ci =lock_cluster (si, offset);/* reuse swap entry of cache-only swap if not busy. */ if (vm_swap_full () && si->swap_map[offset] ==SWAP_HAS_CACHE ) {int swap_was_freed; +VM_WARN_ON (order >0 );
Instead of add WARN here, I think that it's better to add WARN at the beginning of "scan" label. We should never scan if "order > 0", it can capture even more abnormal status.
Quoted 7 lines
mm/swapfile.c +0
-0unlock_cluster (ci);spin_unlock (&si->lock); swap_was_freed =__try_to_reclaim_swap (si, offset,TTRS_ANYWAY );@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);
Add barrier() here corresponds to original WRITE_ONCE()? unlock_cluster(ci) may be NOP for some swap devices.
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 978
Quoted 8 lines
mm/swapfile.c +2
-1+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci); -swap_range_alloc (si, offset,1 ); +swap_range_alloc (si, offset, nr_pages); slots[n_ret++] =swp_entry (si->type, offset);/* got enough slots or reach max slots? */
If "order > 0", "nr" must be 1. So, we will "goto done" in the following code.
/* got enough slots or reach max slots? */
if ((n_ret == nr) || (offset >= si->highest_bit))
goto done;We can add VM_WARN_ON() here to capture some abnormal status.
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 999
Quoted 9 lines
mm/swapfile.c +3
-1@@ -936,8 +993,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* try to get more slots in cluster */ if (si->cluster_info) { -if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) goto checks; +if (order >0 ) + goto done;
Don't need to add this, if "order > 0", we will never go here.
Quoted 100 lines
mm/swapfile.c +9
-44}else if (si->cluster_nr && !si->swap_map[++offset]) {/* non-ssd case, still more slots in cluster? */ --si->cluster_nr;@@ -964,7 +1023,8 @@ static int scan_swap_map_slots(struct swap_info_struct *si, } done: -set_cluster_next (si, offset +1 ); +if (order ==0 ) +set_cluster_next (si, offset +1 ); si->flags -=SWP_SCANNING ;return n_ret;@@ -997,38 +1057,6 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) -{ -unsigned long idx; -struct swap_cluster_info *ci; -unsigned long offset; - -/* -* Should not even be attempting cluster allocations when huge -* page swap is disabled. Warn and fail the allocation. -*/ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { -VM_WARN_ON_ONCE (1 ); -return 0 ; - } - -if (cluster_list_empty (&si->free_clusters)) -return 0 ; - - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); - -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); -unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); - -return 1 ; -} -static void swap_free_cluster (struct swap_info_struct *si,unsigned long idx) {unsigned long offset = idx *SWAPFILE_CLUSTER ;@@ -1050,8 +1078,8 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int n_ret =0 ;int node; -/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +/* Only single THP request supported */ +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1088,14 +1116,10 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); - }else - n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , - n_goal, swp_entries); + n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , + n_goal, swp_entries, size);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -1647,7 +1671,7 @@ swp_entry_t get_swap_page_of_type(int type)/* This is called for allocating swap entry, not cache */ spin_lock (&si->lock); -if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry)) +if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry,1 ))atomic_long_dec (&nr_swap_pages);spin_unlock (&si->lock); fail:@@ -3101,7 +3125,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) p->flags |=SWP_SYNCHRONOUS_IO ;if (p->bdev &&bdev_nonrot (p->bdev)) { -int cpu; +int cpu, i;unsigned long ci, nr_cluster; p->flags |=SWP_SOLIDSTATE ;
mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 3170
Quoted 9 lines
mm/swapfile.c +2
-1@@ -3139,7 +3163,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)struct percpu_cluster *cluster; cluster =per_cpu_ptr (p->percpu_cluster, cpu); - cluster->next =SWAP_NEXT_INVALID ; +for (i =0 ; i <SWAP_NR_ORDERS ; i++) + cluster->next[i] =SWAP_NEXT_INVALID ; } }else {atomic_inc (&nr_rotate_swap);
You also need to check whether we should add swap_entry_size() for some functions to optimize for small system. We may need to add swap_order() too.
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 158 lines · Ying
Quoted 142 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 8 ++- mm/swapfile.c | 167 +++++++++++++++++++++++++------------------ 2 files changed, 103 insertions(+), 72 deletions(-)include/linux/swap.h +7
-1@@ -268,13 +268,19 @@ struct swap_cluster_info {*/ #define SWAP_NEXT_INVALID 0 +#ifdef CONFIG_THP_SWAP +#define SWAP_NR_ORDERS (PMD_ORDER + 1) +#else +#define SWAP_NR_ORDERS 1 +#endif +/* * We assign a cluster to each CPU, so each CPU can allocate swap entry from * its own cluster and swapout sequentially. The purpose is to optimize swapout * throughput. */ struct percpu_cluster { -unsigned int next;/* Likely next allocation offset */ +unsigned int next[SWAP_NR_ORDERS ];/* Likely next allocation offset */ };struct swap_cluster_list {mm/swapfile.c +29
-7@@ -551,10 +551,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -563,9 +565,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased by 1. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -595,7 +607,7 @@ static void dec_cluster_info_page(struct swap_info_struct *p,*/ static bool scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +unsigned long offset,int order) {struct percpu_cluster *percpu_cluster;bool conflict;@@ -609,24 +621,39 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,return false; percpu_cluster =this_cpu_ptr (si->percpu_cluster); - percpu_cluster->next =SWAP_NEXT_INVALID ; + percpu_cluster->next[order] =SWAP_NEXT_INVALID ; +return true; +} + +static inline bool swap_range_empty (char *swap_map,unsigned int start, +unsigned int nr_pages) +{ +unsigned int i; + +for (i =0 ; i < nr_pages; i++) { +if (swap_map[start + i])mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 642
Quoted 10 lines
mm/swapfile.c +4
-2+return false; + } +return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* Try to get a swap entry (or size indicated by order) from current cpu's swap IMO, it's not necessary to make mTHP a special case other than base page. So, this can be changed to
Quoted 1 line
* Try to get swap entries with specified order from current cpu's swap
Sure, will fix in next version.
Quoted 56 lines
Quoted 56 lines
mm/swapfile.c +12
-8+* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. */ static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +unsigned long *offset,unsigned long *scan_base,int order) { +unsigned int nr_pages =1 << order;struct percpu_cluster *cluster;struct swap_cluster_info *ci;unsigned int tmp, max; new_cluster: cluster =this_cpu_ptr (si->percpu_cluster); - tmp = cluster->next; + tmp = cluster->next[order];if (tmp ==SWAP_NEXT_INVALID ) {if (!cluster_list_empty (&si->free_clusters)) { tmp = cluster_next(&si->free_clusters.head) *@@ -647,26 +674,27 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,/* * Other CPUs can use our cluster if they can't find a free cluster, -* check if there is still free entry in the cluster +* check if there is still free entry in the cluster, maintaining +* natural alignment. */ max = min_t(unsigned long, si->max, ALIGN(tmp + 1, SWAPFILE_CLUSTER));if (tmp < max) { ci =lock_cluster (si, tmp);while (tmp < max) { -if (!si->swap_map[tmp]) +if (swap_range_empty (si->swap_map, tmp, nr_pages))break ; - tmp++; + tmp += nr_pages; }unlock_cluster (ci); }if (tmp >= max) { - cluster->next =SWAP_NEXT_INVALID ; + cluster->next[order] =SWAP_NEXT_INVALID ; goto new_cluster; } *offset = tmp; *scan_base = tmp; - tmp +=1 ; - cluster->next = tmp < max ? tmp :SWAP_NEXT_INVALID ; + tmp += nr_pages; + cluster->next[order] = tmp < max ? tmp :SWAP_NEXT_INVALID ;return true; }@@ -796,13 +824,14 @@ static bool swap_offset_available_and_locked(struct swap_info_struct *si,
Quoted 7 lines
mm/swapfile.c static bool swap_offset_available_and_locked(struct swap_info_struct *si, · line 827
Quoted 4 lines
mm/swapfile.c +1
-1static int scan_swap_map_slots (struct swap_info_struct *si,unsigned char usage,int nr, -swp_entry_t slots[]) +swp_entry_t slots[],unsigned int nr_pages)IMHO, it's better to use order as parameter directly. We can change the parameter of
get_swap_pages()too.
I agree that this will make the interface clearer/self documenting. I'll do it in the next version.
Quoted 70 lines
Quoted 70 lines
mm/swapfile.c +30
-3{struct swap_cluster_info *ci;unsigned long offset;unsigned long scan_base;unsigned long last_in_cluster =0 ;int latency_ration =LATENCY_LIMIT ; +int order =ilog2 (nr_pages);int n_ret =0 ;bool scanned_many = false;@@ -817,6 +846,26 @@ static int scan_swap_map_slots(struct swap_info_struct *si,* And we let swap pages go all over an SSD partition. Hugh */ +if (nr_pages >1 ) { +/* +* Should not even be attempting large allocations when huge +* page swap is disabled. Warn and fail the allocation. +*/ +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) { +VM_WARN_ON_ONCE (1 ); +return 0 ; + } + +/* +* Swapfile is not block device or not using clusters so unable +* to allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info) +return 0 ; + } + si->flags +=SWP_SCANNING ;/* * Use percpu scan base for SSD to reduce lock contention on @@ -831,8 +880,11 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* SSD algorithm */ if (si->cluster_info) { -if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) { +if (order >0 ) + goto no_page; goto scan; + } }else if (unlikely (!si->cluster_nr--)) {if (si->pages - si->inuse_pages <SWAPFILE_CLUSTER ) { si->cluster_nr =SWAPFILE_CLUSTER -1 ;@@ -874,26 +926,30 @@ static int scan_swap_map_slots(struct swap_info_struct *si, checks:if (si->cluster_info) { -while (scan_swap_map_ssd_cluster_conflict (si, offset)) { +while (scan_swap_map_ssd_cluster_conflict (si, offset, order)) {/* take a break if we already got some slots */ if (n_ret) goto done;if (!scan_swap_map_try_ssd_cluster (si, &offset, - &scan_base)) + &scan_base, order)) { +if (order >0 ) + goto no_page; goto scan; + } } }if (!(si->flags &SWP_WRITEOK ))
Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 945
Quoted 5 lines
mm/swapfile.c +1
-1goto no_page;if (!si->highest_bit) goto no_page; -if (offset > si->highest_bit) +if (order ==0 && offset > si->highest_bit)I don't think that we need to check "order == 0" here. The original condition will always be false for "order != 0".
I spent ages looking at this and couldn't quite convince myself that this is definitely safe. Certainly it would be catastrophic if we modified the returned offset for a non-order-0 case (the code below assumes order-0 when checking). So I decided in the end to be safe and add this condition. Looking again, I agree with you. Will fix in next version.
Quoted 12 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 952
Quoted 8 lines
mm/swapfile.c +1
-0scan_base = offset = si->lowest_bit; ci =lock_cluster (si, offset);/* reuse swap entry of cache-only swap if not busy. */ if (vm_swap_full () && si->swap_map[offset] ==SWAP_HAS_CACHE ) {int swap_was_freed; +VM_WARN_ON (order >0 );Instead of add WARN here, I think that it's better to add WARN at the beginning of "scan" label. We should never scan if "order > 0", it can capture even more abnormal status.
OK, will do.
Quoted 9 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 964
Quoted 9 lines
mm/swapfile.c +1
-0unlock_cluster (ci);spin_unlock (&si->lock); swap_was_freed =__try_to_reclaim_swap (si, offset,TTRS_ANYWAY );@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) { +VM_WARN_ON (order >0 );
And remove this one too? (relying on the one in scan instead)
Quoted 4 lines
Quoted 4 lines
mm/swapfile.c +0
-0unlock_cluster (ci);if (!n_ret) goto scan;else
Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 5 lines
mm/swapfile.c +1
-2goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.
Yep, good spot!
Quoted 4 lines
Quoted 4 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);
Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 978
Quoted 5 lines
mm/swapfile.c +1
-1-swap_range_alloc (si, offset,1 ); +swap_range_alloc (si, offset, nr_pages); slots[n_ret++] =swp_entry (si->type, offset);/* got enough slots or reach max slots? */ If "order > 0", "nr" must be 1. So, we will "goto done" in the following code.
I've deliberately implemented scan_swap_map_slots() so that it allows nr > 1 for order > 0. And leave it to the higher layers to decide on policy.
Quoted 6 lines
/* got enough slots or reach max slots? */ if ((n_ret == nr) || (offset >= si->highest_bit)) goto done;We can add
VM_WARN_ON()here to capture some abnormal status.
That was actually how I implemented initially. But decided that it doesn't cost anything to allow nr > 1 for order > 0, and IMHO makes the function easier to understand because we remove this uneccessary constraint.
Quoted 12 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 999
Quoted 10 lines
mm/swapfile.c +3
-1@@ -936,8 +993,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* try to get more slots in cluster */ if (si->cluster_info) { -if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) goto checks; +if (order >0 ) + goto done;Don't need to add this, if "order > 0", we will never go here.
As per above.
Quoted 103 lines
Quoted 103 lines
mm/swapfile.c +9
-44}else if (si->cluster_nr && !si->swap_map[++offset]) {/* non-ssd case, still more slots in cluster? */ --si->cluster_nr;@@ -964,7 +1023,8 @@ static int scan_swap_map_slots(struct swap_info_struct *si, } done: -set_cluster_next (si, offset +1 ); +if (order ==0 ) +set_cluster_next (si, offset +1 ); si->flags -=SWP_SCANNING ;return n_ret;@@ -997,38 +1057,6 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) -{ -unsigned long idx; -struct swap_cluster_info *ci; -unsigned long offset; - -/* -* Should not even be attempting cluster allocations when huge -* page swap is disabled. Warn and fail the allocation. -*/ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { -VM_WARN_ON_ONCE (1 ); -return 0 ; - } - -if (cluster_list_empty (&si->free_clusters)) -return 0 ; - - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); - -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); -unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); - -return 1 ; -} -static void swap_free_cluster (struct swap_info_struct *si,unsigned long idx) {unsigned long offset = idx *SWAPFILE_CLUSTER ;@@ -1050,8 +1078,8 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int n_ret =0 ;int node; -/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +/* Only single THP request supported */ +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1088,14 +1116,10 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); - }else - n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , - n_goal, swp_entries); + n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , + n_goal, swp_entries, size);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -1647,7 +1671,7 @@ swp_entry_t get_swap_page_of_type(int type)/* This is called for allocating swap entry, not cache */ spin_lock (&si->lock); -if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry)) +if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry,1 ))atomic_long_dec (&nr_swap_pages);spin_unlock (&si->lock); fail:@@ -3101,7 +3125,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) p->flags |=SWP_SYNCHRONOUS_IO ;if (p->bdev &&bdev_nonrot (p->bdev)) { -int cpu; +int cpu, i;unsigned long ci, nr_cluster; p->flags |=SWP_SOLIDSTATE ;@@ -3139,7 +3163,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)struct percpu_cluster *cluster;
Quoted 11 lines
mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 3170
Quoted 7 lines
mm/swapfile.c +2
-1cluster =per_cpu_ptr (p->percpu_cluster, cpu); - cluster->next =SWAP_NEXT_INVALID ; +for (i =0 ; i <SWAP_NR_ORDERS ; i++) + cluster->next[i] =SWAP_NEXT_INVALID ; } }else {atomic_inc (&nr_rotate_swap);You also need to check whether we should add
swap_entry_size()for some functions to optimize for small system. We may need to addswap_order()too.
I was planning to convert swap_entry_size() to swap_entry_order() as part of switching to pass order instead of nr_pages. There is one other site that uses swap_entry_size() and needs a size, so was going to just change it to 1 << swap_entry_order(). Does that work for you?
I'll do an audit for places to use swap_entry_order() but quick scan just now suggests that the constant should propagate to all the static functions from get_swap_pages().
Thanks,
Ryan
Quoted 4 lines
--
Best Regards,
Huang, Ying
Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 337 lines · Ryan Roberts
Quoted 158 lines · Ying
Quoted 142 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 8 ++- mm/swapfile.c | 167 +++++++++++++++++++++++++------------------ 2 files changed, 103 insertions(+), 72 deletions(-)include/linux/swap.h +7
-1@@ -268,13 +268,19 @@ struct swap_cluster_info {*/ #define SWAP_NEXT_INVALID 0 +#ifdef CONFIG_THP_SWAP +#define SWAP_NR_ORDERS (PMD_ORDER + 1) +#else +#define SWAP_NR_ORDERS 1 +#endif +/* * We assign a cluster to each CPU, so each CPU can allocate swap entry from * its own cluster and swapout sequentially. The purpose is to optimize swapout * throughput. */ struct percpu_cluster { -unsigned int next;/* Likely next allocation offset */ +unsigned int next[SWAP_NR_ORDERS ];/* Likely next allocation offset */ };struct swap_cluster_list {mm/swapfile.c +29
-7@@ -551,10 +551,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -563,9 +565,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased by 1. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -595,7 +607,7 @@ static void dec_cluster_info_page(struct swap_info_struct *p,*/ static bool scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +unsigned long offset,int order) {struct percpu_cluster *percpu_cluster;bool conflict;@@ -609,24 +621,39 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,return false; percpu_cluster =this_cpu_ptr (si->percpu_cluster); - percpu_cluster->next =SWAP_NEXT_INVALID ; + percpu_cluster->next[order] =SWAP_NEXT_INVALID ; +return true; +} + +static inline bool swap_range_empty (char *swap_map,unsigned int start, +unsigned int nr_pages) +{ +unsigned int i; + +for (i =0 ; i < nr_pages; i++) { +if (swap_map[start + i])mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 642
Quoted 10 lines
mm/swapfile.c +4
-2+return false; + } +return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* Try to get a swap entry (or size indicated by order) from current cpu's swap IMO, it's not necessary to make mTHP a special case other than base page. So, this can be changed to
Quoted 1 line
* Try to get swap entries with specified order from current cpu's swap
Sure, will fix in next version.
Quoted 56 lines
Quoted 56 lines
mm/swapfile.c +12
-8+* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. */ static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +unsigned long *offset,unsigned long *scan_base,int order) { +unsigned int nr_pages =1 << order;struct percpu_cluster *cluster;struct swap_cluster_info *ci;unsigned int tmp, max; new_cluster: cluster =this_cpu_ptr (si->percpu_cluster); - tmp = cluster->next; + tmp = cluster->next[order];if (tmp ==SWAP_NEXT_INVALID ) {if (!cluster_list_empty (&si->free_clusters)) { tmp = cluster_next(&si->free_clusters.head) *@@ -647,26 +674,27 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,/* * Other CPUs can use our cluster if they can't find a free cluster, -* check if there is still free entry in the cluster +* check if there is still free entry in the cluster, maintaining +* natural alignment. */ max = min_t(unsigned long, si->max, ALIGN(tmp + 1, SWAPFILE_CLUSTER));if (tmp < max) { ci =lock_cluster (si, tmp);while (tmp < max) { -if (!si->swap_map[tmp]) +if (swap_range_empty (si->swap_map, tmp, nr_pages))break ; - tmp++; + tmp += nr_pages; }unlock_cluster (ci); }if (tmp >= max) { - cluster->next =SWAP_NEXT_INVALID ; + cluster->next[order] =SWAP_NEXT_INVALID ; goto new_cluster; } *offset = tmp; *scan_base = tmp; - tmp +=1 ; - cluster->next = tmp < max ? tmp :SWAP_NEXT_INVALID ; + tmp += nr_pages; + cluster->next[order] = tmp < max ? tmp :SWAP_NEXT_INVALID ;return true; }@@ -796,13 +824,14 @@ static bool swap_offset_available_and_locked(struct swap_info_struct *si,Quoted 7 lines
mm/swapfile.c static bool swap_offset_available_and_locked(struct swap_info_struct *si, · line 827
Quoted 4 lines
mm/swapfile.c +1
-1static int scan_swap_map_slots (struct swap_info_struct *si,unsigned char usage,int nr, -swp_entry_t slots[]) +swp_entry_t slots[],unsigned int nr_pages)IMHO, it's better to use order as parameter directly. We can change the parameter of
get_swap_pages()too.I agree that this will make the interface clearer/self documenting. I'll do it in the next version.
Quoted 70 lines
Quoted 70 lines
mm/swapfile.c +30
-3{struct swap_cluster_info *ci;unsigned long offset;unsigned long scan_base;unsigned long last_in_cluster =0 ;int latency_ration =LATENCY_LIMIT ; +int order =ilog2 (nr_pages);int n_ret =0 ;bool scanned_many = false;@@ -817,6 +846,26 @@ static int scan_swap_map_slots(struct swap_info_struct *si,* And we let swap pages go all over an SSD partition. Hugh */ +if (nr_pages >1 ) { +/* +* Should not even be attempting large allocations when huge +* page swap is disabled. Warn and fail the allocation. +*/ +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) { +VM_WARN_ON_ONCE (1 ); +return 0 ; + } + +/* +* Swapfile is not block device or not using clusters so unable +* to allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info) +return 0 ; + } + si->flags +=SWP_SCANNING ;/* * Use percpu scan base for SSD to reduce lock contention on @@ -831,8 +880,11 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* SSD algorithm */ if (si->cluster_info) { -if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (!scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) { +if (order >0 ) + goto no_page; goto scan; + } }else if (unlikely (!si->cluster_nr--)) {if (si->pages - si->inuse_pages <SWAPFILE_CLUSTER ) { si->cluster_nr =SWAPFILE_CLUSTER -1 ;@@ -874,26 +926,30 @@ static int scan_swap_map_slots(struct swap_info_struct *si, checks:if (si->cluster_info) { -while (scan_swap_map_ssd_cluster_conflict (si, offset)) { +while (scan_swap_map_ssd_cluster_conflict (si, offset, order)) {/* take a break if we already got some slots */ if (n_ret) goto done;if (!scan_swap_map_try_ssd_cluster (si, &offset, - &scan_base)) + &scan_base, order)) { +if (order >0 ) + goto no_page; goto scan; + } } }if (!(si->flags &SWP_WRITEOK ))Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 945
Quoted 5 lines
mm/swapfile.c +1
-1goto no_page;if (!si->highest_bit) goto no_page; -if (offset > si->highest_bit) +if (order ==0 && offset > si->highest_bit)I don't think that we need to check "order == 0" here. The original condition will always be false for "order != 0".
I spent ages looking at this and couldn't quite convince myself that this is definitely safe. Certainly it would be catastrophic if we modified the returned offset for a non-order-0 case (the code below assumes order-0 when checking). So I decided in the end to be safe and add this condition. Looking again, I agree with you. Will fix in next version.
Quoted 12 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 952
Quoted 8 lines
mm/swapfile.c +1
-0scan_base = offset = si->lowest_bit; ci =lock_cluster (si, offset);/* reuse swap entry of cache-only swap if not busy. */ if (vm_swap_full () && si->swap_map[offset] ==SWAP_HAS_CACHE ) {int swap_was_freed; +VM_WARN_ON (order >0 );Instead of add WARN here, I think that it's better to add WARN at the beginning of "scan" label. We should never scan if "order > 0", it can capture even more abnormal status.
OK, will do.
Quoted 7 lines
Quoted 7 lines
mm/swapfile.c +0
-0unlock_cluster (ci);spin_unlock (&si->lock); swap_was_freed =__try_to_reclaim_swap (si, offset,TTRS_ANYWAY );@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }
Quoted 4 lines
Quoted 2 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 964
Quoted 2 lines
mm/swapfile.c +1
-0if (si->swap_map[offset]) { +VM_WARN_ON (order >0 );And remove this one too? (relying on the one in scan instead)
Yes. I think so.
Quoted 36 lines
Quoted 4 lines
Quoted 4 lines
mm/swapfile.c +0
-0unlock_cluster (ci);if (!n_ret) goto scan;else Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 5 lines
mm/swapfile.c +1
-2goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.Yep, good spot!
Quoted 4 lines
Quoted 4 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 978
Quoted 5 lines
mm/swapfile.c +1
-1-swap_range_alloc (si, offset,1 ); +swap_range_alloc (si, offset, nr_pages); slots[n_ret++] =swp_entry (si->type, offset);/* got enough slots or reach max slots? */ If "order > 0", "nr" must be 1. So, we will "goto done" in the following code.
I've deliberately implemented
scan_swap_map_slots()so that it allows nr > 1 for order > 0. And leave it to the higher layers to decide on policy.Quoted 5 lines
/* got enough slots or reach max slots? */ if ((n_ret == nr) || (offset >= si->highest_bit)) goto done;
Quoted 5 lines
Quoted 1 line
We can add
VM_WARN_ON()here to capture some abnormal status.That was actually how I implemented initially. But decided that it doesn't cost anything to allow nr > 1 for order > 0, and IMHO makes the function easier to understand because we remove this uneccessary constraint.
This sounds reasonable to me.
Quoted 130 lines
Quoted 12 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 999
Quoted 10 lines
mm/swapfile.c +3
-1@@ -936,8 +993,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si,/* try to get more slots in cluster */ if (si->cluster_info) { -if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base)) +if (scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order)) goto checks; +if (order >0 ) + goto done;Don't need to add this, if "order > 0", we will never go here.
As per above.
Quoted 103 lines
Quoted 103 lines
mm/swapfile.c +9
-44}else if (si->cluster_nr && !si->swap_map[++offset]) {/* non-ssd case, still more slots in cluster? */ --si->cluster_nr;@@ -964,7 +1023,8 @@ static int scan_swap_map_slots(struct swap_info_struct *si, } done: -set_cluster_next (si, offset +1 ); +if (order ==0 ) +set_cluster_next (si, offset +1 ); si->flags -=SWP_SCANNING ;return n_ret;@@ -997,38 +1057,6 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) -{ -unsigned long idx; -struct swap_cluster_info *ci; -unsigned long offset; - -/* -* Should not even be attempting cluster allocations when huge -* page swap is disabled. Warn and fail the allocation. -*/ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { -VM_WARN_ON_ONCE (1 ); -return 0 ; - } - -if (cluster_list_empty (&si->free_clusters)) -return 0 ; - - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); - -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); -unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); - -return 1 ; -} -static void swap_free_cluster (struct swap_info_struct *si,unsigned long idx) {unsigned long offset = idx *SWAPFILE_CLUSTER ;@@ -1050,8 +1078,8 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int n_ret =0 ;int node; -/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +/* Only single THP request supported */ +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1088,14 +1116,10 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); - }else - n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , - n_goal, swp_entries); + n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , + n_goal, swp_entries, size);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -1647,7 +1671,7 @@ swp_entry_t get_swap_page_of_type(int type)/* This is called for allocating swap entry, not cache */ spin_lock (&si->lock); -if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry)) +if ((si->flags &SWP_WRITEOK ) &&scan_swap_map_slots (si,1 ,1 , &entry,1 ))atomic_long_dec (&nr_swap_pages);spin_unlock (&si->lock); fail:@@ -3101,7 +3125,7 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) p->flags |=SWP_SYNCHRONOUS_IO ;if (p->bdev &&bdev_nonrot (p->bdev)) { -int cpu; +int cpu, i;unsigned long ci, nr_cluster; p->flags |=SWP_SOLIDSTATE ;@@ -3139,7 +3163,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)struct percpu_cluster *cluster;Quoted 11 lines
mm/swapfile.c SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags) · line 3170
Quoted 7 lines
mm/swapfile.c +2
-1cluster =per_cpu_ptr (p->percpu_cluster, cpu); - cluster->next =SWAP_NEXT_INVALID ; +for (i =0 ; i <SWAP_NR_ORDERS ; i++) + cluster->next[i] =SWAP_NEXT_INVALID ; } }else {atomic_inc (&nr_rotate_swap);You also need to check whether we should add
swap_entry_size()for some functions to optimize for small system. We may need to addswap_order()too.
Quoted 4 lines
I was planning to convert
swap_entry_size()toswap_entry_order()as part of switching to pass order instead ofnr_pages. There is one other site that usesswap_entry_size()and needs a size, so was going to just change it to 1 <<swap_entry_order(). Does that work for you?
Yes.
Quoted 4 lines
I'll do an audit for places to use
swap_entry_order()but quick scan just now suggests that the constant should propagate to all the static functions fromget_swap_pages().
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
In reply to Huang, YingHi Huang, Ying,
Quoted 45 lines · Ying
Quoted 43 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
[...]
Quoted 17 lines
Quoted 4 lines
mm/swapfile.c +0
-0@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.
Looking at this a bit more closely, I'm not sure this is needed. Even if there is no cluster, the swap_info is still locked, so unlocking that will act as a barrier. There are a number of other callsites that memset(si->swap_map) without an explicit barrier and with the swap_info locked.
Looking at the original commit that added the WRITE_ONCE() it was worried about a race with reading swap_map in _swap_info_get(). But that site is now annotated with a data_race(), which will suppress the warning. And I don't believe there are any places that read swap_map locklessly and depend upon observing ordering between it and other state? So I think the si unlock is sufficient?
I'm not planning to add barrier() here. Let me know if you disagree.
Thanks,
Ryan
Quoted 3 lines
Quoted 3 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);
Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 75 lines · Ryan Roberts
Hi Huang, Ying,
Quoted 45 lines · Ying
Quoted 43 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
[...]
Quoted 17 lines
Quoted 4 lines
mm/swapfile.c +0
-0@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.Looking at this a bit more closely, I'm not sure this is needed. Even if there is no cluster, the
swap_infois still locked, so unlocking that will act as a barrier. There are a number of other callsites thatmemset(si->swap_map)without an explicit barrier and with theswap_infolocked.
Quoted 7 lines
Looking at the original commit that added the
WRITE_ONCE()it was worried about a race with readingswap_mapin_swap_info_get(). But that site is now annotated with adata_race(), which will suppress the warning. And I don't believe there are any places that readswap_maplocklessly and depend upon observing ordering between it and other state? So I think the si unlock is sufficient?I'm not planning to add
barrier()here. Let me know if you disagree.
swap_map[] may be read locklessly in swap_offset_available_and_locked() in parallel. IIUC, WRITE_ONCE() here is to make the writing take effect as early as possible there.
Quoted 4 lines
Quoted 4 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 973
Quoted 4 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 83 lines · Ying
Quoted 81 lines · Ryan Roberts
Hi Huang, Ying,
Quoted 45 lines · Ying
Quoted 43 lines · Ryan Roberts
Multi-size THP enables performance improvements by allocating large, pte-mapped folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting when the large folios are swapped out.
Therefore, solve this regression by adding support for swapping out mTHP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in (m)THP here - this is still done page-by-page, like for PMD-sized THP. But swapping-out mTHP is a prerequisite for swapping-in mTHP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller can fall back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
This change only modifies swap to be able to accept any order mTHP. It doesn't change the callers to elide doing the actual split. That will be done in separate changes.
[...]
Quoted 17 lines
Quoted 4 lines
mm/swapfile.c +0
-0@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.Looking at this a bit more closely, I'm not sure this is needed. Even if there is no cluster, the
swap_infois still locked, so unlocking that will act as a barrier. There are a number of other callsites thatmemset(si->swap_map)without an explicit barrier and with theswap_infolocked.Looking at the original commit that added the
WRITE_ONCE()it was worried about a race with readingswap_mapin_swap_info_get(). But that site is now annotated with adata_race(), which will suppress the warning. And I don't believe there are any places that readswap_maplocklessly and depend upon observing ordering between it and other state? So I think the si unlock is sufficient?
Quoted 5 lines
Quoted 1 line
I'm not planning to add
barrier()here. Let me know if you disagree.swap_map[] may be read locklessly in
swap_offset_available_and_locked()in parallel. IIUC,WRITE_ONCE()here is to make the writing take effect as early as possible there.
Afraid I'm not convinced by that argument; if it's racing, it's racing - the lockless side needs to be robust (it is). Adding the compiler barrier limits the compiler's options which could lead to slower code in this path. If your argument is that you want to reduce the window where swap_offset_available_and_locked() could observe a free swap slot but then see that its taken after it gets the si lock, that seems like a micro-optimization to me, which we should avoid if we can.
By remnoving the WRITE_ONCE() and using memset, the lockless reader could observe tearing though. I don't think that should cause a problem (because everything is rechecked with under the lock), but if we want to avoid it, then perhaps we just need to loop over WRITE_ONCE() here instead of using memset?
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 973
Quoted 5 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);--
Best Regards,
Huang, Ying
Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];
void writer(void)
{
memset(x, 1, sizeof(x));
/* To make memset() take effect ASAP */
barrier();
}unsigned char reader(int n)
{
return READ_ONCE(x[n]);
}"
where, writer() and reader() may be called on 2 CPUs without any lock. It's acceptable for reader() to read the written value a little later. Our questions are,
1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use barrier() and READ_ONCE() in writer() and reader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Quoted 15 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }
Quoted 7 lines
unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
where,
writer()andreader()may be called on 2 CPUs without any lock.
For the situation we are discussing, writer() is always called with a spin lock held. So spin_unlock() will act as the barrier in this case; that's my argument for not needing the explicit barrier(), anyway. Happy to be told I'm wrong.
Quoted 14 lines
It's acceptable for
reader()to read the written value a little later. Our questions are,1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?--
Best Regards,
Huang, Ying
Re: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Quoted 22 lines · Ryan Roberts
Quoted 21 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
Quoted 5 lines
Quoted 1 line
where,
writer()andreader()may be called on 2 CPUs without any lock.For the situation we are discussing,
writer()is always called with a spin lock held. Sospin_unlock()will act as the barrier in this case; that's my argument for not needing the explicitbarrier(), anyway. Happy to be told I'm wrong.
Yes. spin_unlock() is a barrier too. There are some operations between writer() and spin_unlock(), so I want to check whether it make any sense to add a barrier earlier.
Quoted 10 lines
Quoted 10 lines
It's acceptable for
reader()to read the written value a little later. Our questions are,1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?
--
Best Regards,
Huang, Ying
Chris Li
maintainerRe: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
In reply to Huang, YingHi Ying,
Very interesting question.
Quoted 16 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }
Quoted 8 lines
unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later.
I am trying to see if your program can convert into a litmus test so the linux memory model tools can answer it for you.
Because you allow reader() to read written value a little later, there is nothing the test can verify against. The reader can see both before or after the writer's update, both are valid observations.
To make your test example more complete, you need the reader/writer to do more actions to expose the race. For example, " if (READ_ONCE(x[n]) y = 1;" Then you can ask the question whether it is possible to observe x[n] == 0 and y== 1. That might not be the test condition you have in mind, you can get the idea.
We want to have a test example that shows the result observable state to indicate the bad things did happen(or not possible).
Quoted 5 lines
Our questions are,
1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
We need to define what is the expected behavior outcome to be "correct", possibly including the before and after barrier actions.
Chris
Quoted 9 lines
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?--
Best Regards,
Huang, Ying
Re: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
In reply to Huang, Ying[Use Paul's reachable address in CC;
trimmed CC list, keeping only those who have responded so far.]
Hello Huang,
Let me chime in.
Quoted 25 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later. Our questions are,
Quoted 7 lines
1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?
Why don't you consult Documentation/memory-barriers.txt, especially the section titled "COMPILER BARRIER"?
TL;DR:
barrier(), WRITE_ONCE(), and READ_ONCE() are compiler barriers, not memory barriers. They just restrict compiler optimizations and don't have any effect with regard to "make writing take effect ASAP".
If you have further questions, please don't hesitate to ask.
Regards,
Akira (a LKMM Reveiwer).
Quoted 4 lines
--
Best Regards,
Huang, Ying
Re: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Quoted 30 lines · Akira Yokosawa
[Use Paul's reachable address in CC;
trimmed CC list, keeping only those who have responded so far.]Hello Huang,
Let me chime in.Quoted 23 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later.
What are your consistency requirements? For but one example, if reader(3) gives the new value, is it OK for a later call to reader(2) to give the old value?
Until we know what your requirements are, it is hard to say whether the above code meets those requirements. In the meantime, I can imagine requirements that it meets and others that it does not.
Also, Akira's points below are quite important.
Thanx, Paul
Quoted 29 lines
Quoted 9 lines
Our questions are,
1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?Why don't you consult
Documentation/memory-barriers.txt, especially the section titled "COMPILER BARRIER"?TL;DR:
barrier(),WRITE_ONCE(), andREAD_ONCE()are compiler barriers, not memory barriers. They just restrict compiler optimizations and don't have any effect with regard to "make writing take effect ASAP".If you have further questions, please don't hesitate to ask.
Regards,
Akira (a LKMM Reveiwer).Quoted 4 lines
--
Best Regards,
Huang, Ying
Re: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
[Use Paul's reachable address in CC;
trimmed CC list, keeping only those who have responded so far.]Hello Huang,
Let me chime in.Quoted 21 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later.What are your consistency requirements? For but one example, if reader(3) gives the new value, is it OK for a later call to reader(2) to give the old value?
writer() will be called with a lock held (sorry, my previous words aren't correct here). After the racy checking in reader(), we will acquire the lock and check "x[n]" again to confirm. And, there are no dependencies between different "n". All in all, we can accept almost all races between writer() and reader().
My question is, if there are some operations between writer() and unlocking in its caller, whether does barrier() in writer() make any sense? Make write instructions appear a little earlier in compiled code? Mark the memory may be read racy? Or doesn't make sense at all?
Quoted 5 lines
Until we know what your requirements are, it is hard to say whether the above code meets those requirements. In the meantime, I can imagine requirements that it meets and others that it does not.
Also, Akira's points below are quite important.
Replied for his email.
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
Quoted 44 lines · Ying
Quoted 29 lines · Paul E. McKenney
Quoted 28 lines · Akira Yokosawa
[Use Paul's reachable address in CC;
trimmed CC list, keeping only those who have responded so far.]Hello Huang,
Let me chime in.Quoted 21 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later.What are your consistency requirements? For but one example, if reader(3) gives the new value, is it OK for a later call to reader(2) to give the old value?
writer()will be called with a lock held (sorry, my previous words aren't correct here). After the racy checking inreader(), we will acquire the lock and check "x[n]" again to confirm. And, there are no dependencies between different "n". All in all, we can accept almost all races betweenwriter()andreader().
Quoted 4 lines
My question is, if there are some operations between
writer()and unlocking in its caller, whether doesbarrier()inwriter()make any sense? Make write instructions appear a little earlier in compiled code? Mark the memory may be read racy? Or doesn't make sense at all?
A compiler barrier is neccessary but not sufficient to guarrantee that the stores become visible to the reader; you would also need a memory barrier to stop the HW from reordering IIUC. So I really fail to see the value of adding barrier().
As you state above there is no correctness issue here. Its just a question of whether the barrier() can make the store appear earlier to the reader for a (micro!) performance optimization. You'll get both the compiler and memory barrier from the slightly later spin_unlock(). The patch that added the original WRITE_ONCE() was concerned with squashing kcsan warnings, not with performance optimization. (And the addition of the WRITE_ONCE() wasn't actually needed to achieve the aim).
So I'm planning to repost my series (hopefully tomorrow) without the barrier() present, unless you still want to try to convince me that it is useful.
Thanks,
Ryan
Quoted 12 lines
Quoted 6 lines
Until we know what your requirements are, it is hard to say whether the above code meets those requirements. In the meantime, I can imagine requirements that it meets and others that it does not.
Also, Akira's points below are quite important.
Replied for his email.
--
Best Regards,
Huang, Ying
Re: Can you help us on memory barrier usage? (was Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders)
In reply to Akira YokosawaQuoted 2 lines · Akira Yokosawa
[Use Paul's reachable address in CC;
trimmed CC list, keeping only those who have responded so far.]
Thanks a lot!
Quoted 37 lines
Hello Huang,
Let me chime in.Quoted 25 lines · Ying
Hi, Paul,
Can you help us on WRITE_ONCE()/READ_ONCE()/barrier() usage as follows? For some example kernel code as follows,
"
unsigned char x[16];void writer(void)
{ memset(x, 1, sizeof(x)); /* To make memset() take effect ASAP */ barrier(); }unsigned char reader(int n)
{ return READ_ONCE(x[n]); }"
where,
writer()andreader()may be called on 2 CPUs without any lock. It's acceptable forreader()to read the written value a little later. Our questions are,Quoted 7 lines
1. because it's impossible for accessing "unsigned char" to cause tearing. So, WRITE_ONCE()/READ_ONCE()/barrier() isn't necessary for correctness, right?
2. we use
barrier()andREAD_ONCE()inwriter()andreader(), because we want to make writing take effect ASAP. Is it a good practice? Or it's a micro-optimization that should be avoided?
Quoted 8 lines
Why don't you consult
Documentation/memory-barriers.txt, especially the section titled "COMPILER BARRIER"?TL;DR:
barrier(),WRITE_ONCE(), andREAD_ONCE()are compiler barriers, not memory barriers. They just restrict compiler optimizations and don't have any effect with regard to "make writing take effect ASAP".
Yes. In theory, this is absolutely correct.
My question is, in practice, will compiler barriers make CPU runs (or sees) memory read/write instructions a little earlier via avoiding to reorder the operations after read/write (e.g., becomes before read/write)?
Quoted 1 line
If you have further questions, please don't hesitate to ask.
--
Best Regards,
Huang, Ying
Re: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
In reply to Ryan RobertsQuoted 39 lines · Ryan Roberts
Quoted 38 lines · Ying
Quoted 36 lines · Ryan Roberts
Hi Huang, Ying,
Quoted 1 line · Ying
Ryan Roberts <[email protected]> writes:
[...]
Quoted 17 lines
Quoted 4 lines
mm/swapfile.c +0
-0@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.Looking at this a bit more closely, I'm not sure this is needed. Even if there is no cluster, the
swap_infois still locked, so unlocking that will act as a barrier. There are a number of other callsites thatmemset(si->swap_map)without an explicit barrier and with theswap_infolocked.Looking at the original commit that added the
WRITE_ONCE()it was worried about a race with readingswap_mapin_swap_info_get(). But that site is now annotated with adata_race(), which will suppress the warning. And I don't believe there are any places that readswap_maplocklessly and depend upon observing ordering between it and other state? So I think the si unlock is sufficient?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm not planning to add
barrier()here. Let me know if you disagree.swap_map[] may be read locklessly in
swap_offset_available_and_locked()in parallel. IIUC,WRITE_ONCE()here is to make the writing take effect as early as possible there.Afraid I'm not convinced by that argument; if it's racing, it's racing - the
It's not a race.
Quoted 6 lines
lockless side needs to be robust (it is). Adding the compiler barrier limits the compiler's options which could lead to slower code in this path. If your argument is that you want to reduce the window where
swap_offset_available_and_locked()could observe a free swap slot but then see that its taken after it gets the si lock, that seems like a micro-optimization to me, which we should avoid if we can.
Yes. I think that it is a micro-optimization too. I had thought that it is a common practice to use WRITE_ONCE()/READ_ONCE() or barrier() in intentional racy data accessing to make the change available as soon as possible. But I may be wrong here.
Quoted 4 lines
By remnoving the
WRITE_ONCE()and using memset, the lockless reader could observe tearing though. I don't think that should cause a problem (because everything is rechecked with under the lock), but if we want to avoid it, then perhaps we just need to loop overWRITE_ONCE()here instead of using memset?
IIUC, in practice that isn't necessary, because type of si->swap_map[] is "unsigned char". It isn't possible to tear "unsigned char". In theory, it may be better to use WRITE_ONCE() because we may change the type of si->swap_map[] at some time (who knows). I don't have a strong opinion here.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 973
Quoted 2 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v4 4/6] mm: swap: Allow storage of all mTHP orders
Quoted 58 lines · Ying
Quoted 39 lines · Ryan Roberts
Quoted 38 lines · Ying
Quoted 36 lines · Ryan Roberts
Hi Huang, Ying,
Quoted 1 line · Ying
Ryan Roberts <[email protected]> writes:
[...]
Quoted 17 lines
Quoted 4 lines
mm/swapfile.c +0
-0@@ -905,17 +961,18 @@ static int scan_swap_map_slots(struct swap_info_struct *si, }if (si->swap_map[offset]) {mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 971
Quoted 10 lines
mm/swapfile.c +2
-2+VM_WARN_ON (order >0 );unlock_cluster (ci);if (!n_ret) goto scan;else goto done; } -WRITE_ONCE (si->swap_map[offset], usage); -inc_cluster_info_page (si, si->cluster_info, offset); +memset (si->swap_map + offset, usage, nr_pages);Add
barrier()here corresponds to originalWRITE_ONCE()?unlock_cluster(ci)may be NOP for some swap devices.Looking at this a bit more closely, I'm not sure this is needed. Even if there is no cluster, the
swap_infois still locked, so unlocking that will act as a barrier. There are a number of other callsites thatmemset(si->swap_map)without an explicit barrier and with theswap_infolocked.Looking at the original commit that added the
WRITE_ONCE()it was worried about a race with readingswap_mapin_swap_info_get(). But that site is now annotated with adata_race(), which will suppress the warning. And I don't believe there are any places that readswap_maplocklessly and depend upon observing ordering between it and other state? So I think the si unlock is sufficient?Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'm not planning to add
barrier()here. Let me know if you disagree.swap_map[] may be read locklessly in
swap_offset_available_and_locked()in parallel. IIUC,WRITE_ONCE()here is to make the writing take effect as early as possible there.Afraid I'm not convinced by that argument; if it's racing, it's racing - the
It's not a race.
Quoted 6 lines
lockless side needs to be robust (it is). Adding the compiler barrier limits the compiler's options which could lead to slower code in this path. If your argument is that you want to reduce the window where
swap_offset_available_and_locked()could observe a free swap slot but then see that its taken after it gets the si lock, that seems like a micro-optimization to me, which we should avoid if we can.
Quoted 4 lines
Yes. I think that it is a micro-optimization too. I had thought that it is a common practice to use WRITE_ONCE()/READ_ONCE() or
barrier()in intentional racy data accessing to make the change available as soon as possible. But I may be wrong here.
My understanding is that WRITE_ONCE() forces the compiler to emit a store at that point in the program; it can't just rely on knowing that it has previously written the same value to that location, it can't reorder the load to later in the program and it must store the whole word atomically so that no tearing can be observed. But given that swap_map is only ever written with the si lock held, I don't believe we require the first two of those semantics. It should be enough to know that the compiler has emitted all the stores (if it determines they are required) prior to the spin_unlock(). I'm not sure about the anti-tearing guarrantee.
Happy to be told I'm wrong here!
Quoted 8 lines
Quoted 5 lines
By remnoving the
WRITE_ONCE()and using memset, the lockless reader could observe tearing though. I don't think that should cause a problem (because everything is rechecked with under the lock), but if we want to avoid it, then perhaps we just need to loop overWRITE_ONCE()here instead of using memset?IIUC, in practice that isn't necessary, because type of si->swap_map[] is "unsigned char". It isn't possible to tear "unsigned char".
In practice, perhaps. But I guess the compiler is free to update the char bit-by-bit if it wants to, if the store is not marked WRITE_ONCE()?
Quoted 4 lines
In
theory, it may be better to useWRITE_ONCE()because we may change the type of si->swap_map[] at some time (who knows). I don't have a strong opinion here.
The way I see it, the precedent is already set; there are a number of places that already use memset to update swap_map. They are all under the si lock, and none use barrier(). If its wrong here, then its wrong in those places too, I believe.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 973
Quoted 3 lines
mm/swapfile.c +1
-0+add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci);--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerPATCH v4 5/6 mm: vmscan: Avoid split during shrink_folio_list()
In reply to Ryan RobertsNow that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.
If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
mm/vmscan.c +5 -4
@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (!list_empty (&folio->_deferred_list) &&split_folio_to_list (folio, folio_list)) goto activate_locked;
Signature · 1 line
2.25.1
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 23 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/*
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1230
Quoted 10 lines
mm/vmscan.c +5
-4-* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&
Hi Ryan,
After reconsidering our previous discussion about PMD-mapped large folios, I've pondered
the possibility of PMD-mapped Transparent Huge Pages (THPs) being mapped by multiple
processes. In such a scenario, if one process decides to unmap a portion of the folio while
others retain the entire mapping, it raises questions about how the system should handle
this situation. Would the large folio be placed in a deferred list? If so, splitting it might not
yield benefits, as neither I/O nor swap slots would increase in this case by not splitting it.
Regarding PTE-mapped large folios, the absence of an indicator like "entire_map" makes it
challenging to identify cases where the entire folio is mapped. Thus, splitting seems to be
the only viable solution in such circumstances.
Quoted 5 lines
mm/vmscan.c +0
-1split_folio_to_list(folio, folio_list)) goto activate_locked; --2.25.1
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 37 lines · Barry Song
Quoted 23 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1230
Quoted 10 lines
mm/vmscan.c +5
-4-* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&Hi Ryan,
After reconsidering our previous discussion about PMD-mapped large
Quoted 8 lines
folios, I've pondered
the possibility of PMD-mapped Transparent Huge Pages (THPs) being mapped by multiple
processes. In such a scenario, if one process decides to unmap a portion of the folio while
others retain the entire mapping, it raises questions about how the system should handle
this situation. Would the large folio be placed in a deferred list?
No - if the large folio is entirely mapped (via PMD), then the folio will not be put on the deferred split list in the first place. See __folio_remove_rmap():
last = (last < ENTIRELY_MAPPED);
means that nr will never be incremented above 0. (_nr_pages_mapped is incremented by ENTIRELY_MAPPED for every PMD map).
Quoted 19 lines
If
so, splitting it might not
yield benefits, as neither I/O nor swap slots would increase in this case by not splitting it.Regarding PTE-mapped large folios, the absence of an indicator like "
entire_map" makes it
challenging to identify cases where the entire folio is mapped. Thus, splitting seems to be
the only viable solution in such circumstances.Quoted 5 lines
mm/vmscan.c +0
-1split_folio_to_list(folio, folio_list)) goto activate_locked; --2.25.1
Thanks
Barry
Barry Song
Reviewed-byRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 48 lines · Ryan Roberts
Quoted 37 lines · Barry Song
Quoted 23 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1230
Quoted 10 lines
mm/vmscan.c +5
-4-* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&Hi Ryan,
After reconsidering our previous discussion about PMD-mapped largeQuoted 8 lines
folios, I've pondered
the possibility of PMD-mapped Transparent Huge Pages (THPs) being mapped by multiple
processes. In such a scenario, if one process decides to unmap a portion of the folio while
others retain the entire mapping, it raises questions about how the system should handle
this situation. Would the large folio be placed in a deferred list?
Quoted 7 lines
No - if the large folio is entirely mapped (via PMD), then the folio will not be put on the deferred split list in the first place. See
__folio_remove_rmap():last = (last < ENTIRELY_MAPPED);means that nr will never be incremented above 0. (
_nr_pages_mappedis incremented byENTIRELY_MAPPEDfor every PMD map).
you are right, I missed this part, we are breaking early in RMAP_LEVEL_PTE. so we won't get to if (nr). Thanks for your clarification. now we get unified code
for both pmd-mapped and pte-mapped large folios. feel free to add,
Quoted 12 lines
Quoted 12 lines
If
so, splitting it might not
yield benefits, as neither I/O nor swap slots would increase in this case by not splitting it.Regarding PTE-mapped large folios, the absence of an indicator like "
entire_map" makes it
challenging to identify cases where the entire folio is mapped. Thus, splitting seems to be
the only viable solution in such circumstances.
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
mm/vmscan.c +0
-1split_folio_to_list(folio, folio_list)) goto activate_locked; --2.25.1
Thanks
Barry
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
In reply to Ryan RobertsQuoted 25 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO.
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1233
Quoted 10 lines
mm/vmscan.c +5
-1+* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (!list_empty (&folio->_deferred_list) &&split_folio_to_list (folio, folio_list)) goto activate_locked;
Not sure if we might have to annotate that with data_race().
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO.
Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().
I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without data_race() in the code base. But list_empty() is implemented like this:
static inline int list_empty(const struct list_head *head)
{
return READ_ONCE(head->next) == head;
}So I assumed the READ_ONCE() makes everything safe without a lock? Perhaps not sufficient for KCSAN?
Quoted 3 lines
Reviewed-by David Hildenbrand <[email protected]>
Thanks!
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:
Quoted 7 lines
static inline int
list_empty(const struct list_head *head){ return READ_ONCE(head->next) == head; }So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?
Yeah, there is only one use of data_race with that list.
It was added in f3ebdf042df4 ("THP: avoid lock when check whether THP is in deferred list").
Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
[1] https://lore.kernel.org/linux-mm/[email protected]/
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Hi Yin Fengwei,
Quoted 59 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:Quoted 7 lines
static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").
Quoted 7 lines
Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Do you have any recollection of why you added the data_race() markup?
Thanks,
Ryan
Quoted 1 line
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 65 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 62 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:Quoted 7 lines
static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
Quoted 6 lines
Quoted 4 lines
If not required, likely we should just cleanup the single user.
Do you have any recollection of why you added the
data_race()markup?
Per my understanding, this is used to mark that the code accesses folio->_deferred_list without lock intentionally, while folio->_deferred_list may be changed in parallel. IIUC, this is what data_race() is used for. Or, my understanding is wrong?
--
Best Regards,
Huang, Ying
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 51 lines · Ying
Quoted 49 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 46 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?
I don't think READ_ONCE() can replace the lock.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?
Yes. This is my understanding also.
Regards
Yin, Fengwei
Quoted 4 lines
--
Best Regards,
Huang, Ying
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 77 lines · Fengwei
Quoted 51 lines · Ying
Quoted 49 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 46 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>--- mm/vmscan.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a data_race() in deferred_split_folio() then, before taking the lock?
It's used a bit inconsistently here.
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 63 lines · David Hildenbrand
Quoted 62 lines · Fengwei
Quoted 59 lines · Ying
Quoted 57 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 54 lines · David Hildenbrand
Quoted 53 lines · Ryan Roberts
Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct
list_head*folio_list,
if (!can_split_folio(folio, NULL)) gotoactivate_locked; /*
- * Split folios without a PMD map right - * away. Chances are some or all of the - * tail pages can be freed without IO. + * Split partially mapped folios map + * right away. Chances are some or all + * of the tail pages can be freed + * without IO.Quoted 8 lines
Quoted 6 lines
*/
- if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&split_folio_to_list(folio, folio_list)) gotoactivate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and
withoutdata_race()in the code base. Butlist_empty()is implemented like this:static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not
sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.
But it doesn't ensure we get a consistent value and that the compiler orders the load correctly. There are lots of patterns in the kernel that use READ_ONCE() without a lock and they don't use data_race() - e.g. ptep_get_lockless().
It sounds like none of us really understand what data_race() is for, so I guess I'll just do a KCSAN build and invoke the code path to see if it complains.
Quoted 28 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a
data_race()indeferred_split_folio()then, before taking the lock?It's used a bit inconsistently here.
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 64 lines · Ryan Roberts
Quoted 63 lines · David Hildenbrand
Quoted 62 lines · Fengwei
Quoted 59 lines · Ying
Quoted 57 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 54 lines · David Hildenbrand
Quoted 53 lines · Ryan Roberts
Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct
list_head*folio_list,
if (!can_split_folio(folio, NULL)) gotoactivate_locked; /*
- * Split folios without a PMD map right - * away. Chances are some or all of the - * tail pages can be freed without IO. + * Split partially mapped folios map + * right away. Chances are some or all + * of the tail pages can be freed + * without IO.Quoted 8 lines
Quoted 6 lines
*/
- if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&split_folio_to_list(folio, folio_list)) gotoactivate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and
withoutdata_race()in the code base. Butlist_empty()is implemented like this:static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not
sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.But it doesn't ensure we get a consistent value and that the compiler orders the
Sorry - fat fingers... I meant it *does* ensure we get a consistent value (i.e. untorn)
Quoted 36 lines
load correctly. There are lots of patterns in the kernel that use
READ_ONCE()without a lock and they don't usedata_race()- e.g.ptep_get_lockless().It sounds like none of us really understand what
data_race()is for, so I guess I'll just do a KCSAN build and invoke the code path to see if it complains.Quoted 28 lines
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a
data_race()indeferred_split_folio()then, before taking the lock?It's used a bit inconsistently here.
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
In reply to Ryan RobertsQuoted 64 lines · Ryan Roberts
Quoted 63 lines · David Hildenbrand
Quoted 62 lines · Fengwei
Quoted 59 lines · Ying
Quoted 57 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 54 lines · David Hildenbrand
Quoted 53 lines · Ryan Roberts
Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct
list_head*folio_list,
if (!can_split_folio(folio, NULL)) gotoactivate_locked; /*
- * Split folios without a PMD map right - * away. Chances are some or all of the - * tail pages can be freed without IO. + * Split partially mapped folios map + * right away. Chances are some or all + * of the tail pages can be freed + * without IO.Quoted 8 lines
Quoted 6 lines
*/
- if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&split_folio_to_list(folio, folio_list)) gotoactivate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and
withoutdata_race()in the code base. Butlist_empty()is implemented like this:static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not
sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.But it doesn't ensure we get a consistent value and that the compiler orders the load correctly. There are lots of patterns in the kernel that use
READ_ONCE()without a lock and they don't usedata_race()- e.g.ptep_get_lockless().
They (ptep_get_lockless() and deferred_list) have different access pattern (or race pattern) here. I don't think they are comparable.
Quoted 3 lines
It sounds like none of us really understand what
data_race()is for, so I guess I'll just do a KCSAN build and invoke the code path to see if it complains.
READ_ONCE() in list_empty will shutdown the KCSAN also.
Quoted 31 lines
Quoted 30 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a
data_race()indeferred_split_folio()then, before taking the lock?It's used a bit inconsistently here.
Ryan Roberts
reviewerRe: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
Quoted 75 lines · Yin Fengwei
Quoted 64 lines · Ryan Roberts
Quoted 63 lines · David Hildenbrand
Quoted 62 lines · Fengwei
Quoted 59 lines · Ying
Quoted 57 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 54 lines · David Hildenbrand
Quoted 53 lines · Ryan Roberts
Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +0
-0@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct
list_head*folio_list,
if (!can_split_folio(folio, NULL)) gotoactivate_locked; /*
- * Split folios without a PMD map right - * away. Chances are some or all of the - * tail pages can be freed without IO. + * Split partially mapped folios map + * right away. Chances are some or all + * of the tail pages can be freed + * without IO.Quoted 8 lines
Quoted 6 lines
*/
- if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) &&split_folio_to_list(folio, folio_list)) gotoactivate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and
withoutdata_race()in the code base. Butlist_empty()is implemented like this:static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not
sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.But it doesn't ensure we get a consistent value and that the compiler orders the load correctly. There are lots of patterns in the kernel that use
READ_ONCE()without a lock and they don't usedata_race()- e.g.ptep_get_lockless().
Quoted 7 lines
They (
ptep_get_lockless()anddeferred_list) have different access pattern (or race pattern) here. I don't think they are comparable.Quoted 3 lines
It sounds like none of us really understand what
data_race()is for, so I guess I'll just do a KCSAN build and invoke the code path to see if it complains.
READ_ONCE()inlist_emptywill shutdown the KCSAN also.
OK, I found some time to run the test with KCSAN; nothing fires.
But then I read the docs and looked at the code a bit. Documentation/dev-tools/kcsan.rst states:
In an execution, two memory accesses form a *data race* if they *conflict*, they happen concurrently in different threads, and at least one of them is a *plain access*; they *conflict* if both access the same memory location, and at least one is a write.
It also clarifies the READ_ONCE() is a "marked access". So we would have a data race if there was a concurrent, *plain* write to folio->_deferred_list.next. This can occur in a couple of places I believe, for example:
deferred_split_folio()
list_add_tail()
__list_add()
new->next = next;deferred_split_scan()
list_move()
list_add()
__list_add()
new->next = next;So if either partially deferred_split_folio() or deferred_split_scan() can run concurrently with shrink_folio_list(), for the same folio (I beleive both can can), then we have a race, and this list_empty() check needs to be protected with data_race(). The race is safe/by design, but it does need to be marked.
I'll fix this in my next version.
Thanks,
Ryan
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
Do you have any recollection of why you added the
data_race()markup?Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a
data_race()indeferred_split_folio()then, before taking the lock?It's used a bit inconsistently here.
Re: [PATCH v4 5/6] mm: vmscan: Avoid split during shrink_folio_list()
In reply to David HildenbrandQuoted 80 lines · David Hildenbrand
Quoted 79 lines · Fengwei
Quoted 51 lines · Ying
Quoted 49 lines · Ryan Roberts
Hi Yin Fengwei,
Quoted 46 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · David Hildenbrand
Quoted 30 lines · Ryan Roberts
Now that swap supports storing all mTHP sizes, avoid splitting large folios before swap-out. This benefits performance of the swap-out path by eliding
split_folio_to_list(), which is expensive, and also sets us up for swapping in large folios in a future series.If the folio is partially mapped, we continue to split it since we want to avoid the extra IO overhead and storage of writing out pages uneccessarily.
Signed-off-by Ryan Roberts <[email protected]>---
mm/vmscan.c| 9 +++++----1 file changed, 5 insertions(+), 4 deletions(-)mm/vmscan.c +4
-3@@ -1222,11 +1222,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!can_split_folio(folio, NULL)) goto activate_locked; /* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split partially mapped folios map +* right away. Chances are some or all +* of the tail pages can be freed +* without IO. Quoted 8 lines
Quoted 6 lines
mm/vmscan.c +1
-1*/ - if (!folio_entire_mapcount(folio) && + if (!list_empty(&folio->_deferred_list) && split_folio_to_list(folio, folio_list))goto
activate_locked;Not sure if we might have to annotate that with
data_race().I asked that exact question to Matthew in another context bt didn't get a response. There are examples of checking if the deferred list is empty with and without
data_race()in the code base. Butlist_empty()is implemented like this:Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
static inline int
list_empty(const struct list_head *head){
returnREAD_ONCE(head->next)== head;
}So I assumed the
READ_ONCE()makes everything safe without a lock? Perhaps not sufficient for KCSAN?I don't think
READ_ONCE()can replace the lock.Quoted 16 lines
Quoted 15 lines
Quoted 6 lines
Yeah, there is only one use of
data_racewith that list.It was added in
f3ebdf042df4("THP: avoid lock when check whether THP is in deferred list").Quoted 7 lines
Looks like that was added right in v1 of that change [1], so my best guess is that it is not actually required.
If not required, likely we should just cleanup the single user.
Do you have any recollection of why you added the
data_race()markup?
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Per my understanding, this is used to mark that the code accesses
folio->_deferred_listwithout lock intentionally, whilefolio->_deferred_listmay be changed in parallel. IIUC, this is whatdata_race()is used for. Or, my understanding is wrong?Yes. This is my understanding also.
Why don't we have a
data_race()indeferred_split_folio()then, before taking the lock?
No idea why there is no data_race() added. But I think we should add data_race().
Regards
Yin, Fengwei
Quoted 3 lines
It's used a bit inconsistently here.
Ryan Roberts
reviewerPATCH v4 6/6 mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
In reply to Ryan RobertsRework madvise_cold_or_pageout_pte_range() to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).
Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)
mm/madvise.c +51 -38
@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;
@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {
@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; +const fpb_t fpb_flags =FPB_IGNORE_DIRTY | +FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL ); + +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* * Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + } }/*
Signature · 1 line
2.25.1
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 18 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
Quoted 7 lines
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Signed-off-by Ryan Roberts <[email protected]>
This looks so much better than our initial RFC.
Thank you for your excellent work!
Quoted 60 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +8
-24@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ;
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 10 lines
mm/madvise.c +6
-4-arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; +const fpb_t fpb_flags =FPB_IGNORE_DIRTY | +FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );
I wonder if we have a quick way to avoid folio_pte_batch() if users are doing madvise() on a portion of a large folio.
Quoted 30 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/*
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;
This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr * PAGE_SIZE)
otherwise, nr won't be 0. so we will increase addr and pte by nr.
Quoted 6 lines
mm/madvise.c +0
-1}/* -- 2.25.1
Overall, LGTM,
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.
Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!
Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);
If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call folio_pte_batch(). Just split it.
Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }
Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for madvise_free_pte_range().
Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}
Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>
Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 110 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.
Quoted 3 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio)
what about
"If (pte_pfn(pte) == folio_pfn(folio) && max_nr >= nr_pages)"
just to account for cases where the user's end address falls within
the middle of a large folio?
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) {
batch_count = 0;
if (need_resched()) {
arch_leave_lazy_mmu_mode();
pte_unmap_unlock(start_pte, ptl);
cond_resched();
goto restart;
}
}We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
Quoted 95 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 113 lines · Barry Song
Quoted 112 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
Quoted 8 lines
Quoted 1 line
if (pte_pfn(pte) == folio_pfn(folio)what about
"If (
pte_pfn(pte)==folio_pfn(folio)&&max_nr>=nr_pages)"just to account for cases where the user's end address falls within
the middle of a large folio?
yes, even better. I'll add this for the next version.
Quoted 14 lines
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) { batch_count = 0; if (need_resched()) { arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; } }
Quoted 3 lines
We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
I thought about that, but the vast majority of the work is per-folio, not per-pte. So I concluded it would be best to continue to increment per-folio.
Quoted 96 lines
Quoted 96 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 140 lines · Ryan Roberts
Quoted 113 lines · Barry Song
Quoted 112 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
Quoted 8 lines
Quoted 1 line
if (pte_pfn(pte) == folio_pfn(folio)what about
"If (
pte_pfn(pte)==folio_pfn(folio)&&max_nr>=nr_pages)"just to account for cases where the user's end address falls within
the middle of a large folio?yes, even better. I'll add this for the next version.
Quoted 14 lines
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) { batch_count = 0; if (need_resched()) { arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; } }
Quoted 6 lines
Quoted 3 lines
We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
I thought about that, but the vast majority of the work is per-folio, not per-pte. So I concluded it would be best to continue to increment per-folio.
Okay. The original patch commit b2f557a21bc8 ("mm/madvise: add cond_resched() in madvise_cold_or_pageout_pte_range()") primarily addressed the real-time wake-up latency issue. MADV_PAGEOUT and MADV_COLD are much less critical compared
to other scenarios where operations like do_anon_page or do_swap_page necessarily need PTL to progress. Therefore, adopting
an approach that relatively aggressively releases the PTL seems to neither harm MADV_PAGEOUT/COLD nor disadvantage
others.
We are slightly increasing the duration of holding the PTL due to the iteration of folio_pte_batch() potentially taking longer than the case of small folios, which do not require it. However, compared to operations like folio_isolate_lru() and folio_deactivate(), this increase seems negligible. Recently, we have actually removed ptep_test_and_clear_young() for MADV_PAGEOUT,
which should also benefit real-time scenarios. Nonetheless, there is a small risk with large folios, such as 1 MiB mTHP, where
we may need to loop 256 times in folio_pte_batch().
I would vote for increasing 'nr' or maybe max(log2(nr), 1) rather than 1 for two reasons:
1. We are not making MADV_PAGEOUT/COLD worse; in fact, we are improving them by reducing the time taken to put the same number of pages into the reclaim list.
2. MADV_PAGEOUT/COLD scenarios are not urgent compared to others that genuinely require the PTL to progress. Moreover,
the majority of time spent on PAGEOUT is actually reclaim_pages().
Quoted 89 lines
Quoted 89 lines
Quoted 89 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 9 lines
Quoted 8 lines
mm/madvise.c +0
-1}/* -- 2.25.1
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 158 lines · Barry Song
Quoted 140 lines · Ryan Roberts
Quoted 113 lines · Barry Song
Quoted 112 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
Quoted 8 lines
Quoted 1 line
if (pte_pfn(pte) == folio_pfn(folio)what about
"If (
pte_pfn(pte)==folio_pfn(folio)&&max_nr>=nr_pages)"just to account for cases where the user's end address falls within
the middle of a large folio?yes, even better. I'll add this for the next version.
Quoted 14 lines
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) { batch_count = 0; if (need_resched()) { arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; } }Quoted 6 lines
Quoted 3 lines
We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
I thought about that, but the vast majority of the work is per-folio, not per-pte. So I concluded it would be best to continue to increment per-folio.
Okay. The original patch commit
b2f557a21bc8("mm/madvise: addcond_resched()inmadvise_cold_or_pageout_pte_range()") primarily addressed the real-time wake-up latency issue.MADV_PAGEOUTandMADV_COLDare much less critical compared
to other scenarios where operations likedo_anon_pageordo_swap_pagenecessarily need PTL to progress. Therefore, adopting
an approach that relatively aggressively releases the PTL seems to neither harm MADV_PAGEOUT/COLD nor disadvantage
others.
Quoted 3 lines
We are slightly increasing the duration of holding the PTL due to the iteration of
folio_pte_batch()potentially taking longer than the case of small folios, which do not require it.
If we can't scan all the PTEs in a page table without dropping the PTL intermittently we have bigger problems. This all works perfectly fine in all the other PTE iterators; see zap_pte_range() for example.
Quoted 7 lines
However, compared
to operations likefolio_isolate_lru()andfolio_deactivate(), this increase seems negligible. Recently, we have actually removedptep_test_and_clear_young()forMADV_PAGEOUT,
which should also benefit real-time scenarios. Nonetheless, there is a small risk with large folios, such as 1 MiB mTHP, where
we may need to loop 256 times infolio_pte_batch().
As I understand it, RT and THP are mutually exclusive. RT can't handle the extra latencies THPs can cause in allocation path, etc. So I don't think you will see a problem here.
Quoted 11 lines
I would vote for increasing 'nr' or maybe
max(log2(nr), 1)rather than 1 for two reasons:1. We are not making MADV_PAGEOUT/COLD worse; in fact, we are improving them by reducing the time taken to put the same number of pages into the reclaim list.
2. MADV_PAGEOUT/COLD scenarios are not urgent compared to others that genuinely require the PTL to progress. Moreover,
the majority of time spent on PAGEOUT is actuallyreclaim_pages().
I understand your logic. But I'd rather optimize for fewer lock acquisitions for the !RT+THP case, since RT+THP is not supported.
Quoted 99 lines
Quoted 90 lines
Quoted 90 lines
Quoted 90 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 9 lines
Quoted 8 lines
mm/madvise.c +0
-1}/* -- 2.25.1
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Thanks
Barry
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 160 lines · Ryan Roberts
Quoted 158 lines · Barry Song
Quoted 140 lines · Ryan Roberts
Quoted 113 lines · Barry Song
Quoted 112 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
Quoted 8 lines
Quoted 1 line
if (pte_pfn(pte) == folio_pfn(folio)what about
"If (
pte_pfn(pte)==folio_pfn(folio)&&max_nr>=nr_pages)"just to account for cases where the user's end address falls within
the middle of a large folio?yes, even better. I'll add this for the next version.
Quoted 14 lines
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) { batch_count = 0; if (need_resched()) { arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; } }Quoted 6 lines
Quoted 3 lines
We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
I thought about that, but the vast majority of the work is per-folio, not per-pte. So I concluded it would be best to continue to increment per-folio.
Okay. The original patch commit
b2f557a21bc8("mm/madvise: addcond_resched()inmadvise_cold_or_pageout_pte_range()") primarily addressed the real-time wake-up latency issue.MADV_PAGEOUTandMADV_COLDare much less critical compared
to other scenarios where operations likedo_anon_pageordo_swap_pagenecessarily need PTL to progress. Therefore, adopting
an approach that relatively aggressively releases the PTL seems to neither harm MADV_PAGEOUT/COLD nor disadvantage
others.
Quoted 7 lines
Quoted 3 lines
We are slightly increasing the duration of holding the PTL due to the iteration of
folio_pte_batch()potentially taking longer than the case of small folios, which do not require it.If we can't scan all the PTEs in a page table without dropping the PTL intermittently we have bigger problems. This all works perfectly fine in all the other PTE iterators; see
zap_pte_range()for example.
I've no doubt about folio_pte_batch(). was just talking about the original rt issue
it might affect.
Quoted 9 lines
Quoted 8 lines
However, compared
to operations likefolio_isolate_lru()andfolio_deactivate(), this increase seems negligible. Recently, we have actually removedptep_test_and_clear_young()forMADV_PAGEOUT,
which should also benefit real-time scenarios. Nonetheless, there is a small risk with large folios, such as 1 MiB mTHP, where
we may need to loop 256 times infolio_pte_batch().
Quoted 3 lines
As I understand it, RT and THP are mutually exclusive. RT can't handle the extra latencies THPs can cause in allocation path, etc. So I don't think you will see a problem here.
I was actually taking a different approach on the phones as obviously we have some
UX(user-experience)/UI/audio related tasks which cannot tolerate allocation latency. with
a TAO-similar optimization(we did it by ext_migratetype for some pageblocks), we actually don't push buddy to do compaction or reclamation for forming 64KiB folio.
We immediately fallback to small folios if a zero-latency 64KiB allocation can't be
obtained from some kinds of pools - ext_migratetype pageblocks.
Quoted 9 lines
Quoted 9 lines
I would vote for increasing 'nr' or maybe
max(log2(nr), 1)rather than 1 for two reasons:1. We are not making MADV_PAGEOUT/COLD worse; in fact, we are improving them by reducing the time taken to put the same number of pages into the reclaim list.
Quoted 6 lines
Quoted 3 lines
2. MADV_PAGEOUT/COLD scenarios are not urgent compared to others that genuinely require the PTL to progress. Moreover,
the majority of time spent on PAGEOUT is actuallyreclaim_pages().I understand your logic. But I'd rather optimize for fewer lock acquisitions for the !RT+THP case, since RT+THP is not supported.
Fair enough. I agree we can postpone this until RT and THP become an available option.
For now, keeping this patch simpler seems to be better.
Quoted 101 lines
Quoted 100 lines
Quoted 91 lines
Quoted 91 lines
Quoted 91 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 9 lines
Quoted 8 lines
mm/madvise.c +0
-1}/* -- 2.25.1
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 186 lines · Barry Song
Quoted 160 lines · Ryan Roberts
Quoted 158 lines · Barry Song
Quoted 140 lines · Ryan Roberts
Quoted 113 lines · Barry Song
Quoted 112 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
Quoted 8 lines
Quoted 1 line
if (pte_pfn(pte) == folio_pfn(folio)what about
"If (
pte_pfn(pte)==folio_pfn(folio)&&max_nr>=nr_pages)"just to account for cases where the user's end address falls within
the middle of a large folio?yes, even better. I'll add this for the next version.
Quoted 14 lines
BTW, another minor issue is here:
if (++batch_count == SWAP_CLUSTER_MAX) { batch_count = 0; if (need_resched()) { arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; } }Quoted 6 lines
Quoted 3 lines
We are increasing 1 for nr ptes, thus, we are holding PTL longer than small folios case? we used to increase 1 for each PTE. Does it matter?
I thought about that, but the vast majority of the work is per-folio, not per-pte. So I concluded it would be best to continue to increment per-folio.
Okay. The original patch commit
b2f557a21bc8("mm/madvise: addcond_resched()inmadvise_cold_or_pageout_pte_range()") primarily addressed the real-time wake-up latency issue.MADV_PAGEOUTandMADV_COLDare much less critical compared
to other scenarios where operations likedo_anon_pageordo_swap_pagenecessarily need PTL to progress. Therefore, adopting
an approach that relatively aggressively releases the PTL seems to neither harm MADV_PAGEOUT/COLD nor disadvantage
others.Quoted 7 lines
Quoted 3 lines
We are slightly increasing the duration of holding the PTL due to the iteration of
folio_pte_batch()potentially taking longer than the case of small folios, which do not require it.If we can't scan all the PTEs in a page table without dropping the PTL intermittently we have bigger problems. This all works perfectly fine in all the other PTE iterators; see
zap_pte_range()for example.I've no doubt about
folio_pte_batch(). was just talking about the original rt issue
it might affect.Quoted 9 lines
Quoted 8 lines
However, compared
to operations likefolio_isolate_lru()andfolio_deactivate(), this increase seems negligible. Recently, we have actually removedptep_test_and_clear_young()forMADV_PAGEOUT,
which should also benefit real-time scenarios. Nonetheless, there is a small risk with large folios, such as 1 MiB mTHP, where
we may need to loop 256 times infolio_pte_batch().Quoted 3 lines
As I understand it, RT and THP are mutually exclusive. RT can't handle the extra latencies THPs can cause in allocation path, etc. So I don't think you will see a problem here.
Quoted 10 lines
I was actually taking a different approach on the phones as obviously we have some
UX(user-experience)/UI/audio related tasks which cannot tolerate allocation latency. with
a TAO-similaroptimization(we did it by ext_migratetype for some pageblocks), we actually don't push buddy to do compaction or reclamation for forming 64KiB folio.
We immediately fallback to small folios if a zero-latency 64KiB allocation can't be
obtained from some kinds of pools -ext_migratetypepageblocks.
You can opt-in to avoiding latency due to compaction, etc. with various settings in /sys/kernel/mm/transparent_hugepage/defrag. That applies to mTHP as well. See Documentation/admin-guide/mm/transhuge.rst. Obviously this is not as useful as the TAO approach because it does nothing to avoid fragmentation in the first place.
The other source of latency for THP allocation, which I believe RT doesn't like, is the cost of zeroing the huge page, IIRC.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
I would vote for increasing 'nr' or maybe
max(log2(nr), 1)rather than 1 for two reasons:1. We are not making MADV_PAGEOUT/COLD worse; in fact, we are improving them by reducing the time taken to put the same number of pages into the reclaim list.
2. MADV_PAGEOUT/COLD scenarios are not urgent compared to others that genuinely require the PTL to progress. Moreover,
the majority of time spent on PAGEOUT is actuallyreclaim_pages().
Quoted 6 lines
Quoted 2 lines
I understand your logic. But I'd rather optimize for fewer lock acquisitions for the !RT+THP case, since RT+THP is not supported.
Fair enough. I agree we can postpone this until RT and THP become an available option.
For now, keeping this patch simpler seems to be better.
OK thanks.
Quoted 102 lines
Quoted 102 lines
Quoted 101 lines
Quoted 92 lines
Quoted 92 lines
Quoted 92 lines
nr = folio_pte_batch(folio, addr, pte, ptent,
max_nr,
fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 62 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }Quoted 6 lines
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 9 lines
Quoted 8 lines
mm/madvise.c +0
-1}/* -- 2.25.1
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Thanks
Barry
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
In reply to Ryan RobertsQuoted 185 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.Quoted 66 lines
Quoted 31 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 7 lines
mm/madvise.c +0
-3if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) {mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 511
Quoted 10 lines
mm/madvise.c +5
-5- ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)
Quoted 5 lines
Quoted 2 lines
otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().
Thanks a lot, Ryan!
I'll make sure to follow a similar pattern for madvise_free_pte_range().
Best,
Lance
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
In reply to Ryan RobertsQuoted 110 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.
Quoted 8 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.
if (folio_test_large(folio)) {[...]
nr = folio_pte_batch(folio, addr, pte, ptent, max_nr,
fpb_flags, NULL);
+ if (folio_estimated_sharers(folio) > 1)
+ continue;
Could we use folio_estimated_sharers as an early exit point here?
if (nr < folio_nr_pages(folio)) {
int err;- if (folio_estimated_sharers(folio) > 1)
- continue;
[...]
Quoted 49 lines
Quoted 49 lines
Quoted 32 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 4 lines
mm/madvise.c +0
-0if (pageout_anon_only_filter && !folio_test_anon (folio))continue ;
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-4-VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte,Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 8 lines
mm/madvise.c +4
-4- tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr);
IIRC, some of the architecture(ex, PPC) don't update TLB with set_pte_at and tlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?
Thanks,
Lance
Quoted 29 lines
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 1 line
mm/madvise.c +1
-0+ }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Hi Lance, Barry,
Sorry - I totally missed this when you originally sent it!
Quoted 127 lines · Lance Yang
Quoted 110 lines · Ryan Roberts
Quoted 25 lines · Barry Song
Quoted 24 lines · Ryan Roberts
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Quoted 8 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.if (folio_test_large(folio)) {[...]
nr = folio_pte_batch(folio, addr, pte, ptent,max_nr,
fpb_flags, NULL);(quoted diff) +2
-0+ if (folio_estimated_sharers(folio) > 1) + continue;
Quoted 1 line
Could we use
folio_estimated_sharersas an early exit point here?
I'm not sure what this is saving where you have it? Did you mean to put it before folio_pte_batch()? Currently it is just saving a single conditional.
But now that I think about it a bit more, I remember why I was originally unconditionally calling folio_pte_batch(). Given its a large folio, if the split fails, we can move the cursor to the pte where the next folio begins so we don't have to iterate through one pte at a time which would cause us to keep calling folio_estimated_sharers(), folio_test_anon(), etc on the same folio until we get to the next boundary.
Of course the common case at this point will be for the split to succeed, but then we are going to iterate over ever single PTE anyway - one way or another they are all fetched into cache. So I feel like its neater not to add the conditionals for calling folio_pte_batch(), and just leave this as I have it here.
Quoted 70 lines
if (nr < folio_nr_pages(folio)) { int err;mm/madvise.c +0
-2-if (folio_estimated_sharers (folio) >1 ) -continue ;Quoted 50 lines
Quoted 50 lines
Quoted 33 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 4 lines
mm/madvise.c +0
-0if (pageout_anon_only_filter && !folio_test_anon (folio))continue ;Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-4-VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte,Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 8 lines
mm/madvise.c +4
-4- tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr);
Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?
Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
Quoted 35 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 1 line
mm/madvise.c +1
-0+ }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 4 lines · Ryan Roberts
Hi Lance, Barry,
Sorry - I totally missed this when you originally sent it!
No worries at all :)
Quoted 130 lines · Lance Yang
Quoted 127 lines · Ryan Roberts
Quoted 110 lines · Barry Song
Quoted 25 lines · Ryan Roberts
Quoted 24 lines
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Quoted 8 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.if (folio_test_large(folio)) {[...]
nr = folio_pte_batch(folio, addr, pte, ptent,max_nr,
fpb_flags, NULL);(quoted diff) +2
-0+ if (folio_estimated_sharers(folio) > 1) + continue;
Quoted 4 lines
Quoted 1 line
Could we use
folio_estimated_sharersas an early exit point here?I'm not sure what this is saving where you have it? Did you mean to put it before
folio_pte_batch()? Currently it is just saving a single conditional.
Apologies for the confusion. I made a diff to provide clarity.
mm/madvise.c +0 -1
@@ -462,12 +462,11 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, fpb_flags,NULL ); -
// Could we use folio_estimated_sharers as an early exit point here?
+ if (folio_estimated_sharers(folio) > 1)
+ continue;
if (nr < folio_nr_pages(folio)) {
int err;- if (folio_estimated_sharers(folio) > 1)
- continue;
if (pageout_anon_only_filter &&
!folio_test_anon(folio))
continue;
if (!folio_trylock(folio))Quoted 83 lines
But now that I think about it a bit more, I remember why I was originally unconditionally calling
folio_pte_batch(). Given its a large folio, if the split fails, we can move the cursor to the pte where the next folio begins so we don't have to iterate through one pte at a time which would cause us to keep callingfolio_estimated_sharers(),folio_test_anon(), etc on the same folio until we get to the next boundary.Of course the common case at this point will be for the split to succeed, but then we are going to iterate over ever single PTE anyway - one way or another they are all fetched into cache. So I feel like its neater not to add the conditionals for calling
folio_pte_batch(), and just leave this as I have it here.Quoted 70 lines
if (nr < folio_nr_pages(folio)) { int err;(quoted diff) +0
-2- if (folio_estimated_sharers(folio) > 1) - continue;Quoted 62 lines
Quoted 62 lines
Quoted 33 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 16 lines
mm/madvise.c +4
-8if (pageout_anon_only_filter && !folio_test_anon (folio))continue ; -VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr);
Quoted 6 lines
Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the architecture(ex, PPC) don't update TLB with ptep_test_and_clear_young()
and tlb_remove_tlb_entry().
In my new patch[1], I use refresh_full_ptes() andtlb_remove_tlb_entries() to batch-update the
access and dirty bits.
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
Lance
Quoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 1 line
mm/madvise.c +1
-0+ }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 164 lines · Lance Yang
Quoted 4 lines · Ryan Roberts
Hi Lance, Barry,
Sorry - I totally missed this when you originally sent it!
No worries at all :)
Quoted 130 lines · Lance Yang
Quoted 127 lines · Ryan Roberts
Quoted 110 lines · Barry Song
Quoted 25 lines · Ryan Roberts
Quoted 24 lines
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Quoted 8 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.if (folio_test_large(folio)) {[...]
nr = folio_pte_batch(folio, addr, pte, ptent,max_nr,
fpb_flags, NULL);(quoted diff) +2
-0+ if (folio_estimated_sharers(folio) > 1) + continue;Quoted 4 lines
Quoted 1 line
Could we use
folio_estimated_sharersas an early exit point here?I'm not sure what this is saving where you have it? Did you mean to put it before
folio_pte_batch()? Currently it is just saving a single conditional.Apologies for the confusion. I made a diff to provide clarity.
mm/madvise.c +2
-3@@ -462,12 +462,11 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, fpb_flags,NULL ); -// Could we use folio_estimated_sharers as an early exit point here? +if (folio_estimated_sharers (folio) >1 ) +continue ;if (nr <folio_nr_pages (folio)) {int err; -if (folio_estimated_sharers (folio) >1 ) -continue ;if (pageout_anon_only_filter && !folio_test_anon (folio))
Quoted 2 lines
continue;
if (!folio_trylock(folio))I'm still not really getting it; with my code, if nr < the folio size, we will try to split and if we estimate that the folio is not exclusive we will avoid locking the folio, etc. If nr == folio size, we will proceed to the precise exclusivity check (which is cheap once we know the folio is fully mapped by this process).
With your change, we will always do the estimated exclusive check then proceed to the precise check; seems like duplication to me?
Quoted 88 lines
Quoted 88 lines
But now that I think about it a bit more, I remember why I was originally unconditionally calling
folio_pte_batch(). Given its a large folio, if the split fails, we can move the cursor to the pte where the next folio begins so we don't have to iterate through one pte at a time which would cause us to keep callingfolio_estimated_sharers(),folio_test_anon(), etc on the same folio until we get to the next boundary.Of course the common case at this point will be for the split to succeed, but then we are going to iterate over ever single PTE anyway - one way or another they are all fetched into cache. So I feel like its neater not to add the conditionals for calling
folio_pte_batch(), and just leave this as I have it here.Quoted 73 lines
if (nr < folio_nr_pages(folio)) { int err;(quoted diff) +0
-2- if (folio_estimated_sharers(folio) > 1) - continue;Quoted 50 lines
Quoted 50 lines
Quoted 33 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 4 lines
mm/madvise.c +0
-0if (pageout_anon_only_filter && !folio_test_anon (folio))continue ;Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-4-VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte,Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 509
Quoted 8 lines
mm/madvise.c +4
-4- tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr);IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?
Quoted 6 lines
Quoted 2 lines
Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().
Err, I assumed tlb_remove_tlb_entry() meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.
Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive. ptep_test_and_clear_young() doesn't suffer that problem.
Quoted 43 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 1 line
mm/madvise.c +1
-0+ }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 176 lines · Ryan Roberts
Quoted 164 lines · Lance Yang
Quoted 4 lines · Ryan Roberts
Hi Lance, Barry,
Sorry - I totally missed this when you originally sent it!
No worries at all :)
Quoted 130 lines · Lance Yang
Quoted 127 lines · Ryan Roberts
Quoted 110 lines · Barry Song
Quoted 25 lines · Ryan Roberts
Quoted 24 lines
Rework
madvise_cold_or_pageout_pte_range()to avoid splitting any large folio that is fully and contiguously mapped in the pageout/cold vm range. This change means that large folios will be maintained all the way to swap storage. This both improves performance during swap-out, by eliding the cost of splitting the folio, and sets us up nicely for maintaining the large folio when it is swapped back in (to be covered in a separate series).Folios that are not fully mapped in the target range are still split, but note that behavior is changed so that if the split fails for any reason (folio locked, shared, etc) we now leave it as is and move to the next pte in the range and continue work on the proceeding folios. Previously any failure of this sort would cause the entire operation to give up and no folios mapped at higher addresses were paged out or made cold. Given large folios are becoming more common, this old behavior would have likely lead to wasted opportunities.
While we are at it, change the code that clears young from the ptes to use
ptep_test_and_clear_young(), which is more efficent than get_and_clear/modify/set, especially for contpte mappings on arm64, where the old approach would require unfolding/refolding and the new approach can be done in place.Quoted 4 lines
Quoted 1 line
Signed-off-by Ryan Roberts <[email protected]>This looks so much better than our initial RFC.
Thank you for your excellent work!Thanks - its a team effort - I had your PoC and David's previous batching work to use as a template.
Quoted 66 lines
Quoted 66 lines
--- mm/madvise.c | 89 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 51 insertions(+), 38 deletions(-)mm/madvise.c +9
-28@@ -336,6 +336,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,LIST_HEAD (folio_list);bool pageout_anon_only_filter;unsigned int batch_count =0 ; +int nr;if (fatal_signal_pending (current))return -EINTR ;@@ -423,7 +424,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,return 0 ;flush_tlb_batched_pending (mm);arch_enter_lazy_mmu_mode (); -for (; addr < end; pte++, addr +=PAGE_SIZE ) { +for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { + nr =1 ; ptent =ptep_get (pte);if (++batch_count ==SWAP_CLUSTER_MAX ) {@@ -447,55 +449,66 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,continue ;/* -* Creating a THP page is expensive so split it only if we -* are sure it's worth. Split it if we are only owner. +* If we encounter a large folio, only split it if it is not +* fully mapped within the range we are operating on. Otherwise +* leave it as is so that it can be swapped out whole. If we +* fail to split a folio, leave it in place and advance to the +* next pte in the range. */ if (folio_test_large (folio)) { -int err; - -if (folio_estimated_sharers (folio) >1 ) -break ; -if (pageout_anon_only_filter && !folio_test_anon (folio)) -break ; -if (!folio_trylock (folio)) -break ; -folio_get (folio); -arch_leave_lazy_mmu_mode (); -pte_unmap_unlock (start_pte, ptl); - start_pte =NULL ; - err =split_folio (folio); -folio_unlock (folio); -folio_put (folio); -if (err) -break ; - start_pte = pte = -pte_offset_map_lock (mm, pmd, addr, &ptl); -if (!start_pte) -break ; -arch_enter_lazy_mmu_mode (); - pte--; - addr -=PAGE_SIZE ; -continue ; + const fpb_t fpb_flags = FPB_IGNORE_DIRTY |Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 464
Quoted 5 lines
mm/madvise.c +5
-0+FPB_IGNORE_SOFT_DIRTY ; +int max_nr = (end - addr) /PAGE_SIZE ; + + nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, + fpb_flags,NULL );I wonder if we have a quick way to avoid
folio_pte_batch()if users are doingmadvise()on a portion of a large folio.Quoted 8 lines
Good idea. Something like this?:
if (pte_pfn(pte) == folio_pfn(folio) nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL);If we are not mapping the first page of the folio, then it can't be a full mapping, so no need to call
folio_pte_batch(). Just split it.if (folio_test_large(folio)) {[...]
nr = folio_pte_batch(folio, addr, pte, ptent,max_nr,
fpb_flags, NULL);(quoted diff) +2
-0+ if (folio_estimated_sharers(folio) > 1) + continue;Quoted 4 lines
Quoted 1 line
Could we use
folio_estimated_sharersas an early exit point here?I'm not sure what this is saving where you have it? Did you mean to put it before
folio_pte_batch()? Currently it is just saving a single conditional.Apologies for the confusion. I made a diff to provide clarity.
mm/madvise.c +2
-3@@ -462,12 +462,11 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, nr =folio_pte_batch (folio, addr, pte, ptent, max_nr, fpb_flags,NULL ); -// Could we use folio_estimated_sharers as an early exit point here? +if (folio_estimated_sharers (folio) >1 ) +continue ;if (nr <folio_nr_pages (folio)) {int err; -if (folio_estimated_sharers (folio) >1 ) -continue ;if (pageout_anon_only_filter && !folio_test_anon (folio))Quoted 2 lines
continue; if (!folio_trylock(folio))I'm still not really getting it; with my code, if nr < the folio size, we will try to split and if we estimate that the folio is not exclusive we will avoid locking the folio, etc. If nr == folio size, we will proceed to the precise exclusivity check (which is cheap once we know the folio is fully mapped by this process).
Quoted 2 lines
With your change, we will always do the estimated exclusive check then proceed to the precise check; seems like duplication to me?
Agreed. The estimated exclusive check is indeed redundant with my change.
Quoted 104 lines
Quoted 89 lines
Quoted 89 lines
But now that I think about it a bit more, I remember why I was originally unconditionally calling
folio_pte_batch(). Given its a large folio, if the split fails, we can move the cursor to the pte where the next folio begins so we don't have to iterate through one pte at a time which would cause us to keep callingfolio_estimated_sharers(),folio_test_anon(), etc on the same folio until we get to the next boundary.Of course the common case at this point will be for the split to succeed, but then we are going to iterate over ever single PTE anyway - one way or another they are all fetched into cache. So I feel like its neater not to add the conditionals for calling
folio_pte_batch(), and just leave this as I have it here.Quoted 73 lines
if (nr < folio_nr_pages(folio)) { int err;(quoted diff) +0
-2- if (folio_estimated_sharers(folio) > 1) - continue;Quoted 50 lines
Quoted 50 lines
Quoted 33 lines
mm/madvise.c +27
-0+ +if (nr <folio_nr_pages (folio)) { +int err; + +if (folio_estimated_sharers (folio) >1 ) +continue ; +if (pageout_anon_only_filter && !folio_test_anon (folio)) +continue ; +if (!folio_trylock (folio)) +continue ; +folio_get (folio); +arch_leave_lazy_mmu_mode (); +pte_unmap_unlock (start_pte, ptl); + start_pte =NULL ; + err =split_folio (folio); +folio_unlock (folio); +folio_put (folio); +if (err) +continue ; + start_pte = pte = +pte_offset_map_lock (mm, pmd, addr, &ptl); +if (!start_pte) +break ; +arch_enter_lazy_mmu_mode (); + nr =0 ; +continue ; + } }/* mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 502
Quoted 10 lines
mm/madvise.c +5
-2* Do not interfere with other mappings of this folio and -* non-LRU folio. +* non-LRU folio. If we have a large folio at this point, we +* know it is fully mapped so if its mapcount is the same as its +* number of pages, it must be exclusive. */ -if (!folio_test_lru (folio) ||folio_mapcount (folio) !=1 ) +if (!folio_test_lru (folio) || +folio_mapcount (folio) !=folio_nr_pages (folio))continue ;This looks so perfect and is exactly what I wanted to achieve.
Quoted 4 lines
mm/madvise.c +0
-0if (pageout_anon_only_filter && !folio_test_anon (folio))continue ;Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-4-VM_BUG_ON_FOLIO (folio_test_large (folio), folio); - -if (!pageout &&pte_young (ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte,Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 509
Quoted 8 lines
mm/madvise.c +4
-4- tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr);IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?Quoted 6 lines
Quoted 2 lines
Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().Err, I assumed
tlb_remove_tlb_entry()meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.
Quoted 5 lines
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive.
ptep_test_and_clear_young()doesn't suffer that problem.
Thanks for explaining. I got it.
I think that other architectures will benefit from the per-pte clear-modify-set approach. IMO, refresh_full_ptes() can be overridden by arm64.
Thanks,
Lance
Quoted 45 lines
Quoted 44 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 510
Quoted 1 line
mm/madvise.c +1
-0+ }This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
mm/madvise.c +0
-0}Quoted 8 lines
Quoted 4 lines
mm/madvise.c +0
-1/* -- 2.25.1
Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
(quoted diff) +0
-4- VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - if (!pageout && pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte,Quoted 8 lines
Quoted 8 lines
Review
Quoted 8 lines
(quoted diff) +4
-4- tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + if (!pageout) { + for (; nr != 0; nr--, pte++, addr += PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) + tlb_remove_tlb_entry(tlb, pte, addr);IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().
Afraid I'm still struggling with this comment. Do you mean to say that powerpc invalidates the TLB entry as part of the call to ptep_test_and_clear_young()? So tlb_remove_tlb_entry() would be redundant here, and likely cause performance degradation on that architecture?
IMHO, ptep_test_and_clear_young() really shouldn't be invalidating the TLB entry, that's what ptep_clear_flush_young() is for.
But I do see that for some cases of the 32-bit ppc, there appears to be a flush:
#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline int __ptep_test_and_clear_young(struct mm_struct *mm,
unsigned long addr, pte_t *ptep)
{
unsigned long old;
old = pte_update(mm, addr, ptep, _PAGE_ACCESSED, 0, 0);
if (old & _PAGE_HASHPTE)
flush_hash_entry(mm, ptep, addr); <<<<<<<<
return (old & _PAGE_ACCESSED) != 0;
}
#define ptep_test_and_clear_young(__vma, __addr, __ptep) \
__ptep_test_and_clear_young((__vma)->vm_mm, __addr, __ptep)Is that what you are describing? Does any anyone know why flush_hash_entry() is called? I'd say that's a bug in ppc and not a reason not to use ptep_test_and_clear_young() in the common code!
Thanks,
Ryan
Quoted 67 lines
Quoted 9 lines
Err, I assumed
tlb_remove_tlb_entry()meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.Quoted 5 lines
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive.
ptep_test_and_clear_young()doesn't suffer that problem.Thanks for explaining. I got it.
I think that other architectures will benefit from the per-pte clear-modify-set approach. IMO,
refresh_full_ptes()can be overridden by arm64.Thanks,
LanceQuoted 45 lines
Quoted 44 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
Quoted 1 line
+ }
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
}Quoted 8 lines
Quoted 4 lines
/*--
2.25.1Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 21 lines · Ryan Roberts
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines
Quoted 14 lines
Quoted 13 lines
Quoted 13 lines
Review
Quoted 13 lines
(quoted diff) +4
-8- VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - if (!pageout && pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + if (!pageout) { + for (; nr != 0; nr--, pte++, addr += PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) + tlb_remove_tlb_entry(tlb, pte, addr);Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().Afraid I'm still struggling with this comment. Do you mean to say that powerpc invalidates the TLB entry as part of the call to
ptep_test_and_clear_young()? Sotlb_remove_tlb_entry()would be redundant here, and likely cause performance degradation on that architecture?
I just thought that using ptep_test_and_clear_young() instead of ptep_get_and_clear_full() + pte_mkold() might not be correct. However, it's most likely that I was mistaken :(
I also have a question. Why aren't we using ptep_test_and_clear_young() in madvise_cold_or_pageout_pte_range(), but instead ptep_get_and_clear_full() + pte_mkold() as we did previously.
/*
* Some of architecture(ex, PPC) don't update TLB
* with set_pte_at and tlb_remove_tlb_entry so for
* the portability, remap the pte with old|clean
* after pte clearing.
*/
According to this comment from madvise_free_pte_range. IIUC, we need to call ptep_get_and_clear_full() to clear the PTE, and then remap the PTE with old|clean.
Thanks,
Lance
Quoted 96 lines
IMHO,
ptep_test_and_clear_young()really shouldn't be invalidating the TLB entry, that's whatptep_clear_flush_young()is for.But I do see that for some cases of the 32-bit ppc, there appears to be a flush:
#define
__HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline int __ptep_test_and_clear_young(structmm_struct*mm,
unsigned long addr,pte_t*ptep){ unsigned long old; old = pte_update(mm, addr, ptep, _PAGE_ACCESSED, 0, 0); if (old & _PAGE_HASHPTE) flush_hash_entry(mm, ptep, addr); <<<<<<<< return (old & _PAGE_ACCESSED) != 0; } #define ptep_test_and_clear_young(__vma, __addr, __ptep) \ __ptep_test_and_clear_young((__vma)->vm_mm, __addr, __ptep)Is that what you are describing? Does any anyone know why
flush_hash_entry()is called? I'd say that's a bug in ppc and not a reason not to useptep_test_and_clear_young()in the common code!Thanks,
RyanQuoted 67 lines
Quoted 9 lines
Err, I assumed
tlb_remove_tlb_entry()meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.Quoted 5 lines
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive.
ptep_test_and_clear_young()doesn't suffer that problem.Thanks for explaining. I got it.
I think that other architectures will benefit from the per-pte clear-modify-set approach. IMO,
refresh_full_ptes()can be overridden by arm64.Thanks,
LanceQuoted 45 lines
Quoted 44 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
Quoted 1 line
+ }
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
}Quoted 8 lines
Quoted 4 lines
/*--
2.25.1Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 26 lines · Lance Yang
Quoted 25 lines · Ryan Roberts
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +4
-8- VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - if (!pageout && pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + if (!pageout) { + for (; nr != 0; nr--, pte++, addr += PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) + tlb_remove_tlb_entry(tlb, pte, addr);Quoted 6 lines
Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().
Quoted 8 lines
Quoted 4 lines
Afraid I'm still struggling with this comment. Do you mean to say that powerpc invalidates the TLB entry as part of the call to
ptep_test_and_clear_young()? Sotlb_remove_tlb_entry()would be redundant here, and likely cause performance degradation on that architecture?I just thought that using
ptep_test_and_clear_young()instead ofptep_get_and_clear_full()+pte_mkold()might not be correct. However, it's most likely that I was mistaken :(
OK, I'm pretty confident that my usage is correct.
Quoted 11 lines
I also have a question. Why aren't we using
ptep_test_and_clear_young()inmadvise_cold_or_pageout_pte_range(), but insteadptep_get_and_clear_full()+pte_mkold()as we did previously./*
* Some ofarchitecture(ex, PPC)don't update TLB
* withset_pte_atandtlb_remove_tlb_entryso for
* the portability, remap the pte with old|clean
* after pte clearing.
*/
Ahh, I see; this is a comment from madvise_free_pte_range() I don't quite understand that comment. I suspect it might be out of date, or saying that doing set_pte_at(pte_mkold(ptep_get(ptent))) is not correct because it is not atomic and the HW could set the dirty bit between the get and the set. Doing the atomic ptep_get_and_clear_full() means you go via a pte_none() state, so if the TLB is racing it will see the entry isn't valid and fault.
Note that madvise_free_pte_range() is trying to clear both the access and dirty bits, whereas madvise_cold_or_pageout_pte_range() is only trying to clear the access bit. There is a special helper to clear the access bit atomically - ptep_test_and_clear_young() - but there is no helper to clear the access *and* dirty bit, I don't believe. There is ptep_set_access_flags(), but that sets flags to a "more permissive setting" (i.e. allows setting the flags, not clearing them). Perhaps this constraint can be relaxed given we will follow up with an explicit TLBI - it would require auditing all the implementations.
Quoted 104 lines
According to this comment from
madvise_free_pte_range. IIUC, we need to callptep_get_and_clear_full()to clear the PTE, and then remap the PTE with old|clean.Thanks,
LanceQuoted 96 lines
IMHO,
ptep_test_and_clear_young()really shouldn't be invalidating the TLB entry, that's whatptep_clear_flush_young()is for.But I do see that for some cases of the 32-bit ppc, there appears to be a flush:
#define
__HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline int __ptep_test_and_clear_young(structmm_struct*mm,
unsigned long addr,pte_t*ptep){ unsigned long old; old = pte_update(mm, addr, ptep, _PAGE_ACCESSED, 0, 0); if (old & _PAGE_HASHPTE) flush_hash_entry(mm, ptep, addr); <<<<<<<< return (old & _PAGE_ACCESSED) != 0; } #define ptep_test_and_clear_young(__vma, __addr, __ptep) \ __ptep_test_and_clear_young((__vma)->vm_mm, __addr, __ptep)Is that what you are describing? Does any anyone know why
flush_hash_entry()is called? I'd say that's a bug in ppc and not a reason not to useptep_test_and_clear_young()in the common code!Thanks,
RyanQuoted 67 lines
Quoted 9 lines
Err, I assumed
tlb_remove_tlb_entry()meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.Quoted 5 lines
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive.
ptep_test_and_clear_young()doesn't suffer that problem.Thanks for explaining. I got it.
I think that other architectures will benefit from the per-pte clear-modify-set approach. IMO,
refresh_full_ptes()can be overridden by arm64.Thanks,
LanceQuoted 45 lines
Quoted 44 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
Quoted 1 line
+ }
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
}Quoted 8 lines
Quoted 4 lines
/*--
2.25.1Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 51 lines · Ryan Roberts
Quoted 26 lines · Lance Yang
Quoted 25 lines · Ryan Roberts
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +4
-8- VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - - if (!pageout && pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + if (!pageout) { + for (; nr != 0; nr--, pte++, addr += PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) + tlb_remove_tlb_entry(tlb, pte, addr);Quoted 6 lines
Quoted 3 lines
IIRC, some of the
architecture(ex, PPC)don't update TLB withset_pte_atandtlb_remove_tlb_entry. So, didn't we consider remapping the PTE with old after pte clearing?Sorry Lance, I don't understand this question, can you rephrase? Are you saying there is a good reason to do the original clear-mkold-set for some arches?
IIRC, some of the
architecture(ex, PPC)don't update TLB withptep_test_and_clear_young()
andtlb_remove_tlb_entry().Quoted 8 lines
Quoted 4 lines
Afraid I'm still struggling with this comment. Do you mean to say that powerpc invalidates the TLB entry as part of the call to
ptep_test_and_clear_young()? Sotlb_remove_tlb_entry()would be redundant here, and likely cause performance degradation on that architecture?I just thought that using
ptep_test_and_clear_young()instead ofptep_get_and_clear_full()+pte_mkold()might not be correct. However, it's most likely that I was mistaken :(OK, I'm pretty confident that my usage is correct.
Quoted 11 lines
I also have a question. Why aren't we using
ptep_test_and_clear_young()inmadvise_cold_or_pageout_pte_range(), but insteadptep_get_and_clear_full()+pte_mkold()as we did previously./*
* Some ofarchitecture(ex, PPC)don't update TLB
* withset_pte_atandtlb_remove_tlb_entryso for
* the portability, remap the pte with old|clean
* after pte clearing.
*/
Quoted 6 lines
Ahh, I see; this is a comment from
madvise_free_pte_range()I don't quite understand that comment. I suspect it might be out of date, or saying that doing set_pte_at(pte_mkold(ptep_get(ptent))) is not correct because it is not atomic and the HW could set the dirty bit between the get and the set. Doing the atomicptep_get_and_clear_full()means you go via apte_none()state, so if the TLB is racing it will see the entry isn't valid and fault.
Thanks for your analysis and explanations!
Quoted 9 lines
Note that
madvise_free_pte_range()is trying to clear both the access and dirty bits, whereasmadvise_cold_or_pageout_pte_range()is only trying to clear the access bit. There is a special helper to clear the access bit atomically -ptep_test_and_clear_young()- but there is no helper to clear the access *and* dirty bit, I don't believe. There isptep_set_access_flags(), but that sets flags to a "more permissive setting" (i.e. allows setting the flags, not clearing them). Perhaps this constraint can be relaxed given we will follow up with an explicit TLBI - it would require auditing all the implementations.
Thanks for bringing this! I'll take a closer look at it.
Thanks again for your time!
Lance
Quoted 106 lines
Quoted 105 lines
According to this comment from
madvise_free_pte_range. IIUC, we need to callptep_get_and_clear_full()to clear the PTE, and then remap the PTE with old|clean.Thanks,
LanceQuoted 96 lines
IMHO,
ptep_test_and_clear_young()really shouldn't be invalidating the TLB entry, that's whatptep_clear_flush_young()is for.But I do see that for some cases of the 32-bit ppc, there appears to be a flush:
#define
__HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG
static inline int __ptep_test_and_clear_young(structmm_struct*mm,
unsigned long addr,pte_t*ptep){ unsigned long old; old = pte_update(mm, addr, ptep, _PAGE_ACCESSED, 0, 0); if (old & _PAGE_HASHPTE) flush_hash_entry(mm, ptep, addr); <<<<<<<< return (old & _PAGE_ACCESSED) != 0; } #define ptep_test_and_clear_young(__vma, __addr, __ptep) \ __ptep_test_and_clear_young((__vma)->vm_mm, __addr, __ptep)Is that what you are describing? Does any anyone know why
flush_hash_entry()is called? I'd say that's a bug in ppc and not a reason not to useptep_test_and_clear_young()in the common code!Thanks,
RyanQuoted 67 lines
Quoted 9 lines
Err, I assumed
tlb_remove_tlb_entry()meant "invalidate the TLB entry for this address please" - albeit its deferred and batched. I'll look into this.Quoted 4 lines
In my new patch[1], I use
refresh_full_ptes()andtlb_remove_tlb_entries()to batch-update the
access and dirty bits.Quoted 5 lines
I want to avoid the per-pte clear-modify-set approach, because this doesn't perform well on arm64 when using contpte mappings; it will cause the contpe mapping to be unfolded by the first clear that touches the contpte block, then refolded by the last set to touch the block. That's expensive.
ptep_test_and_clear_young()doesn't suffer that problem.Thanks for explaining. I got it.
I think that other architectures will benefit from the per-pte clear-modify-set approach. IMO,
refresh_full_ptes()can be overridden by arm64.Thanks,
LanceQuoted 45 lines
Quoted 44 lines
[1] https://lore.kernel.org/linux-mm/[email protected]
Thanks,
LanceQuoted 37 lines
Quoted 36 lines
Thanks,
LanceQuoted 29 lines
Quoted 8 lines
Quoted 1 line
+ }
This looks so smart. if it is not pageout, we have increased pte and addr here; so nr is 0 and we don't need to increase again in for (; addr < end; pte += nr, addr += nr *
PAGE_SIZE)otherwise, nr won't be 0. so we will increase addr and pte by nr.
Indeed. I'm hoping that Lance is able to follow a similar pattern for
madvise_free_pte_range().Quoted 4 lines
Quoted 4 lines
}Quoted 8 lines
Quoted 4 lines
/*--
2.25.1Overall, LGTM,
Reviewed-by Barry Song <[email protected]>Thanks!
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
In reply to Ryan Robertsmm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 512
Quoted 12 lines
mm/madvise.c +5
-6-if (!pageout &&pte_young (ptent)) { - ptent =ptep_get_and_clear_full (mm, addr, pte, - tlb->fullmm); - ptent =pte_mkold (ptent); -set_pte_at (mm, addr, pte, ptent); -tlb_remove_tlb_entry (tlb, pte, addr); +if (!pageout) { +for (; nr !=0 ; nr--, pte++, addr +=PAGE_SIZE ) { +if (ptep_test_and_clear_young (vma, addr, pte)) +tlb_remove_tlb_entry (tlb, pte, addr); + } }
The following might turn out a bit nicer: Make folio_pte_batch() collect "any_young", then doing something like we do with "any_writable" in the fork() case:
...
nr = folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL, any_young); if (any_young) pte_mkyoung(ptent)
...
if (!pageout && pte_young(ptent)) {
mkold_full_ptes(mm, addr, pte, nr, tlb->fullmm);
tlb_remove_tlb_entries(tlb, pte, nr, addr);
}Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 25 lines · David Hildenbrand
Quoted 4 lines
- if (!pageout &&
pte_young(ptent)) {
- ptent =ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm);
- ptent =pte_mkold(ptent);Quoted 8 lines
-
set_pte_at(mm, addr, pte, ptent);
-tlb_remove_tlb_entry(tlb, pte, addr);
+ if (!pageout) {
+ for (; nr != 0; nr--, pte++, addr +=PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) +tlb_remove_tlb_entry(tlb, pte, addr); + }
}The following might turn out a bit nicer: Make
folio_pte_batch()collect "any_young", then doing something like we do with "any_writable" in thefork()case:...
nr =folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL, any_young);
if (any_young)
pte_mkyoung(ptent)
...
Quoted 5 lines
if (!pageout &&
pte_young(ptent)) {mkold_full_ptes(mm, addr, pte, nr, tlb->fullmm);tlb_remove_tlb_entries(tlb, pte, nr, addr); }
I thought about that but decided that it would be better to only TLBI the actual entries that were young. Although looking at tlb_remove_tlb_entry() I see that it just maintains a range between the lowest and highest address, so this won't actually make any difference.
So, yes, this will be a nice improvement, and also prevent the O(n^2) pte reads for the contpte case. I'll change in the next version.
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 32 lines · Ryan Roberts
Quoted 25 lines · David Hildenbrand
Quoted 4 lines
- if (!pageout &&
pte_young(ptent)) {
- ptent =ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm);
- ptent =pte_mkold(ptent);Quoted 8 lines
-
set_pte_at(mm, addr, pte, ptent);
-tlb_remove_tlb_entry(tlb, pte, addr);
+ if (!pageout) {
+ for (; nr != 0; nr--, pte++, addr +=PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) +tlb_remove_tlb_entry(tlb, pte, addr); + }
}The following might turn out a bit nicer: Make
folio_pte_batch()collect "any_young", then doing something like we do with "any_writable" in thefork()case:...
nr =folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL, any_young);
if (any_young)
pte_mkyoung(ptent)
...Quoted 5 lines
if (!pageout &&
pte_young(ptent)) {
mkold_full_ptes(mm, addr, pte, nr,tlb->fullmm);tlb_remove_tlb_entries(tlb, pte, nr, addr);
}
Quoted 4 lines
I thought about that but decided that it would be better to only TLBI the actual entries that were young. Although looking at
tlb_remove_tlb_entry()I see that it just maintains a range between the lowest and highest address, so this won't actually make any difference.
Yes, tlb_remove_tlb_entries() looks scarier than it actually is :)
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
In reply to David HildenbrandQuoted 25 lines · David Hildenbrand
Quoted 4 lines
- if (!pageout &&
pte_young(ptent)) {
- ptent =ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm);
- ptent =pte_mkold(ptent);Quoted 8 lines
-
set_pte_at(mm, addr, pte, ptent);
-tlb_remove_tlb_entry(tlb, pte, addr);
+ if (!pageout) {
+ for (; nr != 0; nr--, pte++, addr +=PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) +tlb_remove_tlb_entry(tlb, pte, addr); + }
}The following might turn out a bit nicer: Make
folio_pte_batch()collect "any_young", then doing something like we do with "any_writable" in thefork()case:...
nr =folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL, any_young);
if (any_young)
pte_mkyoung(ptent)
...
Quoted 2 lines
if (!pageout &&
pte_young(ptent)) {mkold_full_ptes(mm, addr, pte, nr, tlb->fullmm);
I don't think tlb->fullmm makes sense here because we are not clearing the pte, so there is no chance of optimization? So planning to call this mkold_ptes() and remove that param. Have I missed something?
Quoted 3 lines
tlb_remove_tlb_entries(tlb, pte, nr, addr);
}
Re: [PATCH v4 6/6] mm: madvise: Avoid split during MADV_PAGEOUT and MADV_COLD
Quoted 26 lines · Ryan Roberts
Quoted 25 lines · David Hildenbrand
Quoted 4 lines
- if (!pageout &&
pte_young(ptent)) {
- ptent =ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm);
- ptent =pte_mkold(ptent);Quoted 8 lines
-
set_pte_at(mm, addr, pte, ptent);
-tlb_remove_tlb_entry(tlb, pte, addr);
+ if (!pageout) {
+ for (; nr != 0; nr--, pte++, addr +=PAGE_SIZE) { + if (ptep_test_and_clear_young(vma, addr, pte)) +tlb_remove_tlb_entry(tlb, pte, addr); + }
}The following might turn out a bit nicer: Make
folio_pte_batch()collect "any_young", then doing something like we do with "any_writable" in thefork()case:...
nr =folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, NULL, any_young);
if (any_young)
pte_mkyoung(ptent)
...
Quoted 6 lines
Quoted 2 lines
if (!pageout &&
pte_young(ptent)) {
mkold_full_ptes(mm, addr, pte, nr,tlb->fullmm);I don't think
tlb->fullmmmakes sense here because we are not clearing the pte, so there is no chance of optimization? So planning to call thismkold_ptes()and remove that param. Have I missed something?
Agreed.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 45 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |
Quoted 2 lines
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
I don't understand why the performance of 2M THP improves. The swap entry allocation becomes a little slower. Can you provide some perf-profile to root cause it?
--
Best Regards,
Huang, Ying
Quoted 86 lines
This test also acts as a good stress test for swap and, more generally mm. A couple of existing bugs were found as a result [5] [6].
---
The series applies against mm-unstable (d7182786dd0a). Although I've additionally been running with a couple of extra fixes to avoid the issues at [6].Changes since v3 [3]
====================- Renamed
SWAP_NEXT_NULL->SWAP_NEXT_INVALID(per Huang, Ying)
- Simplified max offset calculation (per Huang, Ying)
- Reinstated structpercpu_clusterto contain per-cluster, per-ordernextoffset (per Huang, Ying)
- Removedswap_alloc_large()and merged its functionality intoscan_swap_map_slots()(per Huang, Ying)
- Avoid extra cost of folio ref and lock due to removal ofCLUSTER_FLAG_HUGEby freeing swap entries in batches (see patch 2) (per DavidH)
- vmscan splits folio if its partially mapped (per Barry Song, DavidH)
- Avoid splitting inMADV_PAGEOUTpath (per Barry Song)
- Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch since it's not actually a problem for THP as I first thought.Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
RyanRyan Roberts (6):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap:free_swap_and_cache_nr()as batchedfree_swap_and_cache()
mm: swap: Simplify structpercpu_cluster
mm: swap: Allow storage of all mTHP orders
mm: vmscan: Avoid split duringshrink_folio_list()
mm: madvise: Avoid split duringMADV_PAGEOUTandMADV_COLDinclude/linux/pgtable.h | 28 ++++ include/linux/swap.h | 33 +++-- mm/huge_memory.c | 3 - mm/internal.h | 48 +++++++ mm/madvise.c | 101 ++++++++------ mm/memory.c | 13 +- mm/swapfile.c | 298 ++++++++++++++++++++++------------------ mm/vmscan.c | 9 +- 8 files changed, 332 insertions(+), 201 deletions(-)--
2.25.1
Ryan Roberts
reviewerQuoted 47 lines · Ying
Quoted 45 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |
Quoted 6 lines
Quoted 2 lines
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
I don't understand why the performance of 2M THP improves. The swap entry allocation becomes a little slower. Can you provide some perf-profile to root cause it?
I didn't post the stdev, which is quite large (~10%), so that may explain some of it:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 5.5% | | base-64K | -14.6% | 3.8% | | base-2M | 87.4% | 10.6% | | v4-4K | 1.4% | 3.7% | | v4-64K | 44.2% | 11.8% | | v4-2M | 97.7% | 13.3% |
Regardless, I'll do some perf profiling and post results shortly.
Quoted 91 lines
--
Best Regards,
Huang, YingQuoted 86 lines
This test also acts as a good stress test for swap and, more generally mm. A couple of existing bugs were found as a result [5] [6].
---
The series applies against mm-unstable (d7182786dd0a). Although I've additionally been running with a couple of extra fixes to avoid the issues at [6].Changes since v3 [3]
====================- Renamed
SWAP_NEXT_NULL->SWAP_NEXT_INVALID(per Huang, Ying)
- Simplified max offset calculation (per Huang, Ying)
- Reinstated structpercpu_clusterto contain per-cluster, per-ordernextoffset (per Huang, Ying)
- Removedswap_alloc_large()and merged its functionality intoscan_swap_map_slots()(per Huang, Ying)
- Avoid extra cost of folio ref and lock due to removal ofCLUSTER_FLAG_HUGEby freeing swap entries in batches (see patch 2) (per DavidH)
- vmscan splits folio if its partially mapped (per Barry Song, DavidH)
- Avoid splitting inMADV_PAGEOUTpath (per Barry Song)
- Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch since it's not actually a problem for THP as I first thought.Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
RyanRyan Roberts (6):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap:free_swap_and_cache_nr()as batchedfree_swap_and_cache()
mm: swap: Simplify structpercpu_cluster
mm: swap: Allow storage of all mTHP orders
mm: vmscan: Avoid split duringshrink_folio_list()
mm: madvise: Avoid split duringMADV_PAGEOUTandMADV_COLDinclude/linux/pgtable.h | 28 ++++ include/linux/swap.h | 33 +++-- mm/huge_memory.c | 3 - mm/internal.h | 48 +++++++ mm/madvise.c | 101 ++++++++------ mm/memory.c | 13 +- mm/swapfile.c | 298 ++++++++++++++++++++++------------------ mm/vmscan.c | 9 +- 8 files changed, 332 insertions(+), 201 deletions(-)--
2.25.1
Ryan Roberts
reviewerQuoted 67 lines · Ryan Roberts
Quoted 47 lines · Ying
Quoted 45 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |Quoted 6 lines
Quoted 2 lines
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
I don't understand why the performance of 2M THP improves. The swap entry allocation becomes a little slower. Can you provide some perf-profile to root cause it?
I didn't post the stdev, which is quite large (~10%), so that may explain some of it:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 5.5% | | base-64K | -14.6% | 3.8% | | base-2M | 87.4% | 10.6% | | v4-4K | 1.4% | 3.7% | | v4-64K | 44.2% | 11.8% | | v4-2M | 97.7% | 13.3% |
Quoted 1 line
Regardless, I'll do some perf profiling and post results shortly.
I did a lot more runs (24 for each config) and meaned them to try to remove the noise in the measurements. It's now only showing a 4% improvement for 2M. So I don't think the 2M improvement is real:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 3.2% | | base-64K | -9.1% | 10.1% | | base-2M | 88.9% | 6.8% | | v4-4K | 0.5% | 3.1% | | v4-64K | 44.7% | 8.3% | | v4-2M | 93.3% | 7.8% |
Looking at the perf data, the only thing that sticks out is that a big chunk of time is spent in during contpte_convert(), called as a result of try_to_unmap_one(). This is present in both the before and after configs.
This is an arm64 function to "unfold" contpte mappings. Essentially, the PMD is being split during shrink_folio_list() with TTU_SPLIT_HUGE_PMD, meaning the THPs are PTE-mapped in contpte blocks. Then we are unmapping each pte one-by-one which means the contpte block needs to be unfolded. I think try_to_unmap_one() could potentially be optimized to batch unmap a contiguously mapped folio and avoid this unfold. But that would be an independent and separate piece of work.
Quoted 93 lines
Quoted 92 lines
--
Best Regards,
Huang, YingQuoted 86 lines
This test also acts as a good stress test for swap and, more generally mm. A couple of existing bugs were found as a result [5] [6].
---
The series applies against mm-unstable (d7182786dd0a). Although I've additionally been running with a couple of extra fixes to avoid the issues at [6].Changes since v3 [3]
====================- Renamed
SWAP_NEXT_NULL->SWAP_NEXT_INVALID(per Huang, Ying)
- Simplified max offset calculation (per Huang, Ying)
- Reinstated structpercpu_clusterto contain per-cluster, per-ordernextoffset (per Huang, Ying)
- Removedswap_alloc_large()and merged its functionality intoscan_swap_map_slots()(per Huang, Ying)
- Avoid extra cost of folio ref and lock due to removal ofCLUSTER_FLAG_HUGEby freeing swap entries in batches (see patch 2) (per DavidH)
- vmscan splits folio if its partially mapped (per Barry Song, DavidH)
- Avoid splitting inMADV_PAGEOUTpath (per Barry Song)
- Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch since it's not actually a problem for THP as I first thought.Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
RyanRyan Roberts (6):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap:free_swap_and_cache_nr()as batchedfree_swap_and_cache()
mm: swap: Simplify structpercpu_cluster
mm: swap: Allow storage of all mTHP orders
mm: vmscan: Avoid split duringshrink_folio_list()
mm: madvise: Avoid split duringMADV_PAGEOUTandMADV_COLDinclude/linux/pgtable.h | 28 ++++ include/linux/swap.h | 33 +++-- mm/huge_memory.c | 3 - mm/internal.h | 48 +++++++ mm/madvise.c | 101 ++++++++------ mm/memory.c | 13 +- mm/swapfile.c | 298 ++++++++++++++++++++++------------------ mm/vmscan.c | 9 +- 8 files changed, 332 insertions(+), 201 deletions(-)--
2.25.1
Quoted 87 lines · Ryan Roberts
Quoted 67 lines · Ryan Roberts
Quoted 47 lines · Ying
Quoted 45 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |Quoted 6 lines
Quoted 2 lines
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
I don't understand why the performance of 2M THP improves. The swap entry allocation becomes a little slower. Can you provide some perf-profile to root cause it?
I didn't post the stdev, which is quite large (~10%), so that may explain some of it:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 5.5% | | base-64K | -14.6% | 3.8% | | base-2M | 87.4% | 10.6% | | v4-4K | 1.4% | 3.7% | | v4-64K | 44.2% | 11.8% | | v4-2M | 97.7% | 13.3% |Quoted 1 line
Regardless, I'll do some perf profiling and post results shortly.
I did a lot more runs (24 for each config) and meaned them to try to remove the noise in the measurements. It's now only showing a 4% improvement for 2M. So I don't think the 2M improvement is real:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 3.2% | | base-64K | -9.1% | 10.1% | | base-2M | 88.9% | 6.8% | | v4-4K | 0.5% | 3.1% | | v4-64K | 44.7% | 8.3% | | v4-2M | 93.3% | 7.8% |Looking at the perf data, the only thing that sticks out is that a big chunk of time is spent in during
contpte_convert(), called as a result oftry_to_unmap_one(). This is present in both the before and after configs.
Quoted 6 lines
This is an arm64 function to "unfold" contpte mappings. Essentially, the PMD is being split during
shrink_folio_list()withTTU_SPLIT_HUGE_PMD, meaning the THPs are PTE-mapped in contpte blocks. Then we are unmapping each pte one-by-one which means the contpte block needs to be unfolded. I thinktry_to_unmap_one()could potentially be optimized to batch unmap a contiguously mapped folio and avoid this unfold. But that would be an independent and separate piece of work.
Thanks for more data and detailed explanation.
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerQuoted 89 lines · Ying
Quoted 87 lines · Ryan Roberts
Quoted 67 lines · Ryan Roberts
Quoted 47 lines · Ying
Quoted 45 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |Quoted 6 lines
Quoted 2 lines
So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
I don't understand why the performance of 2M THP improves. The swap entry allocation becomes a little slower. Can you provide some perf-profile to root cause it?
I didn't post the stdev, which is quite large (~10%), so that may explain some of it:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 5.5% | | base-64K | -14.6% | 3.8% | | base-2M | 87.4% | 10.6% | | v4-4K | 1.4% | 3.7% | | v4-64K | 44.2% | 11.8% | | v4-2M | 97.7% | 13.3% |Quoted 1 line
Regardless, I'll do some perf profiling and post results shortly.
I did a lot more runs (24 for each config) and meaned them to try to remove the noise in the measurements. It's now only showing a 4% improvement for 2M. So I don't think the 2M improvement is real:
| kernel | mean_rel | std_rel | |:---------|-----------:|----------:| | base-4K | 0.0% | 3.2% | | base-64K | -9.1% | 10.1% | | base-2M | 88.9% | 6.8% | | v4-4K | 0.5% | 3.1% | | v4-64K | 44.7% | 8.3% | | v4-2M | 93.3% | 7.8% |Looking at the perf data, the only thing that sticks out is that a big chunk of time is spent in during
contpte_convert(), called as a result oftry_to_unmap_one(). This is present in both the before and after configs.
Quoted 8 lines
Quoted 6 lines
This is an arm64 function to "unfold" contpte mappings. Essentially, the PMD is being split during
shrink_folio_list()withTTU_SPLIT_HUGE_PMD, meaning the THPs are PTE-mapped in contpte blocks. Then we are unmapping each pte one-by-one which means the contpte block needs to be unfolded. I thinktry_to_unmap_one()could potentially be optimized to batch unmap a contiguously mapped folio and avoid this unfold. But that would be an independent and separate piece of work.Thanks for more data and detailed explanation.
And thanks for your review! I'll address all your comments (and any others that I get in the meantime) and repost after the merge window. It would be great if we can get this in for v6.10.
Quoted 4 lines
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerQuoted 38 lines · Ryan Roberts
Hi All,
This series adds support for swapping out multi-size THP (mTHP) without needing to first split the large folio via
split_huge_page_to_list_to_order(). It closely follows the approach already used to swap-out PMD-sized THP.There are a couple of reasons for swapping out mTHP without splitting:
- Performance: It is expensive to split a large folio and under extreme memory pressure some workloads regressed performance when using 64K mTHP vs 4K small folios because of this extra cost in the swap-out path. This series not only eliminates the regression but makes it faster to swap out 64K mTHP vs 4K small folios.
- Memory fragmentation avoidance: If we can avoid splitting a large folio memory is less likely to become fragmented, making it easier to re-allocate a large folio in future.
- Performance: Enables a separate series [4] to swap-in whole mTHPs, which means we won't lose the TLB-efficiency benefits of mTHP once the memory has been through a swap cycle.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is sufficient.
Performance Testing
===================I've run some swap performance tests on Ampere Altra VM (arm64) with 8 CPUs. The VM is set up with a 35G block ram device as the swap device and the test is run from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes, each allocating and writing 1G of memory. I've repeated everything 6 times and taken the mean performance improvement relative to 4K page baseline:
Quoted 2 lines
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | |
Oops, just noticed I failed to update these column headers. The baseline is actually mm-unstable (d7182786dd0a) which is based on v6.8-rc5 and already contains "anonfolio" - now called mTHP.
Quoted 95 lines
|:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 1.4% | | 64K THP | -14.6% | 44.2% | | 2M THP | 87.4% | 97.7% |So with this change, the 64K swap performance goes from a 15% regression to a 44% improvement. 4K and 2M swap improves slightly too.
This test also acts as a good stress test for swap and, more generally mm. A couple of existing bugs were found as a result [5] [6].
---
The series applies against mm-unstable (d7182786dd0a). Although I've additionally been running with a couple of extra fixes to avoid the issues at [6].Changes since v3 [3]
====================- Renamed
SWAP_NEXT_NULL->SWAP_NEXT_INVALID(per Huang, Ying)
- Simplified max offset calculation (per Huang, Ying)
- Reinstated structpercpu_clusterto contain per-cluster, per-ordernextoffset (per Huang, Ying)
- Removedswap_alloc_large()and merged its functionality intoscan_swap_map_slots()(per Huang, Ying)
- Avoid extra cost of folio ref and lock due to removal ofCLUSTER_FLAG_HUGEby freeing swap entries in batches (see patch 2) (per DavidH)
- vmscan splits folio if its partially mapped (per Barry Song, DavidH)
- Avoid splitting inMADV_PAGEOUTpath (per Barry Song)
- Dropped "mm: swap: Simplify ssd behavior when scanner steals entry" patch since it's not actually a problem for THP as I first thought.Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/Thanks,
RyanRyan Roberts (6):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap:free_swap_and_cache_nr()as batchedfree_swap_and_cache()
mm: swap: Simplify structpercpu_cluster
mm: swap: Allow storage of all mTHP orders
mm: vmscan: Avoid split duringshrink_folio_list()
mm: madvise: Avoid split duringMADV_PAGEOUTandMADV_COLDinclude/linux/pgtable.h | 28 ++++ include/linux/swap.h | 33 +++-- mm/huge_memory.c | 3 - mm/internal.h | 48 +++++++ mm/madvise.c | 101 ++++++++------ mm/memory.c | 13 +- mm/swapfile.c | 298 ++++++++++++++++++++++------------------ mm/vmscan.c | 9 +- 8 files changed, 332 insertions(+), 201 deletions(-)--
2.25.1