Swap-out small-sized THP without splitting
Subsystems (11)
Ryan Roberts
reviewerv3 cover Swap-out small-sized THP without splitting
Hi All,
This is v3 of a series to add support for swapping out small-sized THP without needing to first split the large folio via __split_huge_page(). It closely follows the approach already used by PMD-sized THP.
"Small-sized THP" is an upcoming feature that enables performance improvements by allocating large folios for anonymous memory, where the large folio size is smaller than the traditional PMD-size. See [3].
In some circumstances I've observed a performance regression (see patch 2 for details), and this series is an attempt to fix the regression in advance of merging small-sized THP support.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is probably sufficient.
The series applies against mm-unstable (1a3c85fa684a)
Changes since v2 [2]
====================
- Reuse scan_swap_map_try_ssd_cluster() between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug where nr_swap_pages would say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).
Thanks to Huang, Ying for the review feedback and suggestions!
Changes since v1 [1]
====================
- patch 1:
- Use cluster_set_count() instead of cluster_set_count_flag() in swap_alloc_cluster() since we no longer have any flag to set. I was unable to kill cluster_set_count_flag() as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array into percpu_cluster to make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated because PMD_ORDER is not compile-time constant for powerpc (fixes build error).
Thanks,
Ryan
P.S. I know we agreed this is not a prerequisite for merging small-sized THP, but given Huang Ying had provided some review feedback, I wanted to progress it. All the actual prerequisites are either complete or being worked on by others.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
Ryan Roberts (4):
mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
mm: swap: Remove struct percpu_cluster
mm: swap: Simplify ssd behavior when scanner steals entry
mm: swap: Swap-out small-sized THP without splitting
include/linux/swap.h | 31 +++--- mm/huge_memory.c | 3 - mm/swapfile.c | 232 ++++++++++++++++++++++++------------------- mm/vmscan.c | 10 +- 4 files changed, 149 insertions(+), 127 deletions(-)
--
2.25.1
Ryan Roberts
reviewerPATCH v3 1/4 mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.
The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass in nr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.
The one snag is that one of the swap_page_trans_huge_swapped() call sites does not have the folio. But it was only being called there to avoid bothering to call __try_to_reclaim_swap() in some cases. __try_to_reclaim_swap() gets the folio and (via some other functions) calls swap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.
Removing CLUSTER_FLAG_HUGE also means we can remove split_swap_cluster() which used to be called during folio splitting, since split_swap_cluster()'s only job was to remove the flag.
--- include/linux/swap.h | 10 ---------- mm/huge_memory.c | 3 --- mm/swapfile.c | 47 ++++++++------------------------------------ 3 files changed, 8 insertions(+), 52 deletions(-)
include/linux/swap.h +0 -10
@@ -259,7 +259,6 @@ struct swap_cluster_info { };#define CLUSTER_FLAG_FREE 1/* This cluster is free */ #define CLUSTER_FLAG_NEXT_NULL 2/* This cluster has no next cluster */ -#define CLUSTER_FLAG_HUGE 4/* This cluster is backing a transparent huge page */ /* * We assign a cluster to each CPU, so each CPU can allocate swap entry from
@@ -595,15 +594,6 @@ static inline int add_swap_extent(struct swap_info_struct *sis, }#endif /* CONFIG_SWAP */ -#ifdef CONFIG_THP_SWAP -extern int split_swap_cluster (swp_entry_t entry); -#else -static inline int split_swap_cluster (swp_entry_t entry) -{ -return 0 ; -} -#endif -#ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness (struct mem_cgroup *memcg) {
mm/huge_memory.c +0 -3
@@ -2598,9 +2598,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,shmem_uncharge (head->mapping->host, nr_dropped);remap_page (folio, nr); -if (folio_test_swapcache (folio)) -split_swap_cluster (folio->swap); -for (i =0 ; i < nr; i++) {struct page *subpage = head + i;if (subpage == page)
mm/swapfile.c +8 -39
107 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.
Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.
That is the __try_to_reclaim_swap() -> folio_free_swap() -> folio_swapped() -> swap_page_trans_huge_swapped() call chain I assume.
The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.
Quoted 7 lines
That is the
__try_to_reclaim_swap()->folio_free_swap()->folio_swapped()->swap_page_trans_huge_swapped()call chain I assume.The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Thinking about it, your change will not only affect THP, but any call to free_swap_and_cache().
Likely that's not what we want. :/
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 28 lines · David Hildenbrand
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.Quoted 7 lines
That is the
__try_to_reclaim_swap()->folio_free_swap()->folio_swapped()->swap_page_trans_huge_swapped()call chain I assume.The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Quoted 5 lines
Thinking about it, your change will not only affect THP, but any call to
free_swap_and_cache().Likely that's not what we want. :/
Is folio_trylock() really that expensive given the code path is already locking multiple spinlocks, and I don't think we would expect the folio lock to be very contended?
I guess filemap_get_folio() could be a bit more expensive, but again, is this really a deal-breaker?
I'm just trying to refamiliarize myself with this series, but I think I ended up allocating a cluster per cpu per order. So one potential solution would be to turn the flag into a size and store it in the cluster info. (In fact I think I was doing that in an early version of this series - will have to look at why I got rid of that). Then we could avoid needing to figure out nr_pages from the folio.
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Hi David,
Quoted 43 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.Quoted 7 lines
That is the
__try_to_reclaim_swap()->folio_free_swap()->folio_swapped()->swap_page_trans_huge_swapped()call chain I assume.The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Quoted 5 lines
Thinking about it, your change will not only affect THP, but any call to
free_swap_and_cache().Likely that's not what we want. :/
Is
folio_trylock()really that expensive given the code path is already locking multiple spinlocks, and I don't think we would expect the folio lock to be very contended?I guess
filemap_get_folio()could be a bit more expensive, but again, is this really a deal-breaker?
Quoted 5 lines
I'm just trying to refamiliarize myself with this series, but I think I ended up allocating a cluster per cpu per order. So one potential solution would be to turn the flag into a size and store it in the cluster info. (In fact I think I was doing that in an early version of this series - will have to look at why I got rid of that). Then we could avoid needing to figure out
nr_pagesfrom the folio.
I ran some microbenchmarks to see if these extra operations cause a performance issue - it all looks OK to me.
I modified your "pte-mapped-folio-benchmarks" to add a "munmap-swapped-forked" mode, which prepares the 1G memory mapping by first paging it out with MADV_PAGEOUT, then it forks a child (and keeps that child alive) so that the swap slots have 2 references, then it measures the duration of munmap() in the parent on the entire range. The idea is that free_swap_and_cache() is called for each PTE during munmap(). Prior to my change, swap_page_trans_huge_swapped() will return true, due to the child's references, and __try_to_reclaim_swap() is not called. After my change, we no longer have this short cut.
In both cases the results are within 1% (confirmed across multiple runs of 20 seconds each):
mm-stable: Average: 0.004997
+ change: Average: 0.005037
(these numbers are for Ampere Altra. I also tested on M2 VM - no regression their either).
Do you still have a concern about this change?
An alternative is to store the folio size in the cluster, but that won't be accurate if the folio is later split or if an entry within the cluster is later stolen for an order-0 entry. I think it would still work though; it just means that you might get a false positive in those circumstances, which means taking the "slow" path. But this is a rare event.
Regardless, I prefer not to do this, since it adds complexity and doesn't benefit performance.
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 46 lines · Ryan Roberts
Hi David,
Quoted 43 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.Quoted 7 lines
That is the
__try_to_reclaim_swap()->folio_free_swap()->folio_swapped()->swap_page_trans_huge_swapped()call chain I assume.The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Quoted 5 lines
Thinking about it, your change will not only affect THP, but any call to
free_swap_and_cache().Likely that's not what we want. :/
Is
folio_trylock()really that expensive given the code path is already locking multiple spinlocks, and I don't think we would expect the folio lock to be very contended?I guess
filemap_get_folio()could be a bit more expensive, but again, is this really a deal-breaker?
Quoted 8 lines
Quoted 5 lines
I'm just trying to refamiliarize myself with this series, but I think I ended up allocating a cluster per cpu per order. So one potential solution would be to turn the flag into a size and store it in the cluster info. (In fact I think I was doing that in an early version of this series - will have to look at why I got rid of that). Then we could avoid needing to figure out
nr_pagesfrom the folio.I ran some microbenchmarks to see if these extra operations cause a performance issue - it all looks OK to me.
Sorry, I'm drowning in reviews right now. I was hoping to get some of my own stuff figured out today ... maybe tomorrow.
Quoted 13 lines
I modified your "pte-mapped-folio-benchmarks" to add a "munmap-swapped-forked" mode, which prepares the 1G memory mapping by first paging it out with
MADV_PAGEOUT, then it forks a child (and keeps that child alive) so that the swap slots have 2 references, then it measures the duration ofmunmap()in the parent on the entire range. The idea is thatfree_swap_and_cache()is called for each PTE duringmunmap(). Prior to my change,swap_page_trans_huge_swapped()will return true, due to the child's references, and__try_to_reclaim_swap()is not called. After my change, we no longer have this short cut.In both cases the results are within 1% (confirmed across multiple runs of 20 seconds each):
Quoted 7 lines
mm-stable: Average: 0.004997
+ change: Average: 0.005037(these numbers are for Ampere Altra. I also tested on M2 VM - no regression their either).
Do you still have a concern about this change?
The main concern I had was not about overhead due to atomic operations in the non-concurrent case that you are measuring.
We might now unnecessarily be incrementing the folio refcount and taking the folio lock. That will affects large folios in the swapcache now IIUC. Small folios should be unaffected.
The side effects of that can be:
* Code checking for additional folio reference could now detect some and back out. (the "mapcount + swapcache*folio_nr_pages != folio_refcount" stuff)
* Code that might really benefit from trylocking the folio might fail to do so.
For example, splitting a large folio might now fail more often simply because some process zaps a swap entry and the additional reference+page lock were optimized out previously.
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
Arguably, zapping a present PTE also leaves the refcount elevated for a while until the mapcount was freed. But here, it could be avoided.
Digging a bit, it was introduced in:
commit e07098294adfd03d582af7626752255e3d170393
Author: Huang Ying <[email protected]> Date: Wed Sep 6 16:22:16 2017 -0700
mm, THP, swap: support to reclaim swap space for THP swapped out
The normal swap slot reclaiming can be done when the swap count reaches SWAP_HAS_CACHE. But for the swap slot which is backing a THP, all swap slots backing one THP must be reclaimed together, because the swap slot may be used again when the THP is swapped out again later. So the swap slots backing one THP can be reclaimed together when the swap count for all swap slots for the THP reached SWAP_HAS_CACHE. In the patch, the functions to check whether the swap count for all swap slots backing one THP reached SWAP_HAS_CACHE are implemented and used when checking whether a swap slot can be reclaimed.
To make it easier to determine whether a swap slot is backing a THP, a new swap cluster flag named CLUSTER_FLAG_HUGE is added to mark a swap cluster which is backing a THP (Transparent Huge Page). Because THP swap in as a whole isn't supported now. After deleting the THP from the swap cache (for example, swapping out finished), the CLUSTER_FLAG_HUGE flag will be cleared. So that, the normal pages inside THP can be swapped in individually.
With your change, if we have a swapped out THP with 512 entries and exit(), we would now 512 times in a row grab a folio reference and trylock the folio. In the past, we would have done that at most once.
That doesn't feel quite right TBH ... so I'm wondering if there are any low-hanging fruits to avoid that.
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Hi David, Huang,
Quoted 54 lines · David Hildenbrand
Quoted 53 lines · Ryan Roberts
Hi David,
Quoted 49 lines · Ryan Roberts
Quoted 28 lines · David Hildenbrand
Quoted 19 lines · David Hildenbrand
Quoted 11 lines · Ryan Roberts
As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the
CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in
swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass innr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.Quoted 6 lines
The one snag is that one of the
swap_page_trans_huge_swapped()call sites does not have the folio. But it was only being called there to avoid bothering to call__try_to_reclaim_swap()in some cases.__try_to_reclaim_swap()gets the folio and (via some other functions) callsswap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.Quoted 7 lines
That is the
__try_to_reclaim_swap()->folio_free_swap()->folio_swapped()->swap_page_trans_huge_swapped()call chain I assume.The "difference" is that you will now (1) get another temporary reference on the folio and (2) (try)lock the folio every time you discard a single PTE of a (possibly) large THP.
Quoted 5 lines
Thinking about it, your change will not only affect THP, but any call to
free_swap_and_cache().Likely that's not what we want. :/
Is
folio_trylock()really that expensive given the code path is already locking multiple spinlocks, and I don't think we would expect the folio lock to be very contended?I guess
filemap_get_folio()could be a bit more expensive, but again, is this really a deal-breaker?I'm just trying to refamiliarize myself with this series, but I think I ended up allocating a cluster per cpu per order. So one potential solution would be to turn the flag into a size and store it in the cluster info. (In fact I think I was doing that in an early version of this series - will have to look at why I got rid of that). Then we could avoid needing to figure out
nr_pagesfrom the folio.
Quoted 5 lines
Quoted 2 lines
I ran some microbenchmarks to see if these extra operations cause a performance issue - it all looks OK to me.
Sorry, I'm drowning in reviews right now. I was hoping to get some of my own stuff figured out today ... maybe tomorrow.
No need to apologise - as always I appreciate whatever time you can spare.
Quoted 20 lines
Quoted 20 lines
I modified your "pte-mapped-folio-benchmarks" to add a "munmap-swapped-forked" mode, which prepares the 1G memory mapping by first paging it out with
MADV_PAGEOUT, then it forks a child (and keeps that child alive) so that the swap slots have 2 references, then it measures the duration ofmunmap()in the parent on the entire range. The idea is thatfree_swap_and_cache()is called for each PTE duringmunmap(). Prior to my change,swap_page_trans_huge_swapped()will return true, due to the child's references, and__try_to_reclaim_swap()is not called. After my change, we no longer have this short cut.In both cases the results are within 1% (confirmed across multiple runs of 20 seconds each):
mm-stable: Average: 0.004997
+ change: Average: 0.005037(these numbers are for Ampere Altra. I also tested on M2 VM - no regression their either).
Quoted 8 lines
Quoted 1 line
Do you still have a concern about this change?
The main concern I had was not about overhead due to atomic operations in the non-concurrent case that you are measuring.
We might now unnecessarily be incrementing the folio refcount and taking the folio lock. That will affects large folios in the swapcache now IIUC. Small folios should be unaffected.
Yes I think you are right, because count == SWAP_HAS_CACHE is already checking the small page is not swapped. So my perf tests weren't actually doing what I thought they were.
Quoted 11 lines
The side effects of that can be:
* Code checking for additional folio reference could now detect some and back out. (the "mapcount + swapcache*folio_nr_pages !=folio_refcount" stuff)
* Code that might really benefit from trylocking the folio might fail to do so.For example, splitting a large folio might now fail more often simply because some process zaps a swap entry and the additional reference+page lock were optimized out previously.
Understood. Of course this is the type of fuzzy reasoning that is very dificult to test objectively :). But it makes sense and I suppose I'll have to come up with an alternative approach (see below).
Quoted 3 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
[1]
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 19 lines
Arguably, zapping a present PTE also leaves the refcount elevated for a while until the mapcount was freed. But here, it could be avoided.
Digging a bit, it was introduced in:
commit
e07098294adfd03d582af7626752255e3d170393
Author: Huang Ying <[email protected]>
Date: Wed Sep 6 16:22:16 2017 -0700mm, THP, swap: support to reclaim swap space for THP swapped out The normal swap slot reclaiming can be done when the swap count reaches SWAP_HAS_CACHE. But for the swap slot which is backing a THP, all swap slots backing one THP must be reclaimed together, because the swap slot may be used again when the THP is swapped out again later. So the swap slots backing one THP can be reclaimed together when the swap count for all swap slots for the THP reached SWAP_HAS_CACHE. In the patch, the functions to check whether the swap count for all swap slots backing one THP reached
SWAP_HAS_CACHEare implemented and used when checking
Quoted 8 lines
whether a swap slot can be reclaimed.
To make it easier to determine whether a swap slot is backing a THP, a new swap cluster flag namedCLUSTER_FLAG_HUGEis added to mark a swap cluster which is backing a THP (Transparent Huge Page). Because THP swap in as a whole isn't supported now. After deleting the THP from the swap cache (for example, swapping out finished), theCLUSTER_FLAG_HUGEflag will be cleared. So that, the normal pages inside THP can be swapped in individually.
Thanks. I did this same archaeology, but found nothing pointing to the rationale for this optimization, so decided that if its undocumented, then it probably wasn't critical.
Quoted 9 lines
With your change, if we have a swapped out THP with 512 entries and
exit(), we would now 512 times in a row grab a folio reference and trylock the folio. In the past, we would have done that at most once.That doesn't feel quite right TBH ... so I'm wondering if there are any low-hanging fruits to avoid that.
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes; PAGE_SIZE and PMD_SIZE. For PMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it as PMD_SIZE.
Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for swap_page_trans_huge_swapped() meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared in put_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 20 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.
Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 34 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
Quoted 8 lines
Quoted 3 lines
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where free_swap_and_cache() is called, and only 2 of those are really amenable to batching (zap_pte_range() and madvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.
Quoted 7 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry = swp_entry(swp_type(entry), swp_offset(entry) + 1);
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 41 lines · Ryan Roberts
Quoted 40 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 20 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
Quoted 8 lines
Quoted 2 lines
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where
free_swap_and_cache()is called, and only 2 of those are really amenable to batching (zap_pte_range()andmadvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.
We can ignore the s390x one. That s390x code should only apply to KVM guest memory where ordinary THP are not even supported. (and nobody uses mTHP there yet).
Long story short: the VM can hint that some memory pages are now unused and the hypervisor can reclaim them. That's what that callback does (zap guest-provided guest memory). No need to worry about any batching for now.
Then, there is the shmem one in shmem_free_swap(). I really don't know how shmem handles THP+swapout.
But looking at shmem_writepage(), we split any large folios before moving them to the swapcache, so likely we don't care at all, because THP don't apply.
Quoted 6 lines
Quoted 6 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Quoted 6 lines
Quoted 2 lines
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry =
swp_entry(swp_type(entry), swp_offset(entry) + 1);
I was talking about the advancing swapslot processing after decrementing the swapcounts.
Assume you decremented 512 swapcounts and some of them went to 0. AFAIU, you'd have to start with the first swapslot that has now a swapcount=0 one and try to reclaim swap.
Assume you get a small folio, then you'll have to proceed with the next swap slot and try to reclaim swap.
Assume you get a large folio, then you can skip more swapslots (depending on offset into the folio etc).
If you get what I mean. :)
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 55 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster
to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these
cases to be rare.1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We
wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster
to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
Quoted 8 lines
Quoted 2 lines
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where
free_swap_and_cache()is called, and only 2 of those are really amenable to batching (zap_pte_range()andmadvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.
Quoted 6 lines
We can ignore the s390x one. That s390x code should only apply to KVM guest memory where ordinary THP are not even supported. (and nobody uses mTHP there yet).
Long story short: the VM can hint that some memory pages are now unused and the hypervisor can reclaim them. That's what that callback does (zap guest-provided guest memory). No need to worry about any batching for now.
OK good.
Quoted 6 lines
Then, there is the shmem one in
shmem_free_swap(). I really don't know how shmem handles THP+swapout.But looking at
shmem_writepage(), we split any large folios before moving them to the swapcache, so likely we don't care at all, because THP don't apply.
Excellent.
Quoted 21 lines
Quoted 7 lines
Quoted 7 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Quoted 6 lines
Quoted 2 lines
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry =
swp_entry(swp_type(entry), swp_offset(entry) + 1);I was talking about the advancing swapslot processing after decrementing the swapcounts.
Assume you decremented 512 swapcounts and some of them went to 0. AFAIU, you'd have to start with the first swapslot that has now a swapcount=0 one and try to reclaim swap.
Quoted 7 lines
Assume you get a small folio, then you'll have to proceed with the next swap slot and try to reclaim swap.
Assume you get a large folio, then you can skip more swapslots (depending on offset into the folio etc).
If you get what I mean. :)
Ahh gottya. I'll have a play.
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to David HildenbrandQuoted 88 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster
to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these
cases to be rare.1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We
wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster
to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
Quoted 8 lines
Quoted 2 lines
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where
free_swap_and_cache()is called, and only 2 of those are really amenable to batching (zap_pte_range()andmadvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.We can ignore the s390x one. That s390x code should only apply to KVM guest memory where ordinary THP are not even supported. (and nobody uses mTHP there yet).
Long story short: the VM can hint that some memory pages are now unused and the hypervisor can reclaim them. That's what that callback does (zap guest-provided guest memory). No need to worry about any batching for now.
Then, there is the shmem one in
shmem_free_swap(). I really don't know how shmem handles THP+swapout.But looking at
shmem_writepage(), we split any large folios before moving them to the swapcache, so likely we don't care at all, because THP don't apply.Quoted 6 lines
Quoted 6 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Quoted 6 lines
Quoted 2 lines
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry =
swp_entry(swp_type(entry), swp_offset(entry) + 1);I was talking about the advancing swapslot processing after decrementing the swapcounts.
Assume you decremented 512 swapcounts and some of them went to 0. AFAIU, you'd have to start with the first swapslot that has now a swapcount=0 one and try to reclaim swap.
Quoted 8 lines
Assume you get a small folio, then you'll have to proceed with the next swap slot and try to reclaim swap.
Assume you get a large folio, then you can skip more swapslots (depending on offset into the folio etc).
If you get what I mean. :)
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to free_swap_and_cache() to still have a cached folio? Does anyone have any idea?
This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling __try_to_reclaim_swap() far too much:
int free_swap_and_cache(swp_entry_t entry)
{
struct swap_info_struct *p;
unsigned char count;
if (non_swap_entry(entry))
return 1;
p = _swap_info_get(entry);
if (p) {
count = __swap_entry_free(p, entry);
if (count == SWAP_HAS_CACHE)
__try_to_reclaim_swap(p, swp_offset(entry),
TTRS_UNMAPPED | TTRS_FULL);
}
return p != NULL;
}The trouble is, whenever its called, count is always 0, so __try_to_reclaim_swap() never gets called.
My test case is allocating 1G anon memory, then doing madvise(MADV_PAGEOUT) over it. Then doing either a munmap() or madvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after __swap_entry_free() so __try_to_reclaim_swap() is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.
I'm guessing the swapcache was already reclaimed as part of MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...
Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 27 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }
Quoted 8 lines
The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.
I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 11 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 31 lines · Matthew Wilcox
Quoted 30 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.
Quoted 8 lines
Quoted 5 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
I guess I can try forking after swap out, then fault it back in in the child and exit. Then do the munmap in the parent. I guess that could force it? Thanks for the tip - I'll have a play.
Quoted 15 lines
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 11 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 38 lines · Ryan Roberts
Quoted 37 lines · Matthew Wilcox
Quoted 35 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.
Quoted 6 lines
Quoted 2 lines
I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
I thought with most (disk) backends you will add it to the swapcache and leave it there until there is actual memory pressure. Only then, under memory pressure, you'd actually reclaim the folio.
You can fault it back in from the swapcache without having to go to disk.
That's how you can today end up with a THP in the swapcache: during swapin from disk (after the folio was reclaimed) you'd currently only get order-0 folios.
At least that was my assumption with my MADV_PAGEOUT testing so far :)
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Ryan Roberts
Quoted 42 lines · Matthew Wilcox
Quoted 31 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's
concern that we would end up calling__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry); if (count ==SWAP_HAS_CACHE)__try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); }
return p != NULL;
}The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 7 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over
it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this
function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Quoted 7 lines
Quoted 3 lines
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
I thought with most (disk) backends you will add it to the swapcache and leave it there until there is actual memory pressure. Only then, under memory pressure, you'd actually reclaim the folio.
OK, my problem is that I'm using a VM, whose disk shows up as rotating media, so the swap subsystem refuses to swap out THPs to that and they get split. To solve that, (and to speed up testing) I moved to the block ram disk, which convinces swap to swap-out THPs. But that causes the folios to be removed from the swap cache (I assumed because its syncrhonous, but maybe there is a flag somewhere to affect that behavior?) And I can't convince QEMU to emulate an SSD to the guest under macos. Perhaps the easiest thing is to hack it to ignore the rotating media flag.
Quoted 8 lines
You can fault it back in from the swapcache without having to go to disk.
That's how you can today end up with a THP in the swapcache: during swapin from disk (after the folio was reclaimed) you'd currently only get order-0 folios.
At least that was my assumption with my
MADV_PAGEOUTtesting so far :)
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 54 lines · Ryan Roberts
Quoted 45 lines · David Hildenbrand
Quoted 44 lines · Ryan Roberts
Quoted 42 lines · Matthew Wilcox
Quoted 31 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's
concern that we would end up calling__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry); if (count ==SWAP_HAS_CACHE)__try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); }
return p != NULL;
}The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 7 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over
it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this
function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Quoted 7 lines
Quoted 3 lines
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
I thought with most (disk) backends you will add it to the swapcache and leave it there until there is actual memory pressure. Only then, under memory pressure, you'd actually reclaim the folio.
Quoted 8 lines
OK, my problem is that I'm using a VM, whose disk shows up as rotating media, so the swap subsystem refuses to swap out THPs to that and they get split. To solve that, (and to speed up testing) I moved to the block ram disk, which convinces swap to swap-out THPs. But that causes the folios to be removed from the swap cache (I assumed because its syncrhonous, but maybe there is a flag somewhere to affect that behavior?) And I can't convince QEMU to emulate an SSD to the guest under macos. Perhaps the easiest thing is to hack it to ignore the rotating media flag.
I'm trying to remember how I triggered it in the past, I thought cow.c selftest was able to do that.
What certainly works is taking a reference on the page using vmsplice() and then doing the MADV_PAGEOUT. But there has to be a better way :)
I'll dig on Monday!
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to Ryan RobertsQuoted 41 lines · Ryan Roberts
Quoted 31 lines · Matthew Wilcox
Quoted 30 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 8 lines
Quoted 5 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Quoted 7 lines
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
I guess I can try forking after swap out, then fault it back in in the child and exit. Then do the munmap in the parent. I guess that could force it? Thanks for the tip - I'll have a play.
That has sort of solved it, the only problem now is that all the folios in the swap cache are small (because I don't have Barry's large swap-in series). So really I need to figure out how to avoid removing the folio from the cache in the first place...
Quoted 17 lines
Quoted 16 lines
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 11 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 47 lines · Ryan Roberts
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · Matthew Wilcox
Quoted 30 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 8 lines
Quoted 5 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
Quoted 8 lines
Quoted 3 lines
I guess I can try forking after swap out, then fault it back in in the child and exit. Then do the munmap in the parent. I guess that could force it? Thanks for the tip - I'll have a play.
That has sort of solved it, the only problem now is that all the folios in the swap cache are small (because I don't have Barry's large swap-in series). So really I need to figure out how to avoid removing the folio from the cache in the first place...
I am quite sure we have a chance to hit a large swapcache even using zRAM - a sync swapfile and even during swap-out.
I have a test case as below,
1. two threads to run MADV_PAGEOUT
2. two threads to read data being swapped-out
in do_swap_page, from time to time, I can get a large swapcache.
We have a short time window after add_to_swap() and before __removing_mapping() of
vmscan, a large folio is still in swapcache.
So Ryan, I guess you can trigger this by adding one more thread of MADV_DONTNEED to do zap_pte_range?
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 5 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Thanks
Barry
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 65 lines · Barry Song
Quoted 47 lines · Ryan Roberts
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · Matthew Wilcox
Quoted 30 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 8 lines
Quoted 5 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
Quoted 8 lines
Quoted 3 lines
I guess I can try forking after swap out, then fault it back in in the child and exit. Then do the munmap in the parent. I guess that could force it? Thanks for the tip - I'll have a play.
That has sort of solved it, the only problem now is that all the folios in the swap cache are small (because I don't have Barry's large swap-in series). So really I need to figure out how to avoid removing the folio from the cache in the first place...
I am quite sure we have a chance to hit a large swapcache even using zRAM - a sync swapfile and even during swap-out.
I have a test case as below,
1. two threads to runMADV_PAGEOUT
2. two threads to read data being swapped-out
Quoted 8 lines
in
do_swap_page, from time to time, I can get a large swapcache.We have a short time window after
add_to_swap()and before__removing_mapping()of
vmscan, a large folio is still in swapcache.So Ryan, I guess you can trigger this by adding one more thread of
MADV_DONTNEEDto dozap_pte_range?
Ryan, I have modified my test case to have 4 threads:
1. MADV_PAGEOUT
2. MADV_DONTNEED
3. write data
4. read data
and git push the code here so that you can get it,
https://github.com/BarrySong666/swaptest/blob/main/swptest.c
I can reproduce the issue in zap_pte_range() in just a couple of minutes.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 5 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 75 lines · Barry Song
Quoted 65 lines · Barry Song
Quoted 47 lines · Ryan Roberts
Quoted 45 lines · Ryan Roberts
Quoted 31 lines · Matthew Wilcox
Quoted 30 lines · Ryan Roberts
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.Quoted 8 lines
Quoted 5 lines
My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.I think you have to page it back in again, then it will have an entry in the swap cache. Maybe. I know little about anon memory ;-)
Ahh, I was under the impression that the original folio is put into the swap cache at swap out, then (I guess) its removed once the IO is complete? I'm sure I'm miles out... what exactly is the lifecycle of a folio going through swap out?
Quoted 8 lines
Quoted 3 lines
I guess I can try forking after swap out, then fault it back in in the child and exit. Then do the munmap in the parent. I guess that could force it? Thanks for the tip - I'll have a play.
That has sort of solved it, the only problem now is that all the folios in the swap cache are small (because I don't have Barry's large swap-in series). So really I need to figure out how to avoid removing the folio from the cache in the first place...
I am quite sure we have a chance to hit a large swapcache even using zRAM - a sync swapfile and even during swap-out.
I have a test case as below,
1. two threads to runMADV_PAGEOUT
2. two threads to read data being swapped-outQuoted 8 lines
in
do_swap_page, from time to time, I can get a large swapcache.We have a short time window after
add_to_swap()and before__removing_mapping()of
vmscan, a large folio is still in swapcache.So Ryan, I guess you can trigger this by adding one more thread of
MADV_DONTNEEDto dozap_pte_range?
Quoted 8 lines
Ryan, I have modified my test case to have 4 threads:
1.MADV_PAGEOUT
2.MADV_DONTNEED
3. write data
4. read dataand git push the code here so that you can get it,
https://github.com/BarrySong666/swaptest/blob/main/swptest.c
Thanks for this, Barry!
Quoted 26 lines
I can reproduce the issue in
zap_pte_range()in just a couple of minutes.Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
If that doesn't work, perhaps use tmpfs, and use some memory pressure to force that to swap?
Quoted 5 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
RyanThanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to Ryan RobertsQuoted 134 lines · Ryan Roberts
Quoted 88 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster
to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these
cases to be rare.1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We
wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster
to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
Quoted 8 lines
Quoted 2 lines
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where
free_swap_and_cache()is called, and only 2 of those are really amenable to batching (zap_pte_range()andmadvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.We can ignore the s390x one. That s390x code should only apply to KVM guest memory where ordinary THP are not even supported. (and nobody uses mTHP there yet).
Long story short: the VM can hint that some memory pages are now unused and the hypervisor can reclaim them. That's what that callback does (zap guest-provided guest memory). No need to worry about any batching for now.
Then, there is the shmem one in
shmem_free_swap(). I really don't know how shmem handles THP+swapout.But looking at
shmem_writepage(), we split any large folios before moving them to the swapcache, so likely we don't care at all, because THP don't apply.Quoted 6 lines
Quoted 6 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Quoted 6 lines
Quoted 2 lines
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry =
swp_entry(swp_type(entry), swp_offset(entry) + 1);I was talking about the advancing swapslot processing after decrementing the swapcounts.
Assume you decremented 512 swapcounts and some of them went to 0. AFAIU, you'd have to start with the first swapslot that has now a swapcount=0 one and try to reclaim swap.
Quoted 8 lines
Assume you get a small folio, then you'll have to proceed with the next swap slot and try to reclaim swap.
Assume you get a large folio, then you can skip more swapslots (depending on offset into the folio etc).
If you get what I mean. :)
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.
Quoted 3 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck?
Ahh I just switched to SSD as swap device and now its getting called. I guess that's the reason. Sorry for the noise.
Quoted 8 lines
Just a guess...
Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to Ryan RobertsQuoted 134 lines · Ryan Roberts
Quoted 88 lines · David Hildenbrand
Quoted 45 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster
to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these
cases to be rare.1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We
wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster
to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
Quoted 8 lines
Quoted 2 lines
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Yes in principle, but there are 4 places where
free_swap_and_cache()is called, and only 2 of those are really amenable to batching (zap_pte_range()andmadvise_free_pte_range()). So the other two users will still take the "slow" path. Maybe those 2 callsites are the only ones that really matter? I can certainly have a stab at this approach.We can ignore the s390x one. That s390x code should only apply to KVM guest memory where ordinary THP are not even supported. (and nobody uses mTHP there yet).
Long story short: the VM can hint that some memory pages are now unused and the hypervisor can reclaim them. That's what that callback does (zap guest-provided guest memory). No need to worry about any batching for now.
Then, there is the shmem one in
shmem_free_swap(). I really don't know how shmem handles THP+swapout.But looking at
shmem_writepage(), we split any large folios before moving them to the swapcache, so likely we don't care at all, because THP don't apply.Quoted 6 lines
Quoted 6 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Quoted 6 lines
Quoted 2 lines
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
I'm not sure I've understood this. Isn't advancing just a matter of:
entry =
swp_entry(swp_type(entry), swp_offset(entry) + 1);I was talking about the advancing swapslot processing after decrementing the swapcounts.
Assume you decremented 512 swapcounts and some of them went to 0. AFAIU, you'd have to start with the first swapslot that has now a swapcount=0 one and try to reclaim swap.
Quoted 8 lines
Assume you get a small folio, then you'll have to proceed with the next swap slot and try to reclaim swap.
Assume you get a large folio, then you can skip more swapslots (depending on offset into the folio etc).
If you get what I mean. :)
I've implemented the batching as David suggested, and I'm pretty confident it's correct. The only problem is that during testing I can't provoke the code to take the path. I've been pouring through the code but struggling to figure out under what situation you would expect the swap entry passed to
free_swap_and_cache()to still have a cached folio? Does anyone have any idea?This is the original (unbatched) function, after my change, which caused David's concern that we would end up calling
__try_to_reclaim_swap()far too much:int
free_swap_and_cache(swp_entry_t entry){ struct swap_info_struct *p; unsigned char count; if (non_swap_entry(entry)) return 1; p = _swap_info_get(entry); if (p) { count = __swap_entry_free(p, entry); if (count == SWAP_HAS_CACHE) __try_to_reclaim_swap(p, swp_offset(entry), TTRS_UNMAPPED | TTRS_FULL); } return p != NULL; }The trouble is, whenever its called, count is always 0, so
__try_to_reclaim_swap()never gets called.My test case is allocating 1G anon memory, then doing
madvise(MADV_PAGEOUT)over it. Then doing either amunmap()ormadvise(MADV_FREE), both of which cause this function to be called for every PTE, but count is always 0 after__swap_entry_free()so__try_to_reclaim_swap()is never called. I've tried for order-0 as well as PTE- and PMD-mapped 2M THP.
Quoted 7 lines
I'm guessing the swapcache was already reclaimed as part of
MADV_PAGEOUT? I'm using a block ram device as my backing store - I think this does synchronous IO so perhaps if I have a real block device with async IO I might have more luck? Just a guess...Or perhaps this code path is a corner case? In which case, perhaps its not worth adding the batching optimization after all?
I had to disable zswap in the past and was able to trigger this reliably with an ordinary swap backend (e.g., proper disk).
Whenever you involve swap-to-ram, you might just get it reclaimed immediately.
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to David HildenbrandQuoted 43 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 20 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Quoted 7 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
Hi,
I'm struggling to convince myself that free_swap_and_cache() can't race with with swapoff(). Can anyone explain that this is safe?
I *think* they are both serialized by the PTL, since all callers of free_swap_and_cache() (except shmem) have the PTL, and swapoff() calls try_to_unuse() early on, which takes the PTL as it iterates over every vma in every mm. It looks like shmem is handled specially by a call to shmem_unuse(), but I can't see the exact serialization mechanism.
I've implemented a batching function, as David suggested above, but I'm trying to convince myself that it is safe for it to access si->swap_map[] without a lock (i.e. that swapoff() can't concurrently free it). But I think free_swap_and_cache() as it already exists depends on being able to access the si without an explicit lock, so I'm assuming the same mechanism will protect my new changes. But I want to be sure I understand the mechanism...
This is the existing free_swap_and_cache(). I think _swap_info_get() would break if this could race with swapoff(), and __swap_entry_free() looks up the cluster from an array, which would also be freed by swapoff if racing:
int free_swap_and_cache(swp_entry_t entry)
{
struct swap_info_struct *p;
unsigned char count;
if (non_swap_entry(entry))
return 1;
p = _swap_info_get(entry);
if (p) {
count = __swap_entry_free(p, entry);
if (count == SWAP_HAS_CACHE)
__try_to_reclaim_swap(p, swp_offset(entry),
TTRS_UNMAPPED | TTRS_FULL);
}
return p != NULL;
}This is my new function. I want to be sure that it's safe to do the READ_ONCE(si->swap_info[...]):
void free_swap_and_cache_nr(swp_entry_t entry, int nr)
{
unsigned long end = swp_offset(entry) + nr;
unsigned type = swp_type(entry);
struct swap_info_struct *si;
unsigned long offset;
if (non_swap_entry(entry))
return;
si = _swap_info_get(entry);
if (!si || end > si->max)
return;
/*
* First free all entries in the range.
*/
for (offset = swp_offset(entry); offset < end; offset++) {
VM_WARN_ON(data_race(!si->swap_map[offset]));
__swap_entry_free(si, swp_entry(type, offset));
}
/*
* Now go back over the range trying to reclaim the swap cache. This is
* more efficient for large folios because we will only try to reclaim
* the swap once per folio in the common case. If we do
* __swap_entry_free() and __try_to_reclaim_swap() in the same loop, the
* latter will get a reference and lock the folio for every individual
* page but will only succeed once the swap slot for every subpage is
* zero.
*/
for (offset = swp_offset(entry); offset < end; offset += nr) {
nr = 1;
if (READ_ONCE(si->swap_map[offset]) == SWAP_HAS_CACHE) { << HERE
/*
* Folios are always naturally aligned in swap so
* advance forward to the next boundary. Zero means no
* folio was found for the swap entry, so advance by 1
* in this case. Negative value means folio was found
* but could not be reclaimed. Here we can still advance
* to the next boundary.
*/
nr = __try_to_reclaim_swap(si, offset,
TTRS_UNMAPPED | TTRS_FULL);
if (nr == 0)
nr = 1;
else if (nr < 0)
nr = -nr;
nr = ALIGN(offset + 1, nr) - offset;
}
}
}Thanks,
Ryan
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 54 lines · Ryan Roberts
Quoted 43 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 20 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Quoted 7 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
Hi,
Quoted 8 lines
I'm struggling to convince myself that
free_swap_and_cache()can't race with withswapoff(). Can anyone explain that this is safe?I *think* they are both serialized by the PTL, since all callers of
free_swap_and_cache()(except shmem) have the PTL, andswapoff()callstry_to_unuse()early on, which takes the PTL as it iterates over every vma in every mm. It looks like shmem is handled specially by a call toshmem_unuse(), but I can't see the exact serialization mechanism.
As get_swap_device() documents:
"if there aren't some other ways to prevent swapoff, such as the folio in swap cache is locked, page table lock is held, etc., the swap entry may become invalid because of swapoff"
PTL it is, in theory. But I'm afraid that's half the story.
Quoted 7 lines
I've implemented a batching function, as David suggested above, but I'm trying to convince myself that it is safe for it to access si->swap_map[] without a lock (i.e. that
swapoff()can't concurrently free it). But I thinkfree_swap_and_cache()as it already exists depends on being able to access the si without an explicit lock, so I'm assuming the same mechanism will protect my new changes. But I want to be sure I understand the mechanism...
Very valid concern.
Quoted 7 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break if this could race withswapoff(), and__swap_entry_free()looks up the cluster from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
Quoted 10 lines
{
struct swap_info_struct *p;
unsigned char count;
if (non_swap_entry(entry))
return 1;
p = _swap_info_get(entry);
if (p) {
count = __swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count == SWAP_HAS_CACHE)
count is now SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed it. That one would have to be reclaimed asynchronously.
The existing code we would call swap_page_trans_huge_swapped() with the SI it obtained via _swap_info_get().
I also don't see what should be left protecting the SI. It's not locked anymore, the swapcounts are at 0. We don't hold the folio lock.
try_to_unuse() will stop as soon as si->inuse_pages is at 0. Hm ...
Would performing the overall operation under lock_cluster_or_swap_info help? Not so sure :(
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 68 lines · David Hildenbrand
Quoted 58 lines · Ryan Roberts
Quoted 47 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines
How relevant is it? Relevant enough that someone decided to put that optimization in? I don't know :)
I'll have one last go at convincing you: Huang Ying (original author) commented "I believe this should be OK. Better to compare the performance too." at [1]. That implies to me that perhaps the optimization wasn't in response to a specific problem after all. Do you have any thoughts, Huang?
Might make sense to include that in the patch description!
Quoted 24 lines
OK so if we really do need to keep this optimization, here are some ideas:
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster
to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these
cases to be rare.1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We
wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster
to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
Taking a step back: what about we simply batch unmapping of swap entries?
That is, if we're unmapping a PTE range, we'll collect swap entries (under PT lock) that reference consecutive swap offsets in the same swap file.
Quoted 7 lines
There, we can then first decrement all the swap counts, and then try minimizing how often we actually have to try reclaiming swap space (lookup folio, see it's a large folio that we cannot reclaim or could reclaim, ...).
Might need some fine-tuning in swap code to "advance" to the next entry to try freeing up, but we certainly can do better than what we would do right now.
Hi,
Quoted 8 lines
I'm struggling to convince myself that
free_swap_and_cache()can't race with withswapoff(). Can anyone explain that this is safe?I *think* they are both serialized by the PTL, since all callers of
free_swap_and_cache()(except shmem) have the PTL, andswapoff()callstry_to_unuse()early on, which takes the PTL as it iterates over every vma in every mm. It looks like shmem is handled specially by a call toshmem_unuse(), but I can't see the exact serialization mechanism.
Quoted 7 lines
As
get_swap_device()documents:"if there aren't some other ways to prevent swapoff, such as the folio in swap cache is locked, page table lock is held, etc., the swap entry may become invalid because of swapoff"
PTL it is, in theory. But I'm afraid that's half the story.
Ahh I didn't notice that comment - thanks!
Quoted 37 lines
Quoted 8 lines
I've implemented a batching function, as David suggested above, but I'm trying to convince myself that it is safe for it to access si->swap_map[] without a lock (i.e. that
swapoff()can't concurrently free it). But I thinkfree_swap_and_cache()as it already exists depends on being able to access the si without an explicit lock, so I'm assuming the same mechanism will protect my new changes. But I want to be sure I understand the mechanism...Very valid concern.
Quoted 11 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break if this could race withswapoff(), and__swap_entry_free()looks up the cluster from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed it. That one would have to be reclaimed asynchronously.
Quoted 7 lines
The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore, the swapcounts are at 0. We don't hold the folio lock.
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...
But, assuming the caller of free_swap_and_cache() acquires the PTL first, I think this all works out ok? While free_swap_and_cache() is running, try_to_unuse() will wait for the PTL. Or if try_to_unuse() runs first, then free_swap_and_cache() will never be called because the swap entry will have been removed from the PTE?
That just leaves shmem... I suspected there might be some serialization between shmem_unuse() (called from try_to_unuse()) and the shmem free_swap_and_cache() callsites, but I can't see it. Hmm...
Quoted 3 lines
Would performing the overall operation under
lock_cluster_or_swap_infohelp? Not so sure :(
No - that function relies on being able to access the cluster from the array in the swap_info and lock it. And I think that array has the same lifetime as swap_map, so same problem. You'd need get_swap_device()/put_swap_device() and a bunch of refactoring for the internals not to take the locks, I guess. I think its doable, just not sure if neccessary...
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 31 lines
Quoted 31 lines
Quoted 10 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break if this could race withswapoff(), and__swap_entry_free()looks up the cluster from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore, the swapcounts are at 0. We don't hold the folio lock.
Quoted 7 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been removed from the PTE?
But can't try_to_unuse() run, detect !si->inuse_pages and not even bother about scanning any further page tables?
But my head hurts from digging through that code.
Let me try again:
__swap_entry_free() might be the last user and result in "count == SWAP_HAS_CACHE".
swapoff->try_to_unuse() will stop as soon as soon as si->inuse_pages==0.
So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed swap_page_trans_huge_swapped().
Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls free_swap_and_cache().
-> count == SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]
Process 2 quits. Calls free_swap_and_cache().
-> count == SWAP_HAS_CACHE
Process 2 goes ahead, passes swap_page_trans_huge_swapped(), and calls __try_to_reclaim_swap().
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);
What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to swap_page_trans_huge_swapped()?
Quoted 6 lines
That just leaves shmem... I suspected there might be some serialization between
shmem_unuse()(called fromtry_to_unuse()) and the shmemfree_swap_and_cache()callsites, but I can't see it. Hmm...Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Would performing the overall operation under
lock_cluster_or_swap_infohelp? Not so sure :(No - that function relies on being able to access the cluster from the array in the
swap_infoand lock it. And I think that array has the same lifetime asswap_map, so same problem. You'd need get_swap_device()/put_swap_device() and a bunch of refactoring for the internals not to take the locks, I guess. I think its doable, just not sure if neccessary...
Agreed.
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 40 lines · David Hildenbrand
Quoted 32 lines
Quoted 32 lines
Quoted 11 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore, the swapcounts are at 0. We don't hold the folio lock.
Quoted 7 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been removed from the PTE?
Quoted 4 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from swapfile.c.
Quoted 27 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed
swap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().
Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing free_swap_and_cache(). try_to_unuse() iterates over every vma in every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - see unuse_pte(). Process 1 must have beaten try_to_unuse() to the particular pte, because if try_to_unuse() got there first, it would have converted it from a swap entry to present pte and process 1 would never even have called free_swap_and_cache(). So try_to_unuse() will eventually wait on the PTL until process 1 has released it after free_swap_and_cache() completes. Am I missing something? Because that part feels pretty clear to me.
Its the shmem case that I'm struggling to explain.
Quoted 20 lines
Quoted 9 lines
That just leaves shmem... I suspected there might be some serialization between
shmem_unuse()(called fromtry_to_unuse()) and the shmemfree_swap_and_cache()callsites, but I can't see it. Hmm...Quoted 1 line
Quoted 8 lines
Quoted 2 lines
Would performing the overall operation under
lock_cluster_or_swap_infohelp? Not so sure :(No - that function relies on being able to access the cluster from the array in the
swap_infoand lock it. And I think that array has the same lifetime asswap_map, so same problem. You'd need get_swap_device()/put_swap_device() and a bunch of refactoring for the internals not to take the locks, I guess. I think its doable, just not sure if neccessary...Agreed.
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 84 lines · Ryan Roberts
Quoted 40 lines · David Hildenbrand
Quoted 32 lines
Quoted 32 lines
Quoted 11 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore, the swapcounts are at 0. We don't hold the folio lock.
Quoted 7 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been removed from the PTE?Quoted 4 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from
swapfile.c.Quoted 27 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed
swap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?
Quoted 10 lines
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing
free_swap_and_cache().try_to_unuse()iterates over every vma in every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - seeunuse_pte(). Process 1 must have beatentry_to_unuse()to the particular pte, because iftry_to_unuse()got there first, it would have converted it from a swap entry to present pte and process 1 would never even have calledfree_swap_and_cache(). Sotry_to_unuse()will eventually wait on the PTL until process 1 has released it afterfree_swap_and_cache()completes. Am I missing something? Because that part feels pretty clear to me.
Why should try_to_unuse() do *anything* if it already findssi->inuse_pages == 0 because we (p1 } p2) just freed the swapentries and process 2 managed to free the last remaining swapcache entry?
I'm probably missing something important :)
try_to_unuse() really starts with
if (!READ_ONCE(si->inuse_pages)) goto success;
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
+ Hugh
Quoted 101 lines · David Hildenbrand
Quoted 89 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 35 lines
Quoted 35 lines
Quoted 12 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster
from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed
it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore,
the swapcounts are at 0. We don't hold the folio lock.Quoted 8 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been
removed from the PTE?Quoted 4 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from
swapfile.c.Quoted 28 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed
swap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?Quoted 10 lines
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing
free_swap_and_cache().try_to_unuse()iterates over every vma in every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - seeunuse_pte(). Process 1 must have beatentry_to_unuse()to the particular pte, because iftry_to_unuse()got there first, it would have converted it from a swap entry to present pte and process 1 would never even have calledfree_swap_and_cache(). Sotry_to_unuse()will eventually wait on the PTL until process 1 has released it afterfree_swap_and_cache()completes. Am I missing something? Because that part feels pretty clear to me.
Quoted 3 lines
Why should
try_to_unuse()do *anything* if it already findssi->inuse_pages== 0 because we (p1 } p2) just freed the swapentries and process 2 managed to free the last remaining swapcache entry?
Yeah ok. For some reason I thought unuse_mm() was iterating over all mms and so the while (READ_ONCE(si->inuse_pages)) was only evaluated after iterating over every mm. Oops.
So yes, I agree with you; I think this is broken. And I'm a bit worried this could be a can of worms; By the same logic, I think folio_free_swap(), swp_swapcount() and probably others are broken in the same way.
I wonder if we are missing something here? I've added Hugh - I see he has a lot of commits in this area, perhaps he has some advice?
Thanks,
Ryan
Quoted 8 lines
I'm probably missing something important :)
try_to_unuse()really starts withif (!READ_ONCE(si->inuse_pages))
goto success;
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 108 lines · Ryan Roberts
+ Hugh
Quoted 101 lines · David Hildenbrand
Quoted 89 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 35 lines
Quoted 35 lines
Quoted 12 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster
from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed
it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore,
the swapcounts are at 0. We don't hold the folio lock.Quoted 8 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been
removed from the PTE?Quoted 4 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from
swapfile.c.Quoted 28 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed
swap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?Quoted 10 lines
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing
free_swap_and_cache().try_to_unuse()iterates over every vma in every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - seeunuse_pte(). Process 1 must have beatentry_to_unuse()to the particular pte, because iftry_to_unuse()got there first, it would have converted it from a swap entry to present pte and process 1 would never even have calledfree_swap_and_cache(). Sotry_to_unuse()will eventually wait on the PTL until process 1 has released it afterfree_swap_and_cache()completes. Am I missing something? Because that part feels pretty clear to me.Quoted 3 lines
Why should
try_to_unuse()do *anything* if it already findssi->inuse_pages== 0 because we (p1 } p2) just freed the swapentries and process 2 managed to free the last remaining swapcache entry?
Quoted 7 lines
Yeah ok. For some reason I thought
unuse_mm()was iterating over all mms and so thewhile (READ_ONCE(si->inuse_pages))was only evaluated after iterating over every mm. Oops.So yes, I agree with you; I think this is broken. And I'm a bit worried this could be a can of worms; By the same logic, I think
folio_free_swap(),swp_swapcount()and probably others are broken in the same way.
Don't worry too much :-), we have get_swap_device() at least. We can insert it anywhere we want because it's quite lightweight. And, because swapoff() is so rare, the race is theoretical only.
For this specific case, I had thought that PTL is enough. But after looking at this more, I found a race here too. Until __swap_entry_free() return, we are OK, nobody can reduce the swap count because we held the PTL. But, after that, even if its return value is SWAP_HAS_CACHE (that is, in swap cache), parallel swap_unuse() or __try_to_reclaim_swap() may remove the folio from swap cache, so free the swap entry. So, swapoff() can proceed to free the data structures in parallel.
To fix the race, we can add get/put_swap_device() in free_swap_and_cache().
For other places, we can check whether get/put_swap_device() has been called in callers, and the swap reference we held has been decreased (e.g., swap count protected by PTL, SWAP_HAS_CACHE protected by folio lock).
Quoted 15 lines
I wonder if we are missing something here? I've added Hugh - I see he has a lot of commits in this area, perhaps he has some advice?
Thanks,
RyanQuoted 8 lines
I'm probably missing something important :)
try_to_unuse()really starts withif (!READ_ONCE(si->inuse_pages))
goto success;
--
Best Regards,
Huang, Ying
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 131 lines · Ying
Quoted 108 lines · Ryan Roberts
+ Hugh
Quoted 101 lines · David Hildenbrand
Quoted 89 lines · Ryan Roberts
Quoted 44 lines · David Hildenbrand
Quoted 35 lines
Quoted 35 lines
Quoted 12 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster
from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed
it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore,
the swapcounts are at 0. We don't hold the folio lock.Quoted 8 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been
removed from the PTE?Quoted 4 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from
swapfile.c.Quoted 28 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0, before we completed
swap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still references by swap entries.
Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?Quoted 10 lines
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing
free_swap_and_cache().try_to_unuse()iterates over every vma in every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - seeunuse_pte(). Process 1 must have beatentry_to_unuse()to the particular pte, because iftry_to_unuse()got there first, it would have converted it from a swap entry to present pte and process 1 would never even have calledfree_swap_and_cache(). Sotry_to_unuse()will eventually wait on the PTL until process 1 has released it afterfree_swap_and_cache()completes. Am I missing something? Because that part feels pretty clear to me.Quoted 3 lines
Why should
try_to_unuse()do *anything* if it already findssi->inuse_pages== 0 because we (p1 } p2) just freed the swapentries and process 2 managed to free the last remaining swapcache entry?Quoted 7 lines
Yeah ok. For some reason I thought
unuse_mm()was iterating over all mms and so thewhile (READ_ONCE(si->inuse_pages))was only evaluated after iterating over every mm. Oops.So yes, I agree with you; I think this is broken. And I'm a bit worried this could be a can of worms; By the same logic, I think
folio_free_swap(),swp_swapcount()and probably others are broken in the same way.Don't worry too much :-), we have
get_swap_device()at least. We can insert it anywhere we want because it's quite lightweight. And, becauseswapoff()is so rare, the race is theoretical only.For this specific case, I had thought that PTL is enough. But after looking at this more, I found a race here too. Until
__swap_entry_free()return, we are OK, nobody can reduce the swap count because we held the PTL. But, after that, even if its return value isSWAP_HAS_CACHE(that is, in swap cache), parallelswap_unuse()or__try_to_reclaim_swap()may remove the folio from swap cache, so free the swap entry. So,swapoff()can proceed to free the data structures in parallel.
Quoted 7 lines
To fix the race, we can add get/put_swap_device() in
free_swap_and_cache().For other places, we can check whether get/put_swap_device() has been called in callers, and the swap reference we held has been decreased (e.g., swap count protected by PTL,
SWAP_HAS_CACHEprotected by folio lock).
Yes, sounds reasonable. We should likely update the documentation of get_swap_device(), that after decrementing the refcount, the SI might become stale and should not be touched without a prior get_swap_device().
Signature · 3 lines
Cheers, David / dhildenb
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 124 lines · David Hildenbrand
Quoted 123 lines · Ying
Quoted 121 lines · Ryan Roberts
+ Hugh
Quoted 109 lines · David Hildenbrand
Quoted 93 lines · Ryan Roberts
Quoted 45 lines · David Hildenbrand
Quoted 36 lines
Quoted 36 lines
Quoted 12 lines
This is the existing
free_swap_and_cache(). I think_swap_info_get()would break
if this could race withswapoff(), and__swap_entry_free()looks up the cluster
from an array, which would also be freed by swapoff if racing:int
free_swap_and_cache(swp_entry_t entry)
{
structswap_info_struct*p;
unsigned char count;Quoted 6 lines
if (
non_swap_entry(entry))
return 1;p =
_swap_info_get(entry);
if (p) {
count =__swap_entry_free(p, entry);If count dropped to 0 and
Quoted 1 line
if (count ==
SWAP_HAS_CACHE)count is now
SWAP_HAS_CACHE, there is in fact no swap entry anymore. We removed
it. That one would have to be reclaimed asynchronously.The existing code we would call
swap_page_trans_huge_swapped()with the SI it
obtained via_swap_info_get().I also don't see what should be left protecting the SI. It's not locked anymore,
the swapcounts are at 0. We don't hold the folio lock.Quoted 8 lines
Quoted 1 line
try_to_unuse()will stop as soon assi->inuse_pagesis at 0. Hm ...But, assuming the caller of
free_swap_and_cache()acquires the PTL first, I think this all works out ok? Whilefree_swap_and_cache()is running,try_to_unuse()will wait for the PTL. Or iftry_to_unuse()runs first, thenfree_swap_and_cache()will never be called because the swap entry will have been
removed from the PTE?Quoted 5 lines
But can't
try_to_unuse()run, detect !si->inuse_pages and not even bother about
scanning any further page tables?But my head hurts from digging through that code.
Yep, glad I'm not the only one that gets headaches from
swapfile.c.Quoted 30 lines
Let me try again:
__swap_entry_free()might be the last user and result in "count ==SWAP_HAS_CACHE".
swapoff->try_to_unuse()will stop as soon as soon as si->inuse_pages==0.So the question is: could someone reclaim the folio and turn si->inuse_pages==0,
before we completedswap_page_trans_huge_swapped().Imagine the following: 2 MiB folio in the swapcache. Only 2 subpages are still
references by swap entries.Process 1 still references subpage 0 via swap entry. Process 2 still references subpage 1 via swap entry.
Process 1 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHE
[then, preempted in the hypervisor etc.]Process 2 quits. Calls
free_swap_and_cache().
-> count ==SWAP_HAS_CACHEProcess 2 goes ahead, passes
swap_page_trans_huge_swapped(), and calls__try_to_reclaim_swap().Quoted 8 lines
__try_to_reclaim_swap()->folio_free_swap()->delete_from_swap_cache()->put_swap_folio()-> free_swap_slot()->swapcache_free_entries()->swap_entry_free()->swap_range_free()->
...WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries);What stops swapoff to succeed after process 2 reclaimed the swap cache but before process 1 finished its call to
swap_page_trans_huge_swapped()?Quoted 14 lines
Assuming you are talking about anonymous memory, process 1 has the PTL while it's executing
free_swap_and_cache().try_to_unuse()iterates over every vma in
every mm, and it swaps-in a page for every PTE that holds a swap entry for the device being swapoff'ed. It takes the PTL while converting the swap entry to present PTE - seeunuse_pte(). Process 1 must have beatentry_to_unuse()to the
particular pte, because iftry_to_unuse()got there first, it would have converted it from a swap entry to present pte and process 1 would never even have calledfree_swap_and_cache(). Sotry_to_unuse()will eventually wait on the
PTL until process 1 has released it afterfree_swap_and_cache()completes. Am I
missing something? Because that part feels pretty clear to me.Quoted 4 lines
Why should
try_to_unuse()do *anything* if it already findssi->inuse_pages== 0 because we (p1 } p2) just freed the swapentries and process
2 managed to free the last remaining swapcache entry?Yeah ok. For some reason I thought
unuse_mm()was iterating over all mms and so thewhile (READ_ONCE(si->inuse_pages))was only evaluated after iterating over every mm. Oops.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
So yes, I agree with you; I think this is broken. And I'm a bit worried this could be a can of worms; By the same logic, I think
folio_free_swap(),swp_swapcount()and probably others are broken in the same way.Don't worry too much :-), we have
get_swap_device()at least. We can insert it anywhere we want because it's quite lightweight. And, becauseswapoff()is so rare, the race is theoretical only.
Thanks for the response!
Quoted 5 lines
Quoted 5 lines
For this specific case, I had thought that PTL is enough. But after looking at this more, I found a race here too. Until
__swap_entry_free()return, we are OK, nobody can reduce the swap count because we held the PTL.
Even that is not true for the shmem case: As far as I can see, shmem doesn't have the PTL or any other synchronizing lock when it calls free_swap_and_cache(). I don't think that changes anything solution-wise though.
Quoted 9 lines
Quoted 9 lines
But, after that, even if its return value is
SWAP_HAS_CACHE(that is, in swap cache), parallelswap_unuse()or__try_to_reclaim_swap()may remove the folio from swap cache, so free the swap entry. So,swapoff()can proceed to free the data structures in parallel.To fix the race, we can add get/put_swap_device() in
free_swap_and_cache().
Quoted 8 lines
Quoted 4 lines
For other places, we can check whether get/put_swap_device() has been called in callers, and the swap reference we held has been decreased (e.g., swap count protected by PTL,
SWAP_HAS_CACHEprotected by folio lock).Yes, sounds reasonable. We should likely update the documentation of
get_swap_device(), that after decrementing the refcount, the SI might become stale and should not be touched without a priorget_swap_device().
Yep agreed. If nobody else is planning to do it, I'll try to create a test case that provokes the problem then put a patch together to fix it.
Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
In reply to Ryan RobertsQuoted 18 lines · Ryan Roberts
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.
Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
I can't quite bring myself to look up the encoding of swap entries but as long as we're willing to restrict ourselves to naturally aligning the clusters, there's an encoding (which I believe I invented) that lets us encode arbitrary power-of-two sizes with a single bit.
I describe it here:
https://kernelnewbies.org/MatthewWilcox/NaturallyAlignedOrder
Let me know if it's not clear.
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 32 lines · Matthew Wilcox
Quoted 18 lines · Ryan Roberts
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
I can't quite bring myself to look up the encoding of swap entries but as long as we're willing to restrict ourselves to naturally aligning the clusters, there's an encoding (which I believe I invented) that lets us encode arbitrary power-of-two sizes with a single bit.
Quoted 4 lines
I describe it here:
https://kernelnewbies.org/MatthewWilcox/NaturallyAlignedOrderLet me know if it's not clear.
Ahh yes, I'm familiar with this encoding scheme from other settings. Although I've previously thought of it as having a bit to indicate whether the scheme is enabled or not, and if it is enabled then the encoded PFN is:
PFNe = PFNd | (1 << (log2(n) - 1))
Where n is the power-of-2 page count.
Same thing, I think.
I think we would have to steal a bit from the offset to make this work, and it looks like the size of that is bottlnecked on the arch's swp_entry PTE representation. Looks like there is a MIPS config that only has 17 bits for offset to begin with, so I doubt we would be able to spare a bit here? Although it looks possible that there are some unused low bits that could be used...
Ryan Roberts
reviewerRe: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
Quoted 46 lines · Ryan Roberts
Quoted 32 lines · Matthew Wilcox
Quoted 18 lines · Ryan Roberts
Fundamentally, we would like to be able to figure out the size of the swap slot from the swap entry. Today swap supports 2 sizes;
PAGE_SIZEandPMD_SIZE. ForPMD_SIZE, it always uses a full cluster, so can easily add a flag to the cluster to mark it asPMD_SIZE.Going forwards, we want to support all sizes (power-of-2). Most of the time, a cluster will contain only one size of THPs, but this is not the case when a THP in the swapcache gets split or when an order-0 slot gets stolen. We expect these cases to be rare.
1) Keep the size of the smallest swap entry in the cluster header. Most of the time it will be the full size of the swap entry, but sometimes it will cover only a portion. In the latter case you may see a false negative for
swap_page_trans_huge_swapped()meaning we take the slow path, but that is rare. There is one wrinkle: currently the HUGE flag is cleared input_swap_folio(). We wouldn't want to do the equivalent in the new scheme (i.e. set the whole cluster to order-0). I think that is safe, but haven't completely convinced myself yet.Quoted 7 lines
2) allocate 4 bits per (small) swap slot to hold the order. This will give precise information and is conceptually simpler to understand, but will cost more memory (half as much as the initial swap_map[] again).
I still prefer to avoid this at all if we can (and would like to hear Huang's thoughts). But if its a choice between 1 and 2, I prefer 1 - I'll do some prototyping.
I can't quite bring myself to look up the encoding of swap entries but as long as we're willing to restrict ourselves to naturally aligning the clusters, there's an encoding (which I believe I invented) that lets us encode arbitrary power-of-two sizes with a single bit.
Quoted 4 lines
I describe it here:
https://kernelnewbies.org/MatthewWilcox/NaturallyAlignedOrderLet me know if it's not clear.
Ahh yes, I'm familiar with this encoding scheme from other settings. Although I've previously thought of it as having a bit to indicate whether the scheme is enabled or not, and if it is enabled then the encoded PFN is:
PFNe = PFNd | (1 << (log2(n) - 1))
Where n is the power-of-2 page count.
Quoted 8 lines
Same thing, I think.
I think we would have to steal a bit from the offset to make this work, and it looks like the size of that is bottlnecked on the arch's
swp_entryPTE representation. Looks like there is a MIPS config that only has 17 bits for offset to begin with, so I doubt we would be able to spare a bit here? Although it looks possible that there are some unused low bits that could be used...
I think the other problem with this is that it won't tell us which slot in the "swap slot block" each entry is targetting?
Ryan Roberts
reviewerPATCH v3 2/4 mm: swap: Remove struct percpu_cluster
In reply to Ryan Robertsstruct percpu_cluster stores the index of cpu's current cluster and the offset of the next entry that will be allocated for the cpu. These two pieces of information are redundant because the cluster index is just (offset / SWAPFILE_CLUSTER). The only reason for explicitly keeping the cluster index is because the structure used for it also has a flag to indicate "no cluster". However this data structure also contains a spin lock, which is never used in this context, as a side effect the code copies the spinlock_t structure, which is questionable coding practice in my view.
So let's clean this up and store only the next offset, and use a sentinal value (SWAP_NEXT_NULL) to indicate "no cluster". SWAP_NEXT_NULL is chosen to be 0, because 0 will never be seen legitimately; The first page in the swap file is the swap header, which is always marked bad to prevent it from being allocated as an entry. This also prevents the cluster to which it belongs being marked free, so it will never appear on the free list.
This change saves 16 bytes per cpu. And given we are shortly going to extend this mechanism to be per-cpu-AND-per-order, we will end up saving 16 * 9 = 144 bytes per cpu, which adds up if you have 256 cpus in the system.
--- include/linux/swap.h | 21 +++++++++++++-------- mm/swapfile.c | 43 +++++++++++++++++++------------------------ 2 files changed, 32 insertions(+), 32 deletions(-)
include/linux/swap.h +13 -8
@@ -261,14 +261,12 @@ struct swap_cluster_info {#define CLUSTER_FLAG_NEXT_NULL 2/* This cluster has no next cluster */ /* -* We assign a cluster to each CPU, so each CPU can allocate swap entry from -* its own cluster and swapout sequentially. The purpose is to optimize swapout -* throughput. +* The first page in the swap file is the swap header, which is always marked +* bad to prevent it from being allocated as an entry. This also prevents the +* cluster to which it belongs being marked free. Therefore 0 is safe to use as +* a sentinel to indicate cpu_next is not valid in swap_info_struct. */ -struct percpu_cluster { -struct swap_cluster_info index;/* Current cluster index */ -unsigned int next;/* Likely next allocation offset */ -}; +#define SWAP_NEXT_NULL 0struct swap_cluster_list {struct swap_cluster_info head;
@@ -295,7 +293,14 @@ struct swap_info_struct {unsigned int cluster_next;/* likely index for next allocation */ unsigned int cluster_nr;/* countdown to next cluster search */ unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ -struct percpu_cluster __percpu *percpu_cluster;/* per cpu's swap location */ +unsigned int __percpu *cpu_next;/* +* Likely next allocation offset. We +* assign a cluster to each CPU, so each +* CPU can allocate swap entry from its +* own cluster and swapout sequentially. +* The purpose is to optimize swapout +* throughput. +*/ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ struct file *swap_file;/* seldom referenced */
mm/swapfile.c +19 -24
103 lines not on this page · raw message
Signature · 1 line
2.25.1
Ryan Roberts
reviewerPATCH v3 3/4 mm: swap: Simplify ssd behavior when scanner steals entry
In reply to Ryan RobertsWhen a CPU fails to reserve a cluster (due to free list exhaustion), we revert to the scanner to find a free entry somewhere in the swap file. This might cause an entry to be stolen from another CPU's reserved cluster. Upon noticing this, the CPU with the stolen entry would previously scan forward to the end of the cluster trying to find a free entry to use. If there were none, it would try to reserve a new pre-cpu cluster and allocate from that.
This scanning behavior does not scale well to high-order allocations, which will be introduced in a future patch since would need to scan for a contiguous area that was naturally aligned. Given stealing is a rare occurrence, let's remove the scanning behavior from the ssd allocator and simply drop the cluster and try to allocate a new one. Given the purpose of the per-cpu cluster is to ensure a given task's pages are sequential on disk to aid readahead, allocating a new cluster at this point makes most sense.
Furthermore, si->max will always be greater than or equal to the end of the last cluster because any partial cluster will never be put on the free cluster list. Therefore we can simplify this logic too.
These changes make it simpler to generalize scan_swap_map_try_ssd_cluster() to handle any allocation order.
--- mm/swapfile.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
mm/swapfile.c +9 -12
@@ -639,27 +639,24 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,/* * Other CPUs can use our cluster if they can't find a free cluster, -* check if there is still free entry in the cluster +* check if the expected entry is still free. If not, drop it and +* reserve a new cluster. */ - max = min_t(unsigned long, si->max, -ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ); -if (tmp < max) { - ci =lock_cluster (si, tmp); -while (tmp < max) { -if (!si->swap_map[tmp]) -break ; - tmp++; - } + ci =lock_cluster (si, tmp); +if (si->swap_map[tmp]) {unlock_cluster (ci); - } -if (tmp >= max) { *cpu_next =SWAP_NEXT_NULL ; goto new_cluster; } +unlock_cluster (ci); + *offset = tmp; *scan_base = tmp; + + max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ; tmp +=1 ; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ; +return true; }
Signature · 1 line
2.25.1
Ryan Roberts
reviewerPATCH v3 4/4 mm: swap: Swap-out small-sized THP without splitting
In reply to Ryan RobertsThe upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.
Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 << PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).
The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern. __scan_swap_map_try_ssd_cluster() is introduced to deal with arbitrary orders and scan_swap_map_try_ssd_cluster() is refactored as a wrapper for order-0.
As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run usemem from vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:
Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |
So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)
include/linux/swap.h +5 -5
@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each -* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */
mm/swapfile.c +108 -41
255 lines not on this page · raw message
mm/vmscan.c +6 -4
@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;
Signature · 1 line
2.25.1
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Hi, Ryan,
Quoted 75 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +0
-1@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each
include/linux/swap.h struct swap_info_struct { · line 303
Quoted 10 lines
include/linux/swap.h +5
-4-* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */
This is kind of hard to understand. Better to define some intermediate data structure to improve readability. For example,
#ifdef CONFIG_THP_SWAP
#define NR_SWAP_ORDER PMD_ORDER
#else
#define NR_SWAP_ORDER 1
#endif
struct percpu_clusters {
unsigned int alloc_next[NR_SWAP_ORDER];
};PMD_ORDER isn't a constant on powerpc, but THP_SWAP isn't supported on powerpc too.
Quoted 56 lines
include/linux/swap.h +0
-0struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +19
-7@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;
mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 616
Quoted 5 lines
mm/swapfile.c +1
-1@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false; - *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;
This is added in the previous patch. I don't think SWAP_NEXT_NULL is a good name. Because NEXT isn't a pointer (while cluster_next is). Better to name it as SWAP_NEXT_INVALID, etc.
Quoted 56 lines
mm/swapfile.c +25
-10return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);
mm/swapfile.c static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, · line 683
Quoted 3 lines
mm/swapfile.c +0
-0@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ;
This line is added in a previous patch. Can we just use
max = ALIGN(tmp + 1, SWAPFILE_CLUSTER);
Or, add ALIGN_UP() for this?
Quoted 21 lines
mm/swapfile.c +11
-1- tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1023
Quoted 6 lines
mm/swapfile.c +2
-1@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages)
IMHO, it's better to make scan_swap_map_slots() to support order > 0 instead of making swap_alloc_cluster() to support order != PMD_ORDER. And, we may merge swap_alloc_cluster() with scan_swap_map_slots() after that.
Quoted 137 lines
mm/swapfile.c +50
-21{ -unsigned long idx;struct swap_cluster_info *ci; -unsigned long offset; +unsigned long offset, scan_base; +int order =ilog2 (nr_pages); +bool ret;/* -* Should not even be attempting cluster allocations when huge +* Should not even be attempting large allocations when huge * page swap is disabled. Warn and fail the allocation. */ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages <2 || nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) {VM_WARN_ON_ONCE (1 );return 0 ; } -if (cluster_list_empty (&si->free_clusters)) +/* +* Swapfile is not block device or not using clusters so unable to +* allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info)return 0 ; - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); +again: +/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ; + +/* +* If we failed to allocate or if swapoff is waiting for us (due to lock +* being dropped for discard above), return immediately. +*/ +if (!ret || !(si->flags &SWP_WRITEOK )) +return 0 ; -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); +if (__scan_swap_map_ssd_cluster_conflict (si, offset, order)) + goto again; + + ci =lock_cluster (si, offset); +memset (si->swap_map + offset,SWAP_HAS_CACHE , nr_pages); +add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); +swap_range_alloc (si, offset, nr_pages); + *slot =swp_entry (si->type, offset);return 1 ; }@@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int node;/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); +if (size >1 ) { + n_ret =swap_alloc_large (si, swp_entries, size); }else n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , n_goal, swp_entries);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (p->bdev &&bdev_nonrot (p->bdev)) {int cpu;unsigned long ci, nr_cluster; +int nr_order; +int i; p->flags |=SWP_SOLIDSTATE ; p->cluster_next_cpu = alloc_percpu(unsigned int);@@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)for (ci =0 ; ci < nr_cluster; ci++)spin_lock_init (&((cluster_info + ci)->lock)); - p->cpu_next = alloc_percpu(unsigned int); + nr_order =IS_ENABLED (CONFIG_THP_SWAP ) ?PMD_ORDER +1 :1 ; + p->cpu_next =__alloc_percpu (sizeof (unsigned int ) * nr_order, + __alignof__(unsigned int ));if (!p->cpu_next) { error = -ENOMEM ; goto bad_swap_unlock_inode; } -for_each_possible_cpu (cpu) -per_cpu (*p->cpu_next, cpu) =SWAP_NEXT_NULL ; +for_each_possible_cpu (cpu) { +unsigned int *cpu_next =per_cpu_ptr (p->cpu_next, cpu); + +for (i =0 ; i < nr_order; i++) + cpu_next[i] =SWAP_NEXT_NULL ; + } }else {atomic_inc (&nr_rotate_swap); inced_nr_rotate_swap = true;mm/vmscan.c +1
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 10 lines
mm/vmscan.c +5
-1+* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 99 lines · Ying
Hi, Ryan,
Quoted 75 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +0
-1@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each include/linux/swap.h struct swap_info_struct { · line 303
Quoted 10 lines
include/linux/swap.h +5
-4-* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ This is kind of hard to understand. Better to define some intermediate data structure to improve readability. For example,
#ifdef
CONFIG_THP_SWAP
#defineNR_SWAP_ORDERPMD_ORDER
#else
#defineNR_SWAP_ORDER1
#endif
Quoted 6 lines
struct percpu_clusters { unsigned int alloc_next[NR_SWAP_ORDER]; };
PMD_ORDERisn't a constant on powerpc, butTHP_SWAPisn't supported on powerpc too.
I get your point, but this is just making it more difficult for powerpc to ever enable the feature in future - you're implicitly depending on !powerpc, which seems fragile. How about if I change the first line of the coment to be "per-cpu array indexed by allocation order"? Would that be enough?
Quoted 60 lines
Quoted 60 lines
include/linux/swap.h +0
-0struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +19
-7@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false;
Quoted 6 lines
mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 616
Quoted 2 lines
mm/swapfile.c +1
-1- *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;This is added in the previous patch. I don't think
SWAP_NEXT_NULLis a good name. Because NEXT isn't a pointer (whilecluster_nextis). Better to name it asSWAP_NEXT_INVALID, etc.
ACK, will make change for next version.
Quoted 56 lines
Quoted 56 lines
mm/swapfile.c +25
-10return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);
Quoted 8 lines
mm/swapfile.c static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, · line 683
Quoted 4 lines
mm/swapfile.c +0
-0@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ;This line is added in a previous patch. Can we just use
max = ALIGN(tmp + 1, SWAPFILE_CLUSTER);
Sure. This is how I originally had it, but then decided that the other approach was a bit clearer. But I don't have a strong opinion, so I'll change it as you suggest.
Quoted 27 lines
Or, add
ALIGN_UP()for this?Quoted 24 lines
mm/swapfile.c +11
-1- tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; }
Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1023
Quoted 3 lines
mm/swapfile.c +2
-1-static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages)IMHO, it's better to make
scan_swap_map_slots()to support order > 0 instead of makingswap_alloc_cluster()to support order !=PMD_ORDER. And, we may mergeswap_alloc_cluster()withscan_swap_map_slots()after that.
I did consider adding a 5th patch to rename swap_alloc_large() to something like swap_alloc_one_ssd_entry() (which would then be used for order=0 too) and refactor scan_swap_map_slots() to fully delegate to it for the non-scaning ssd allocation case. Would something like that suit?
I have reservations about making scan_swap_map_slots() take an order and be the sole entry point:
- in the non-ssd case, we can't support order!=0
- there is a lot of other logic to deal with falling back to scanning which we would only want to do for order==0, so we would end up with a few ugly conditionals against order.
- I was concerned the risk of me introducing a bug when refactoring all that subtle logic was high
What do you think? Is not making scan_swap_map_slots() support order > 0 a deal breaker for you?
Thanks,
Ryan
Quoted 152 lines
Quoted 138 lines
mm/swapfile.c +50
-21{ -unsigned long idx;struct swap_cluster_info *ci; -unsigned long offset; +unsigned long offset, scan_base; +int order =ilog2 (nr_pages); +bool ret;/* -* Should not even be attempting cluster allocations when huge +* Should not even be attempting large allocations when huge * page swap is disabled. Warn and fail the allocation. */ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages <2 || nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) {VM_WARN_ON_ONCE (1 );return 0 ; } -if (cluster_list_empty (&si->free_clusters)) +/* +* Swapfile is not block device or not using clusters so unable to +* allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info)return 0 ; - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); +again: +/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ; + +/* +* If we failed to allocate or if swapoff is waiting for us (due to lock +* being dropped for discard above), return immediately. +*/ +if (!ret || !(si->flags &SWP_WRITEOK )) +return 0 ; -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); +if (__scan_swap_map_ssd_cluster_conflict (si, offset, order)) + goto again; + + ci =lock_cluster (si, offset); +memset (si->swap_map + offset,SWAP_HAS_CACHE , nr_pages); +add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); +swap_range_alloc (si, offset, nr_pages); + *slot =swp_entry (si->type, offset);return 1 ; }@@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int node;/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); +if (size >1 ) { + n_ret =swap_alloc_large (si, swp_entries, size); }else n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , n_goal, swp_entries);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (p->bdev &&bdev_nonrot (p->bdev)) {int cpu;unsigned long ci, nr_cluster; +int nr_order; +int i; p->flags |=SWP_SOLIDSTATE ; p->cluster_next_cpu = alloc_percpu(unsigned int);@@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)for (ci =0 ; ci < nr_cluster; ci++)spin_lock_init (&((cluster_info + ci)->lock)); - p->cpu_next = alloc_percpu(unsigned int); + nr_order =IS_ENABLED (CONFIG_THP_SWAP ) ?PMD_ORDER +1 :1 ; + p->cpu_next =__alloc_percpu (sizeof (unsigned int ) * nr_order, + __alignof__(unsigned int ));if (!p->cpu_next) { error = -ENOMEM ; goto bad_swap_unlock_inode; } -for_each_possible_cpu (cpu) -per_cpu (*p->cpu_next, cpu) =SWAP_NEXT_NULL ; +for_each_possible_cpu (cpu) { +unsigned int *cpu_next =per_cpu_ptr (p->cpu_next, cpu); + +for (i =0 ; i < nr_order; i++) + cpu_next[i] =SWAP_NEXT_NULL ; + } }else {atomic_inc (&nr_rotate_swap); inced_nr_rotate_swap = true;mm/vmscan.c +1
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 10 lines
mm/vmscan.c +5
-1+* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;--
Best Regards,
Huang, Ying
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 104 lines · Ryan Roberts
Quoted 103 lines · Ying
Hi, Ryan,
Quoted 75 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +0
-1@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each include/linux/swap.h struct swap_info_struct { · line 303
Quoted 10 lines
include/linux/swap.h +5
-4-* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ This is kind of hard to understand. Better to define some intermediate data structure to improve readability. For example,
#ifdef
CONFIG_THP_SWAP
#defineNR_SWAP_ORDERPMD_ORDER
#else
#defineNR_SWAP_ORDER1
#endifstruct percpu_clusters { unsigned int alloc_next[NR_SWAP_ORDER]; };
Quoted 7 lines
Quoted 2 lines
PMD_ORDERisn't a constant on powerpc, butTHP_SWAPisn't supported on powerpc too.I get your point, but this is just making it more difficult for powerpc to ever enable the feature in future - you're implicitly depending on !powerpc, which seems fragile. How about if I change the first line of the coment to be "per-cpu array indexed by allocation order"? Would that be enough?
Even if PMD_ORDER isn't constant on powerpc, it's not necessary for NR_SWAP_ORDER to be variable. At least (1 << (NR_SWAP_ORDER-1)) should < SWAPFILE_CLUSTER. When someone adds THP swap support on powerpc, he can choose a reasonable constant for NR_SWAP_ORDER (for example, 10 or 7).
Quoted 60 lines
Quoted 60 lines
Quoted 60 lines
include/linux/swap.h +0
-0struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +19
-7@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false;
Quoted 8 lines
Quoted 6 lines
mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 616
Quoted 2 lines
mm/swapfile.c +1
-1- *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;This is added in the previous patch. I don't think
SWAP_NEXT_NULLis a good name. Because NEXT isn't a pointer (whilecluster_nextis). Better to name it asSWAP_NEXT_INVALID, etc.ACK, will make change for next version.
Thanks!
Quoted 63 lines
Quoted 63 lines
Quoted 57 lines
mm/swapfile.c +25
-10return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);mm/swapfile.c static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, · line 683
Quoted 3 lines
mm/swapfile.c +0
-0@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ;This line is added in a previous patch. Can we just use
Quoted 5 lines
Quoted 1 line
max = ALIGN(tmp + 1, SWAPFILE_CLUSTER);Sure. This is how I originally had it, but then decided that the other approach was a bit clearer. But I don't have a strong opinion, so I'll change it as you suggest.
Thanks!
Quoted 41 lines
Quoted 27 lines
Or, add
ALIGN_UP()for this?Quoted 24 lines
mm/swapfile.c +11
-1- tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; }Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1023
Quoted 3 lines
mm/swapfile.c +2
-1-static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages)IMHO, it's better to make
scan_swap_map_slots()to support order > 0 instead of makingswap_alloc_cluster()to support order !=PMD_ORDER. And, we may mergeswap_alloc_cluster()withscan_swap_map_slots()after that.I did consider adding a 5th patch to rename
swap_alloc_large()to something likeswap_alloc_one_ssd_entry()(which would then be used for order=0 too) and refactorscan_swap_map_slots()to fully delegate to it for the non-scaning ssd allocation case. Would something like that suit?
Quoted 4 lines
I have reservations about making
scan_swap_map_slots()take an order and be the sole entry point:- in the non-ssd case, we can't support order!=0
Don't need to check ssd directly, we only support order != 0 if si->cluster_info != NULL.
Quoted 3 lines
- there is a lot of other logic to deal with falling back to scanning which we would only want to do for order==0, so we would end up with a few ugly conditionals against order.
We don't need to care about them in most cases. IIUC, only the "goto scan" after scan_swap_map_try_ssd_cluster() return false need to "goto no_page" for order != 0.
Quoted 2 lines
- I was concerned the risk of me introducing a bug when refactoring all that subtle logic was high
IMHO, readability is more important for long term maintenance. So, we need to refactor the existing code for that.
Quoted 2 lines
What do you think? Is not making
scan_swap_map_slots()support order > 0 a deal breaker for you?
I just think that it's better to use scan_swap_map_slots() for any order other than PMD_ORDER. In that way, we share as much code as possible.
--
Best Regards,
Huang, Ying
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 322 lines · Ying
Quoted 104 lines · Ryan Roberts
Quoted 103 lines · Ying
Hi, Ryan,
Quoted 75 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +0
-1@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each include/linux/swap.h struct swap_info_struct { · line 303
Quoted 10 lines
include/linux/swap.h +5
-4-* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ This is kind of hard to understand. Better to define some intermediate data structure to improve readability. For example,
#ifdef
CONFIG_THP_SWAP
#defineNR_SWAP_ORDERPMD_ORDER
#else
#defineNR_SWAP_ORDER1
#endifstruct percpu_clusters { unsigned int alloc_next[NR_SWAP_ORDER]; };Quoted 7 lines
Quoted 2 lines
PMD_ORDERisn't a constant on powerpc, butTHP_SWAPisn't supported on powerpc too.I get your point, but this is just making it more difficult for powerpc to ever enable the feature in future - you're implicitly depending on !powerpc, which seems fragile. How about if I change the first line of the coment to be "per-cpu array indexed by allocation order"? Would that be enough?
Even if
PMD_ORDERisn't constant on powerpc, it's not necessary forNR_SWAP_ORDERto be variable. At least (1 << (NR_SWAP_ORDER-1)) should <SWAPFILE_CLUSTER. When someone adds THP swap support on powerpc, he can choose a reasonable constant forNR_SWAP_ORDER(for example, 10 or 7).Quoted 60 lines
Quoted 60 lines
Quoted 60 lines
include/linux/swap.h +0
-0struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +19
-7@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false;Quoted 8 lines
Quoted 6 lines
mm/swapfile.c scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si, · line 616
Quoted 2 lines
mm/swapfile.c +1
-1- *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;This is added in the previous patch. I don't think
SWAP_NEXT_NULLis a good name. Because NEXT isn't a pointer (whilecluster_nextis). Better to name it asSWAP_NEXT_INVALID, etc.ACK, will make change for next version.
Thanks!
Quoted 63 lines
Quoted 63 lines
Quoted 57 lines
mm/swapfile.c +25
-10return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);mm/swapfile.c static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, · line 683
Quoted 3 lines
mm/swapfile.c +0
-0@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ;This line is added in a previous patch. Can we just use
Quoted 5 lines
Quoted 1 line
max = ALIGN(tmp + 1, SWAPFILE_CLUSTER);Sure. This is how I originally had it, but then decided that the other approach was a bit clearer. But I don't have a strong opinion, so I'll change it as you suggest.
Thanks!
Quoted 41 lines
Quoted 27 lines
Or, add
ALIGN_UP()for this?Quoted 24 lines
mm/swapfile.c +11
-1- tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; }Quoted 8 lines
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1023
Quoted 3 lines
mm/swapfile.c +2
-1-static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages)IMHO, it's better to make
scan_swap_map_slots()to support order > 0 instead of makingswap_alloc_cluster()to support order !=PMD_ORDER. And, we may mergeswap_alloc_cluster()withscan_swap_map_slots()after that.I did consider adding a 5th patch to rename
swap_alloc_large()to something likeswap_alloc_one_ssd_entry()(which would then be used for order=0 too) and refactorscan_swap_map_slots()to fully delegate to it for the non-scaning ssd allocation case. Would something like that suit?Quoted 4 lines
I have reservations about making
scan_swap_map_slots()take an order and be the sole entry point:- in the non-ssd case, we can't support order!=0
Don't need to check ssd directly, we only support order != 0 if
si->cluster_info!= NULL.Quoted 3 lines
- there is a lot of other logic to deal with falling back to scanning which we would only want to do for order==0, so we would end up with a few ugly conditionals against order.
We don't need to care about them in most cases. IIUC, only the "goto scan" after
scan_swap_map_try_ssd_cluster()return false need to "gotono_page" for order != 0.Quoted 2 lines
- I was concerned the risk of me introducing a bug when refactoring all that subtle logic was high
Quoted 8 lines
IMHO, readability is more important for long term maintenance. So, we need to refactor the existing code for that.
Quoted 2 lines
What do you think? Is not making
scan_swap_map_slots()support order > 0 a deal breaker for you?I just think that it's better to use
scan_swap_map_slots()for any order other thanPMD_ORDER. In that way, we share as much code as possible.
OK, I'll take a look at implementing it as you propose, although I likely won't have bandwidth until start of December. Will repost once I have something.
Thanks,
Ryan
Quoted 4 lines
--
Best Regards,
Huang, Ying
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Ryan RobertsQuoted 8 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Quoted 6 lines
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.
Hi Ryan,
We had a problem while enabling THP SWP on arm64, commit d0637c505f8 ("arm64: enable THP_SWAP for arm64")
this means we have to depend on !system_supports_mte(). static inline bool arch_thp_swp_supported(void)
{
return !system_supports_mte();
}Do we have the same problem for small-sized THP? If yes, MTE has been widely existing in various ARM64 SoC. Does it mean we should begin to fix the issue now?
Quoted 367 lines
Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +5
-5@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each -* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +108
-41@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false; - *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ; - tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages) { -unsigned long idx;struct swap_cluster_info *ci; -unsigned long offset; +unsigned long offset, scan_base; +int order =ilog2 (nr_pages); +bool ret;/* -* Should not even be attempting cluster allocations when huge +* Should not even be attempting large allocations when huge * page swap is disabled. Warn and fail the allocation. */ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages <2 || nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) {VM_WARN_ON_ONCE (1 );return 0 ; } -if (cluster_list_empty (&si->free_clusters)) +/* +* Swapfile is not block device or not using clusters so unable to +* allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info)return 0 ; - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); +again: +/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ; + +/* +* If we failed to allocate or if swapoff is waiting for us (due to lock +* being dropped for discard above), return immediately. +*/ +if (!ret || !(si->flags &SWP_WRITEOK )) +return 0 ; -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); +if (__scan_swap_map_ssd_cluster_conflict (si, offset, order)) + goto again; + + ci =lock_cluster (si, offset); +memset (si->swap_map + offset,SWAP_HAS_CACHE , nr_pages); +add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); +swap_range_alloc (si, offset, nr_pages); + *slot =swp_entry (si->type, offset);return 1 ; }@@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int node;/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); +if (size >1 ) { + n_ret =swap_alloc_large (si, swp_entries, size); }else n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , n_goal, swp_entries);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (p->bdev &&bdev_nonrot (p->bdev)) {int cpu;unsigned long ci, nr_cluster; +int nr_order; +int i; p->flags |=SWP_SOLIDSTATE ; p->cluster_next_cpu = alloc_percpu(unsigned int);@@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)for (ci =0 ; ci < nr_cluster; ci++)spin_lock_init (&((cluster_info + ci)->lock)); - p->cpu_next = alloc_percpu(unsigned int); + nr_order =IS_ENABLED (CONFIG_THP_SWAP ) ?PMD_ORDER +1 :1 ; + p->cpu_next =__alloc_percpu (sizeof (unsigned int ) * nr_order, + __alignof__(unsigned int ));if (!p->cpu_next) { error = -ENOMEM ; goto bad_swap_unlock_inode; } -for_each_possible_cpu (cpu) -per_cpu (*p->cpu_next, cpu) =SWAP_NEXT_NULL ; +for_each_possible_cpu (cpu) { +unsigned int *cpu_next =per_cpu_ptr (p->cpu_next, cpu); + +for (i =0 ; i < nr_order; i++) + cpu_next[i] =SWAP_NEXT_NULL ; + } }else {atomic_inc (&nr_rotate_swap); inced_nr_rotate_swap = true;mm/vmscan.c +6
-5@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --
Quoted 2 lines
2.25.1
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 23 lines · Barry Song
Quoted 8 lines · Ryan Roberts
The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.
Quoted 6 lines
Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when
CONFIG_THP_SWAPis enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.Hi Ryan,
We had a problem while enabling THP SWP on arm64, commit
d0637c505f8("arm64: enableTHP_SWAPfor arm64")this means we have to depend on !system_supports_mte(). static inline bool
arch_thp_swp_supported(void)
Quoted 7 lines
{ return !system_supports_mte(); }Do we have the same problem for small-sized THP? If yes, MTE has been widely existing in various ARM64 SoC. Does it mean we should begin to fix the issue now?
Hi Barry,
I'm guessing that the current problem for MTE is that when it saves the tags prior to swap out, it assumes all folios are small (i.e. base page size) and therefore doesn't have the logic to iterate over a large folio, saving the tags for each page?
If that's the issue, then yes we have the same problem for small-sized THP, but this is all safe - arch_thp_swp_supported() will return false and we continue to use that signal to cause the page to be split prior to swap out.
But, yes, it would be nice to fix that! And if I've understood the problem correctly, it doesn't sound like it should be too hard? Is this something you are volunteering for?? :)
Thanks,
Ryan
Quoted 374 lines
Quoted 369 lines
Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.
The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [1, (1 <<
PMD_ORDER)]. This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).The per-order current clusters are maintained per-cpu using the existing infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.
__scan_swap_map_try_ssd_cluster()is introduced to deal with arbitrary orders andscan_swap_map_try_ssd_cluster()is refactored as a wrapper for order-0.As is done for order-0 per-cpu clusters, the scanner now can steal order-0 entries from any per-cpu-per-order reserved cluster. This ensures that when the swap file is getting full, space doesn't get tied up in the per-cpu reserves.
I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run
usememfrom vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean:Mean Performance Improvement vs 4K/baseline
| alloc size | baseline | + this series | | | v6.6-rc4+anonfolio | | |:-----------|--------------------:|--------------------:| | 4K Page | 0.0% | 4.9% | | 64K THP | -44.1% | 10.7% | | 2M THP | 56.0% | 65.9% |So with this change, the regression for 64K swap performance goes away and 4K and 2M swap improves slightly too.
Signed-off-by Ryan Roberts <[email protected]>--- include/linux/swap.h | 10 +-- mm/swapfile.c | 149 +++++++++++++++++++++++++++++++------------ mm/vmscan.c | 10 +-- 3 files changed, 119 insertions(+), 50 deletions(-)include/linux/swap.h +5
-5@@ -295,11 +295,11 @@ struct swap_info_struct {unsigned int __percpu *cluster_next_cpu;/*percpu index for next allocation */ unsigned int __percpu *cpu_next;/* * Likely next allocation offset. We -* assign a cluster to each CPU, so each -* CPU can allocate swap entry from its -* own cluster and swapout sequentially. -* The purpose is to optimize swapout -* throughput. +* assign a cluster per-order to each +* CPU, so each CPU can allocate swap +* entry from its own cluster and +* swapout sequentially. The purpose is +* to optimize swapout throughput. */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev;/* swap device or bdev of swap file */ mm/swapfile.c +108
-41@@ -545,10 +545,12 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx)/* * The cluster corresponding to page_nr will be used. The cluster will be -* removed from free cluster list and its usage counter will be increased. +* removed from free cluster list and its usage counter will be increased by +* count. */ -static void inc_cluster_info_page (struct swap_info_struct *p, -struct swap_cluster_info *cluster_info,unsigned long page_nr) +static void add_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr, +unsigned long count) {unsigned long idx = page_nr /SWAPFILE_CLUSTER ;@@ -557,9 +559,19 @@ static void inc_cluster_info_page(struct swap_info_struct *p,if (cluster_is_free (&cluster_info[idx]))alloc_cluster (p, idx); -VM_BUG_ON (cluster_count (&cluster_info[idx]) >=SWAPFILE_CLUSTER ); +VM_BUG_ON (cluster_count (&cluster_info[idx]) + count >SWAPFILE_CLUSTER );cluster_set_count (&cluster_info[idx], -cluster_count (&cluster_info[idx]) +1 ); +cluster_count (&cluster_info[idx]) + count); +} + +/* +* The cluster corresponding to page_nr will be used. The cluster will be +* removed from free cluster list and its usage counter will be increased. +*/ +static void inc_cluster_info_page (struct swap_info_struct *p, +struct swap_cluster_info *cluster_info,unsigned long page_nr) +{ +add_cluster_info_page (p, cluster_info, page_nr,1 ); }/* @@ -588,8 +600,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,* cluster list. Avoiding such abuse to avoid list corruption. */ static bool -scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, -unsigned long offset) +__scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset,int order) {bool conflict;@@ -601,23 +613,36 @@ scan_swap_map_ssd_cluster_conflict(struct swap_info_struct *si,if (!conflict)return false; - *this_cpu_ptr (si->cpu_next) =SWAP_NEXT_NULL ; +this_cpu_ptr (si->cpu_next)[order] =SWAP_NEXT_NULL ;return true; }/* -* Try to get a swap entry from current cpu's swap entry pool (a cluster). This -* might involve allocating a new cluster for current CPU too. +* It's possible scan_swap_map_slots() uses a free cluster in the middle of free +* cluster list. Avoiding such abuse to avoid list corruption. */ -static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, -unsigned long *offset,unsigned long *scan_base) +static bool +scan_swap_map_ssd_cluster_conflict (struct swap_info_struct *si, +unsigned long offset) +{ +return __scan_swap_map_ssd_cluster_conflict (si, offset,0 ); +} + +/* +* Try to get a swap entry (or size indicated by order) from current cpu's swap +* entry pool (a cluster). This might involve allocating a new cluster for +* current CPU too. +*/ +static bool __scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base,int order) {struct swap_cluster_info *ci; -unsigned int tmp, max; +unsigned int tmp, max, i;unsigned int *cpu_next; +unsigned int nr_pages =1 << order; new_cluster: - cpu_next =this_cpu_ptr (si->cpu_next); + cpu_next = &this_cpu_ptr (si->cpu_next)[order]; tmp = *cpu_next;if (tmp ==SWAP_NEXT_NULL ) {if (!cluster_list_empty (&si->free_clusters)) {@@ -643,10 +668,12 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si,* reserve a new cluster. */ ci =lock_cluster (si, tmp); -if (si->swap_map[tmp]) { -unlock_cluster (ci); - *cpu_next =SWAP_NEXT_NULL ; - goto new_cluster; +for (i =0 ; i < nr_pages; i++) { +if (si->swap_map[tmp + i]) { +unlock_cluster (ci); + *cpu_next =SWAP_NEXT_NULL ; + goto new_cluster; + } }unlock_cluster (ci);@@ -654,12 +681,22 @@ static bool scan_swap_map_try_ssd_cluster(struct swap_info_struct *si, *scan_base = tmp; max =ALIGN_DOWN (tmp,SWAPFILE_CLUSTER ) +SWAPFILE_CLUSTER ; - tmp +=1 ; + tmp += nr_pages; *cpu_next = tmp < max ? tmp :SWAP_NEXT_NULL ;return true; } +/* +* Try to get a swap entry from current cpu's swap entry pool (a cluster). This +* might involve allocating a new cluster for current CPU too. +*/ +static bool scan_swap_map_try_ssd_cluster (struct swap_info_struct *si, +unsigned long *offset,unsigned long *scan_base) +{ +return __scan_swap_map_try_ssd_cluster (si, offset, scan_base,0 ); +} +static void __del_from_avail_list (struct swap_info_struct *p) {int nid;@@ -982,35 +1019,58 @@ static int scan_swap_map_slots(struct swap_info_struct *si,return n_ret; } -static int swap_alloc_cluster (struct swap_info_struct *si,swp_entry_t *slot) +static int swap_alloc_large (struct swap_info_struct *si,swp_entry_t *slot, +unsigned int nr_pages) { -unsigned long idx;struct swap_cluster_info *ci; -unsigned long offset; +unsigned long offset, scan_base; +int order =ilog2 (nr_pages); +bool ret;/* -* Should not even be attempting cluster allocations when huge +* Should not even be attempting large allocations when huge * page swap is disabled. Warn and fail the allocation. */ -if (!IS_ENABLED (CONFIG_THP_SWAP )) { +if (!IS_ENABLED (CONFIG_THP_SWAP ) || + nr_pages <2 || nr_pages >SWAPFILE_CLUSTER || + !is_power_of_2 (nr_pages)) {VM_WARN_ON_ONCE (1 );return 0 ; } -if (cluster_list_empty (&si->free_clusters)) +/* +* Swapfile is not block device or not using clusters so unable to +* allocate large entries. +*/ +if (!(si->flags &SWP_BLKDEV ) || !si->cluster_info)return 0 ; - idx =cluster_list_first (&si->free_clusters); - offset = idx *SWAPFILE_CLUSTER ; - ci =lock_cluster (si, offset); -alloc_cluster (si, idx); -cluster_set_count (ci,SWAPFILE_CLUSTER ); +again: +/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ; + +/* +* If we failed to allocate or if swapoff is waiting for us (due to lock +* being dropped for discard above), return immediately. +*/ +if (!ret || !(si->flags &SWP_WRITEOK )) +return 0 ; -memset (si->swap_map + offset,SWAP_HAS_CACHE ,SWAPFILE_CLUSTER ); +if (__scan_swap_map_ssd_cluster_conflict (si, offset, order)) + goto again; + + ci =lock_cluster (si, offset); +memset (si->swap_map + offset,SWAP_HAS_CACHE , nr_pages); +add_cluster_info_page (si, si->cluster_info, offset, nr_pages);unlock_cluster (ci); -swap_range_alloc (si, offset,SWAPFILE_CLUSTER ); - *slot =swp_entry (si->type, offset); +swap_range_alloc (si, offset, nr_pages); + *slot =swp_entry (si->type, offset);return 1 ; }@@ -1036,7 +1096,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)int node;/* Only single cluster request supported */ -WARN_ON_ONCE (n_goal >1 && size ==SWAPFILE_CLUSTER ); +WARN_ON_ONCE (n_goal >1 && size >1 );spin_lock (&swap_avail_lock);@@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)spin_unlock (&si->lock); goto nextsi; } -if (size ==SWAPFILE_CLUSTER ) { -if (si->flags &SWP_BLKDEV ) - n_ret =swap_alloc_cluster (si, swp_entries); +if (size >1 ) { + n_ret =swap_alloc_large (si, swp_entries, size); }else n_ret =scan_swap_map_slots (si,SWAP_HAS_CACHE , n_goal, swp_entries);spin_unlock (&si->lock); -if (n_ret || size ==SWAPFILE_CLUSTER ) +if (n_ret || size >1 ) goto check_out;cond_resched ();@@ -3041,6 +3100,8 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)if (p->bdev &&bdev_nonrot (p->bdev)) {int cpu;unsigned long ci, nr_cluster; +int nr_order; +int i; p->flags |=SWP_SOLIDSTATE ; p->cluster_next_cpu = alloc_percpu(unsigned int);@@ -3068,13 +3129,19 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)for (ci =0 ; ci < nr_cluster; ci++)spin_lock_init (&((cluster_info + ci)->lock)); - p->cpu_next = alloc_percpu(unsigned int); + nr_order =IS_ENABLED (CONFIG_THP_SWAP ) ?PMD_ORDER +1 :1 ; + p->cpu_next =__alloc_percpu (sizeof (unsigned int ) * nr_order, + __alignof__(unsigned int ));if (!p->cpu_next) { error = -ENOMEM ; goto bad_swap_unlock_inode; } -for_each_possible_cpu (cpu) -per_cpu (*p->cpu_next, cpu) =SWAP_NEXT_NULL ; +for_each_possible_cpu (cpu) { +unsigned int *cpu_next =per_cpu_ptr (p->cpu_next, cpu); + +for (i =0 ; i < nr_order; i++) + cpu_next[i] =SWAP_NEXT_NULL ; + } }else {atomic_inc (&nr_rotate_swap); inced_nr_rotate_swap = true;mm/vmscan.c +6
-5@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Quoted 2 lines
2.25.1
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 3 lines
But, yes, it would be nice to fix that! And if I've understood the problem correctly, it doesn't sound like it should be too hard? Is this something you are volunteering for?? :)
Unfornately right now I haven't a real hardware with MTE which can run the latest kernel. but i have written a RFC, it will be nice to get someone to test it. Let me figure out if we can get someone :-)
[RFC PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
--- arch/arm64/include/asm/pgtable.h | 21 ++++----------------- arch/arm64/mm/mteswap.c | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+), 17 deletions(-)
arch/arm64/include/asm/pgtable.h +4 -17
@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page
@@ -1028,12 +1022,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct page *page);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)
@@ -1049,11 +1039,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */
arch/arm64/mm/mteswap.c +20 -0
@@ -83,3 +83,23 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) { +struct folio *folio =page_folio (page); +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +return mte_save_tags (folio_page (folio, i)); + } +return 0 ; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +mte_restore_tags (entry,folio_page (folio, i)); + } +}
Signature · 1 line
2.25.1
Quoted 2 lines
Thanks,
Ryan
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 7 lines · Barry Song
Quoted 3 lines
But, yes, it would be nice to fix that! And if I've understood the problem correctly, it doesn't sound like it should be too hard? Is this something you are volunteering for?? :)
Unfornately right now I haven't a real hardware with MTE which can run the latest kernel. but i have written a RFC, it will be nice to get someone to test it. Let me figure out if we can get someone :-)
OK, let me know if you find someone. Otherwise I can have a hunt around to see if I can test it.
Quoted 17 lines
[RFC PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
--- arch/arm64/include/asm/pgtable.h | 21 ++++----------------- arch/arm64/mm/mteswap.c | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+), 17 deletions(-)arch/arm64/include/asm/pgtable.h +0
-0
arch/arm64/include/asm/pgtable.h line 52
Quoted 8 lines
arch/arm64/include/asm/pgtable.h +0
-5@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported
IIRC, arm64 was the only arch implementing this, so perhaps it should be ripped out from the core code now?
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +0
-1-/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1028,12 +1022,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE
arch/arm64/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 1026
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +2
-6#define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct page *page);
I think it would be better to modify this API to take a folio explicitly. The caller already has the folio.
Quoted 23 lines
arch/arm64/include/asm/pgtable.h +2
-5#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1049,11 +1039,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +0
-0@@ -83,3 +83,23 @@ void mte_invalidate_tags_area(int type) }
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 93
Quoted 10 lines
arch/arm64/mm/mteswap.c +8
-0xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) { +struct folio *folio =page_folio (page); +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +return mte_save_tags (folio_page (folio, i));
This will return after saving the first page of the folio! You will need to add each page in a loop, and if you get an error at any point, you will need to remove the pages that you already added successfully, by calling arch_swap_invalidate_page() as far as I can see. Steven can you confirm?
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 103
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+ } +return 0 ; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +mte_restore_tags (entry,folio_page (folio, i));
swap-in currently doesn't support large folios - everything is a single page folio. So this isn't technically needed. But from the API POV, it seems reasonable to make this change - except your implementation is broken. You are currently setting every page in the folio to use the same tags as the first page. You need to increment the swap entry for each page.
Thanks,
Ryan
Quoted 2 lines
arch/arm64/mm/mteswap.c +2
-0+ } +}
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 91 lines · Ryan Roberts
Quoted 7 lines · Barry Song
Quoted 3 lines
But, yes, it would be nice to fix that! And if I've understood the problem correctly, it doesn't sound like it should be too hard? Is this something you are volunteering for?? :)
Unfornately right now I haven't a real hardware with MTE which can run the latest kernel. but i have written a RFC, it will be nice to get someone to test it. Let me figure out if we can get someone :-)
OK, let me know if you find someone. Otherwise I can have a hunt around to see if I can test it.
Quoted 17 lines
[RFC PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
--- arch/arm64/include/asm/pgtable.h | 21 ++++----------------- arch/arm64/mm/mteswap.c | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+), 17 deletions(-)arch/arm64/include/asm/pgtable.h +0
-0arch/arm64/include/asm/pgtable.h line 52
Quoted 8 lines
arch/arm64/include/asm/pgtable.h +0
-5@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supportedIIRC, arm64 was the only arch implementing this, so perhaps it should be ripped out from the core code now?
Quoted 6 lines
arch/arm64/include/asm/pgtable.h +0
-1-/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1028,12 +1022,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE arch/arm64/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 1026
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +2
-6#define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct page *page);I think it would be better to modify this API to take a folio explicitly. The caller already has the folio.
Quoted 30 lines
arch/arm64/include/asm/pgtable.h +2
-5#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1049,11 +1039,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +5
-0@@ -83,3 +83,23 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) { +struct folio *folio =page_folio (page);
Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 93
Quoted 3 lines
arch/arm64/mm/mteswap.c +3
-0+long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +return mte_save_tags (folio_page (folio, i));This will return after saving the first page of the folio! You will need to add each page in a loop, and if you get an error at any point, you will need to remove the pages that you already added successfully, by calling
arch_swap_invalidate_page()as far as I can see. Steven can you confirm?
Yes that's right. mte_save_tags() needs to allocate memory so can fail and if failing then arch_prepare_to_swap() would need to put things back how they were with calls to mte_invalidate_tags() (although I think you'd actually want to refactor to create a function which takes a struct page *).
Steve
Quoted 24 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 103
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+ } +return 0 ; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +mte_restore_tags (entry,folio_page (folio, i));swap-in currently doesn't support large folios - everything is a single page folio. So this isn't technically needed. But from the API POV, it seems reasonable to make this change - except your implementation is broken. You are currently setting every page in the folio to use the same tags as the first page. You need to increment the swap entry for each page.
Thanks,
RyanQuoted 2 lines
arch/arm64/mm/mteswap.c +2
-0+ } +}
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in arch_restore_swap() as we are only swapping in one page at this stage.
[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves arch_prepare_to_swap() to take folio rather than page, as we support THP swap-out as a whole. And this patch also drops arch_thp_swp_supported() as ARM64 MTE is the only one who needs it.
--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)
arch/arm64/include/asm/pgtable.h +4 -17
@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page
@@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)
@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */
arch/arm64/mm/mteswap.c +44 -0
53 lines not on this page · raw message
include/linux/huge_mm.h +0 -12
@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */
include/linux/pgtable.h +1 -1
@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }
mm/page_io.c +1 -1
@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);
mm/swap_slots.c +1 -1
@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Signature · 1 line
2.34.1
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 125 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +30
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i) +__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large +*/
Does it make sense to keep arch_swap_restore taking a folio? I'm not sure I understand why the change was made in the first place. It just seems odd to have a function taking a struct folio but making the assumption that it's actually only a single page (and having to use entry to figure out which page).
It seems particularly broken in the case of unuse_pte() which calls page_folio() to get the folio in the first place.
Other than that it looks correct to me.
Thanks,
Steve
Quoted 64 lines
arch/arm64/mm/mteswap.c +4
-0+struct page *page =folio_file_page (folio,swp_offset (entry)); +mte_restore_tags (entry, page); + } +}include/linux/huge_mm.h +0
-12@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +1
-1@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 135 lines · Steven Price
Quoted 133 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +38
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i) +__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be
Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 2 lines
arch/arm64/mm/mteswap.c +2
-0+* large +*/ Does it make sense to keep
arch_swap_restoretaking a folio? I'm not sure I understand why the change was made in the first place. It just seems odd to have a function taking a struct folio but making the assumption that it's actually only a single page (and having to use entry to figure out which page).
Steve, let me give an example. in case we have a large anon folios with 16 pages.
while reclaiming, we do add_to_swap(), this folio is added to swapcache as a whole; then we unmap the folio; in the last step, we try to release the folio.
we have a good chance some processes might access the virtual address after the folio is unmapped but before the folio is finally released. thus, do_swap_page() will find the large folio in swapcache, there is no I/O needed.
Let's assume processes read the 3rd page of the unmapped folio, in do_swap_page(), the code is like,
vm_fault_t do_swap_page(struct vm_fault *vmf)
{
swp_entry_t entry;
...
entry = pte_to_swp_entry(vmf->orig_pte);
folio = swap_cache_get_folio(entry, vma, vmf->address);
if (folio)
page = folio_file_page(folio, swp_offset(entry));
arch_swap_restore(entry, folio);
}entry points to the 3rd page, but folio points to the head page. so we can't use the entry parameter to restore the whole folio in arch_swap_restore()
then we have two choices in arch_swap_restore()
1. we get the 1st page's swap entry and restore all 16 tags in this large folio.
2. we restore the 3rd tag only by getting the right page in the folio
if we choose 1, in all 16 page faults of do_swap_page for the 16 unmapped PTEs, we will restore 16*16=256 tags. One pte will have one page fault since we don't restore 16 PTEs in do_swap_page().
if we choose 2, in all 16 pages fault of do_swap_page for the 16 unmapped PTEs, we will only restore 16 *1=16 tags.
Quoted 68 lines
It seems particularly broken in the case of
unuse_pte()which callspage_folio()to get the folio in the first place.Other than that it looks correct to me.
Thanks,
Steve
Quoted 58 lines
arch/arm64/mm/mteswap.c +4
-0+struct page *page =folio_file_page (folio,swp_offset (entry)); +mte_restore_tags (entry, page); + } +}include/linux/huge_mm.h +0
-12@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +0
-0@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;
Quoted 7 lines
mm/swap_slots.c swp_entry_t folio_alloc_swap(struct folio *folio) · line 313
Quoted 6 lines
mm/swap_slots.c +1
-1if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 172 lines · Barry Song
Quoted 135 lines · Steven Price
Quoted 133 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +38
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i) +__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 2 lines
arch/arm64/mm/mteswap.c +2
-0+* large +*/ Does it make sense to keep
arch_swap_restoretaking a folio? I'm not sure I understand why the change was made in the first place. It just seems odd to have a function taking a struct folio but making the assumption that it's actually only a single page (and having to use entry to figure out which page).Steve, let me give an example. in case we have a large anon folios with 16 pages.
while reclaiming, we do
add_to_swap(), this folio is added to swapcache as a whole; then we unmap the folio; in the last step, we try to release the folio.we have a good chance some processes might access the virtual address after the folio is unmapped but before the folio is finally released. thus,
do_swap_page()will find the large folio in swapcache, there is no I/O needed.Let's assume processes read the 3rd page of the unmapped folio, in
do_swap_page(), the code is like,
vm_fault_tdo_swap_page(struct vm_fault *vmf){ swp_entry_t entry; ... entry = pte_to_swp_entry(vmf->orig_pte); folio = swap_cache_get_folio(entry, vma, vmf->address); if (folio) page = folio_file_page(folio, swp_offset(entry)); arch_swap_restore(entry, folio); }
Quoted 3 lines
entry points to the 3rd page, but folio points to the head page. so we can't use the entry parameter to restore the whole folio in
arch_swap_restore()
Sorry, I don't think I explained myself very clearly. My issue was that with your patch (and currently) we have the situation where arch_swap_restore() can only restore a single page. But the function takes a "struct folio *" argument.
Current mainline assumes that the folio is a single page, and with your patch we now have a big comment explaining what's going on (bonus points for that!) and pick out the correct page from the folio. What I'm puzzled by is why the change was made in the first place to pass a "struct folio *" - if we passed a "struct page *" that:
a) It would be clear that the current API only allows a single page at a time.
b) The correct page could be passed by the caller rather than arch_swap_restore() having to obtain the offset into the folio.
Quoted 10 lines
then we have two choices in
arch_swap_restore()
1. we get the 1st page's swap entry and restore all 16 tags in this large folio.
2. we restore the 3rd tag only by getting the right page in the folioif we choose 1, in all 16 page faults of
do_swap_pagefor the 16 unmapped PTEs, we will restore 16*16=256 tags. One pte will have one page fault since we don't restore 16 PTEs indo_swap_page().if we choose 2, in all 16 pages fault of
do_swap_pagefor the 16 unmapped PTEs, we will only restore 16 *1=16 tags.
So if we choose option 1 then we're changing the API of arch_swap_restore() to actually restore the entire folio and it makes sense to pass a "struct folio *" - and I'm happy with that. But AFAICT that's not what your patch currently implements as it appears to be doing option 2.
I'm quite happy to believe that the overhead of option 2 is small and that might be the right solution, but at the moment we've got an API which implies arch_swap_restore() should be operating on an entire folio.
Note that I don't have any particularly strong views on this - I've not been following the folio work very closely, but I personally find it confusing when a function takes a "struct folio *" but then operates on only one page of it.
Steve
Quoted 78 lines
Quoted 68 lines
It seems particularly broken in the case of
unuse_pte()which callspage_folio()to get the folio in the first place.Other than that it looks correct to me.
Thanks,
Steve
Quoted 58 lines
arch/arm64/mm/mteswap.c +4
-0+struct page *page =folio_file_page (folio,swp_offset (entry)); +mte_restore_tags (entry, page); + } +}include/linux/huge_mm.h +0
-12@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +0
-0@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;Quoted 7 lines
mm/swap_slots.c swp_entry_t folio_alloc_swap(struct folio *folio) · line 313
Quoted 6 lines
mm/swap_slots.c +1
-1if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Barry SongQuoted 99 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +6
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 10 lines
arch/arm64/mm/mteswap.c +8
-0xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio);
nit: there should be a clear line between variable declarations and logic.
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 109
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i)
If i is initially > 0, this will fail to invalidate page 0. If i is initially 0 then it will wrap and run ~forever. I think you meant while (i--)?
Quoted 6 lines
arch/arm64/mm/mteswap.c +6
-0+__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large +*/
So the currently defined API allows a large folio to be passed but the caller is supposed to find the single correct page using the swap entry? That feels quite nasty to me. And that's not what the old version of the function was doing; it always assumed that the folio was small and passed the first page (which also doesn't feel 'nice'). If the old version was wrong, I suggest a separate commit to fix that. If the old version is correct, then I guess this version is wrong.
Thanks,
Ryan
Quoted 64 lines
arch/arm64/mm/mteswap.c +4
-0+struct page *page =folio_file_page (folio,swp_offset (entry)); +mte_restore_tags (entry, page); + } +}include/linux/huge_mm.h +0
-12@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +1
-1@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 105 lines · Ryan Roberts
Quoted 103 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +8
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio)
Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 6 lines
arch/arm64/mm/mteswap.c +6
-0+{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio);nit: there should be a clear line between variable declarations and logic.
right.
Quoted 14 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 109
Quoted 11 lines
arch/arm64/mm/mteswap.c +10
-0+for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i)If i is initially > 0, this will fail to invalidate page 0. If i is initially 0 then it will wrap and run ~forever. I think you meant
while (i--)?
nop. if i=0 and we goto out, that means the page0 has failed to save tags, there is nothing to revert. if i=3 and we goto out, that means 0,1,2 have saved, we restore 0,1,2 and we don't restore 3.
Quoted 16 lines
Quoted 16 lines
arch/arm64/mm/mteswap.c +15
-0+__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large
Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 1 line
arch/arm64/mm/mteswap.c +1
-0+*/ So the currently defined API allows a large folio to be passed but the caller is supposed to find the single correct page using the swap entry? That feels quite nasty to me. And that's not what the old version of the function was doing; it always assumed that the folio was small and passed the first page (which also doesn't feel 'nice'). If the old version was wrong, I suggest a separate commit to fix that. If the old version is correct, then I guess this version is wrong.
the original version(mainline) is wrong but it works as once we find the SoCs support MTE, we will split large folios into small pages. so only small pages will be added into swapcache successfully.
but now we want to swap out large folios even on SoCs with MTE as a whole, we don't split, so this breaks the assumption do_swap_page() will always get small pages.
Quoted 62 lines
Thanks,
RyanQuoted 58 lines
arch/arm64/mm/mteswap.c +4
-0+struct page *page =folio_file_page (folio,swp_offset (entry)); +mte_restore_tags (entry, page); + } +}include/linux/huge_mm.h +0
-12@@ -400,16 +400,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -897,7 +897,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +0
-0@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;
Quoted 7 lines
mm/swap_slots.c swp_entry_t folio_alloc_swap(struct folio *folio) · line 313
Quoted 6 lines
mm/swap_slots.c +1
-1if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 133 lines · Barry Song
Quoted 105 lines · Ryan Roberts
Quoted 103 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +8
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio)Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 6 lines
arch/arm64/mm/mteswap.c +6
-0+{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio);nit: there should be a clear line between variable declarations and logic.
right.
Quoted 14 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 109
Quoted 11 lines
arch/arm64/mm/mteswap.c +10
-0+for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i)If i is initially > 0, this will fail to invalidate page 0. If i is initially 0 then it will wrap and run ~forever. I think you meant
while (i--)?
Quoted 3 lines
nop. if i=0 and we goto out, that means the page0 has failed to save tags, there is nothing to revert. if i=3 and we goto out, that means 0,1,2 have saved, we restore 0,1,2 and we don't restore 3.
I am terribly sorry for my previous noise. You are right, Ryan. i actually meant i--.
Quoted 26 lines
Quoted 17 lines
Quoted 17 lines
arch/arm64/mm/mteswap.c +15
-0+__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 1 line
arch/arm64/mm/mteswap.c +1
-0+*/ So the currently defined API allows a large folio to be passed but the caller is supposed to find the single correct page using the swap entry? That feels quite nasty to me. And that's not what the old version of the function was doing; it always assumed that the folio was small and passed the first page (which also doesn't feel 'nice'). If the old version was wrong, I suggest a separate commit to fix that. If the old version is correct, then I guess this version is wrong.
Quoted 7 lines
the original version(mainline) is wrong but it works as once we find the SoCs support MTE, we will split large folios into small pages. so only small pages will be added into swapcache successfully.
but now we want to swap out large folios even on SoCs with MTE as a whole, we don't split, so this breaks the assumption
do_swap_page()will always get small pages.
let me clarify this more. The current mainline assumes arch_swap_restore() always
get a folio with only one page. this is true as we split large folios if we find SoCs
have MTE. but since we are dropping the split now, that means a large folio can be
gotten by do_swap_page(). we have a chance that try_to_unmap_one() has been done but folio is not put. so PTEs will have swap entry but folio is still there, and do_swap_page()
to hit cache directly and the folio won't be released.
but after getting the large folio in do_swap_page, it still only takes one basepage particularly
for the faulted PTE and maps this 4KB PTE only. so it uses the faulted swap_entry and
the folio as parameters to call arch_swap_restore() which can be something like:
do_swap_page()
{
arch_swap_restore(the swap entry for the faulted 4KB PTE, large folio);
}Quoted 4 lines
Quoted 4 lines
Thanks,
Ryan
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 134 lines · Barry Song
Quoted 133 lines · Barry Song
Quoted 105 lines · Ryan Roberts
Quoted 103 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +8
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio)Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 6 lines
arch/arm64/mm/mteswap.c +6
-0+{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio);nit: there should be a clear line between variable declarations and logic.
right.
Quoted 14 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 109
Quoted 11 lines
arch/arm64/mm/mteswap.c +10
-0+for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i)If i is initially > 0, this will fail to invalidate page 0. If i is initially 0 then it will wrap and run ~forever. I think you meant
while (i--)?
Quoted 6 lines
Quoted 3 lines
nop. if i=0 and we goto out, that means the page0 has failed to save tags, there is nothing to revert. if i=3 and we goto out, that means 0,1,2 have saved, we restore 0,1,2 and we don't restore 3.
I am terribly sorry for my previous noise. You are right, Ryan. i actually meant i--.
No problem - it saves me from writing a long response explaining why --i is wrong, at least!
Quoted 52 lines
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
arch/arm64/mm/mteswap.c +15
-0+__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 1 line
arch/arm64/mm/mteswap.c +1
-0+*/ So the currently defined API allows a large folio to be passed but the caller is supposed to find the single correct page using the swap entry? That feels quite nasty to me. And that's not what the old version of the function was doing; it always assumed that the folio was small and passed the first page (which also doesn't feel 'nice'). If the old version was wrong, I suggest a separate commit to fix that. If the old version is correct, then I guess this version is wrong.
Quoted 7 lines
the original version(mainline) is wrong but it works as once we find the SoCs support MTE, we will split large folios into small pages. so only small pages will be added into swapcache successfully.
but now we want to swap out large folios even on SoCs with MTE as a whole, we don't split, so this breaks the assumption
do_swap_page()will always get small pages.let me clarify this more. The current mainline assumes
arch_swap_restore()always
get a folio with only one page. this is true as we split large folios if we find SoCs
have MTE. but since we are dropping the split now, that means a large folio can be
gotten bydo_swap_page(). we have a chance thattry_to_unmap_one()has been done but folio is not put. so PTEs will have swap entry but folio is still there, anddo_swap_page()
to hit cache directly and the folio won't be released.but after getting the large folio in
do_swap_page, it still only takes one basepage particularly
for the faulted PTE and maps this 4KB PTE only. so it uses the faultedswap_entryand
the folio as parameters to callarch_swap_restore()which can be something like:
Quoted 4 lines
do_swap_page(){ arch_swap_restore(the swap entry for the faulted 4KB PTE, large folio); }
OK, I understand what's going on, but it seems like a bad API decision. I think Steve is saying the same thing; If its only intended to operate on a single page, it would be much clearer to pass the actual page rather than the folio; i.e. leave the complexity of figuring out the target page to the caller, which understands all this.
As a side note, if the folio is still in the cache, doesn't that imply that the tags haven't been torn down yet? So perhaps you can avoid even making the call in this case?
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Thanks,
RyanThanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 203 lines · Ryan Roberts
Quoted 134 lines · Barry Song
Quoted 133 lines · Barry Song
Quoted 105 lines · Ryan Roberts
Quoted 103 lines · Barry Song
Quoted 7 lines
Yes that's right.
mte_save_tags()needs to allocate memory so can fail and if failing thenarch_prepare_to_swap()would need to put things back how they were with calls tomte_invalidate_tags()(although I think you'd actually want to refactor to create a function which takes a struct page *).Steve
Thanks, Steve. combining all comments from You and Ryan, I made a v2. One tricky thing is that we are restoring one page rather than folio in
arch_restore_swap()as we are only swapping in one page at this stage.[RFC v2 PATCH] arm64: mm: swap: save and restore mte tags for large folios
This patch makes MTE tags saving and restoring support large folios, then we don't need to split them into base pages for swapping on ARM64 SoCs with MTE.
This patch moves
arch_prepare_to_swap()to take folio rather than page, as we support THP swap-out as a whole. And this patch also dropsarch_thp_swp_supported()as ARM64 MTE is the only one who needs it.Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------ arch/arm64/mm/mteswap.c | 44 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 51 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +4
-17@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1036,12 +1030,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1057,11 +1047,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +8
-0@@ -68,6 +68,12 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );@@ -83,3 +89,41 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio)Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 6 lines
arch/arm64/mm/mteswap.c +6
-0+{ +int err; +long i; + +if (system_supports_mte ()) { +long nr =folio_nr_pages (folio);nit: there should be a clear line between variable declarations and logic.
right.
Quoted 14 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 109
Quoted 11 lines
arch/arm64/mm/mteswap.c +10
-0+for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ; + +out: +while (--i)If i is initially > 0, this will fail to invalidate page 0. If i is initially 0 then it will wrap and run ~forever. I think you meant
while (i--)?Quoted 6 lines
Quoted 3 lines
nop. if i=0 and we goto out, that means the page0 has failed to save tags, there is nothing to revert. if i=3 and we goto out, that means 0,1,2 have saved, we restore 0,1,2 and we don't restore 3.
I am terribly sorry for my previous noise. You are right, Ryan. i actually meant i--.
No problem - it saves me from writing a long response explaining why --i is wrong, at least!
Quoted 52 lines
Quoted 27 lines
Quoted 18 lines
Quoted 18 lines
arch/arm64/mm/mteswap.c +15
-0+__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +/* +* We don't support large folios swap in as whole yet, but +* we can hit a large folio which is still in swapcache +* after those related processes' PTEs have been unmapped +* but before the swapcache folio is dropped, in this case, +* we need to find the exact page which "entry" is mapping +* to. If we are not hitting swapcache, this folio won't be +* large Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 125
Quoted 1 line
arch/arm64/mm/mteswap.c +1
-0+*/ So the currently defined API allows a large folio to be passed but the caller is supposed to find the single correct page using the swap entry? That feels quite nasty to me. And that's not what the old version of the function was doing; it always assumed that the folio was small and passed the first page (which also doesn't feel 'nice'). If the old version was wrong, I suggest a separate commit to fix that. If the old version is correct, then I guess this version is wrong.
Quoted 7 lines
the original version(mainline) is wrong but it works as once we find the SoCs support MTE, we will split large folios into small pages. so only small pages will be added into swapcache successfully.
but now we want to swap out large folios even on SoCs with MTE as a whole, we don't split, so this breaks the assumption
do_swap_page()will always get small pages.let me clarify this more. The current mainline assumes
arch_swap_restore()always
get a folio with only one page. this is true as we split large folios if we find SoCs
have MTE. but since we are dropping the split now, that means a large folio can be
gotten bydo_swap_page(). we have a chance thattry_to_unmap_one()has been done but folio is not put. so PTEs will have swap entry but folio is still there, anddo_swap_page()
to hit cache directly and the folio won't be released.but after getting the large folio in
do_swap_page, it still only takes one basepage particularly
for the faulted PTE and maps this 4KB PTE only. so it uses the faultedswap_entryand
the folio as parameters to callarch_swap_restore()which can be something like:Quoted 4 lines
do_swap_page(){ arch_swap_restore(the swap entry for the faulted 4KB PTE, large folio); }
Quoted 5 lines
OK, I understand what's going on, but it seems like a bad API decision. I think Steve is saying the same thing; If its only intended to operate on a single page, it would be much clearer to pass the actual page rather than the folio; i.e. leave the complexity of figuring out the target page to the caller, which understands all this.
right.
Quoted 4 lines
As a side note, if the folio is still in the cache, doesn't that imply that the tags haven't been torn down yet? So perhaps you can avoid even making the call in this case?
right. but it is practically very hard as arch_swap_restore() is always called unconditionally.
it is hard to find a decent condition before calling arch_swap_restore(). That is why we
actually have been doing redundant arch_swap_restore() lots of times right now.
For example, A forks B,C,D,E,F,G. now A,B,C,D,E,F,G will share one page before CoW. After
the page is swapped out, if B is the first process to swap in, B will add the page to swapcache,
and restore MTE. After that, A, C, D, E, F,G will directly hit the page swapped in by B, now they
restore MTE again. so the MTE is restored 7 times but actually only B needs to do it.
so it seems we can put a condition to only let B do restore. But it won't work because we can't
guarrent B is the first process who will do PTE mapping. A, C, D, E, F, G can map PTEs earlier than B
even if B is the one who did the I/O swapin. swapin/add swapcache and PTE mapping are
not done atomically. PTE mapping needs to take PTL. so After B has done swapin, A, C,E,F,G
can still begin to use the page earlier than B. So it turns out anyone who first maps the page
should restore MTE, but the question is that: How could A,B,C,D,E,F,G know if it is the first
one mapping the page to PTE?
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Thanks,
RyanThanks
Barry
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Ryan RobertsQuoted 4 lines
Quoted 4 lines
#define
__HAVE_ARCH_PREPARE_TO_SWAParch/arm64/include/asm/pgtable.h +0
-3-static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ())
Quoted 8 lines
arch/arm64/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 1026
Quoted 5 lines
arch/arm64/include/asm/pgtable.h +2
-3-return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct page *page);I think it would be better to modify this API to take a folio explicitly. The caller already has the folio.
agree. that was actually what i thought I should change while making this rfc, though i didn't do it.
Quoted 4 lines
Quoted 4 lines
arch/arm64/mm/mteswap.c +4
-0+int arch_prepare_to_swap (struct page *page) +{ +if (system_supports_mte ()) { +struct folio *folio =page_folio (page);
Quoted 8 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 93
Quoted 3 lines
arch/arm64/mm/mteswap.c +3
-0+long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +return mte_save_tags (folio_page (folio, i));This will return after saving the first page of the folio! You will need to add each page in a loop, and if you get an error at any point, you will need to remove the pages that you already added successfully, by calling
arch_swap_invalidate_page()as far as I can see. Steven can you confirm?
right. oops...
Quoted 17 lines
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 103
Quoted 11 lines
arch/arm64/mm/mteswap.c +10
-0+ } +return 0 ; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) { +long i, nr =folio_nr_pages (folio); +for (i =0 ; i < nr; i++) +mte_restore_tags (entry,folio_page (folio, i));swap-in currently doesn't support large folios - everything is a single page folio. So this isn't technically needed. But from the API POV, it seems reasonable to make this change - except your implementation is broken. You are currently setting every page in the folio to use the same tags as the first page. You need to increment the swap entry for each page.
one case is that we have a chance to "swapin" a folio which is still in swapcache and hasn't been dropped yet. i mean the process's ptes have been swap entry, but the large folio is still in swapcache. in this case, we will hit the swapcache while swapping in, thus we are handling a large folio. in this case, it seems we are restoring tags multiple times? i mean, if large folio has 16 basepages, for each page fault of each base page, we are restoring a large folio, then for 16 page faults, we are duplicating the restore.
any thought to handle this situation? should we move arch_swap_restore() to take page rather than folio since swapin only supports basepage at this moment.
Quoted 2 lines
Thanks,
Ryan
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Ryan Roberts+Chris, Suren and Chuanhua
Hi Ryan,
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1055
Quoted 7 lines
mm/swapfile.c +7
-0+/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ;
nobody is using this scan_base afterwards. it seems a bit weird to pass a pointer.
Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --
Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and split_folio. this might cause applications get wrong data 0 while swapping-in.
in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,
static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end, struct mm_walk *walk)
{
/*
* Creating a THP page is expensive so split it only if we
* are sure it's worth. Split it if we are only owner.
*/
if (folio_test_large(folio)) {
int err;
if (folio_estimated_sharers(folio) != 1)
break;
if (pageout_anon_only_filter && !folio_test_anon(folio))
break;
if (!folio_trylock(folio))
break;
folio_get(folio);
arch_leave_lazy_mmu_mode();
pte_unmap_unlock(start_pte, ptl);
start_pte = NULL;
err = split_folio(folio);
folio_unlock(folio);
folio_put(folio);
if (err)
break;
start_pte = pte =
pte_offset_map_lock(mm, pmd, addr, &ptl);
if (!start_pte)
break;
arch_enter_lazy_mmu_mode();
pte--;
addr -= PAGE_SIZE;
continue;
}
return 0;
}if T3 and T4 swap-in same page, and they both do swap_read_folio(). the first one of T3 and T4 who gets PTL will set pte, and the 2nd one will check pte_same() and find pte has been changed by another thread, thus goto out_nomap in do_swap_page.vm_fault_t do_swap_page(struct vm_fault *vmf)
{
if (!folio) {
if (data_race(si->flags & SWP_SYNCHRONOUS_IO) &&
__swap_count(entry) == 1) {
/* skip swapcache */
folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0,
vma, vmf->address, false);
page = &folio->page;
if (folio) {
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
/* To provide entry to swap_read_folio() */
folio->swap = entry;
swap_read_folio(folio, true, NULL);
folio->private = NULL;
}
} else {
}
/*
* Back out if somebody else already faulted in this pte.
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
goto out_nomap;
swap_free(entry);
pte = mk_pte(page, vma->vm_page_prot);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
return ret;
}while T1 and T2 is working in parallel, T2 will split folio. this can run into race with T1's reclamation without splitting. T2 will split a large folio into a couple of normal pages and reclaim them.
If T3 finishes swap_read_folio and gets PTL earlier than T4, it calls set_pte and swap_free. this will cause zRAM to free the slot. then t4 will get zero data in swap_read_folio() as the below zRAM code will fill zero for freed slots,
static int zram_read_from_zspool(struct zram *zram, struct page *page,
u32 index)
{
...
handle = zram_get_handle(zram, index);
if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) {
unsigned long value;
void *mem;
value = handle ? zram_get_element(zram, index) : 0;
mem = kmap_local_page(page);
zram_fill_page(mem, PAGE_SIZE, value);
kunmap_local(mem);
return 0;
}
}usually, after t3 frees swap and does set_pte, t4's pte_same becomes false, it won't set pte again. So filling zero data into freed slot by zRAM driver is not a problem at all. but the race is that T1 and T2 might do set swap to ptes twice as t1 doesn't split but t2 splits (splitted normal folios are also added into reclaim_list), thus, the corrupted zero data will get a chance to be set into PTE by t4 as t4 reads the new PTE which is set secondly and has the same swap entry as its orig_pte after T3 has swapped-in and free the swap entry.
we have worked around this problem by preventing T4 from splitting large folios and letting it goto skip the large folios entirely in MADV PAGEOUT once we detect a concurrent reclamation for this large folio.
so my understanding is changing vmscan isn't sufficient to support large folio swap-out without splitting. we have to adjust madvise as well. we will have a fix for this problem in
[PATCH RFC 6/6] mm: madvise: don't split mTHP for MADV_PAGEOUT
https://lore.kernel.org/linux-mm/[email protected]/
But i feel this patch should be a part of your swap-out patchset rather than the swap-in series of Chuanhua and me :-)
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 171 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1055
Quoted 7 lines
mm/swapfile.c +7
-0+/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise
MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end, structmm_walk*walk)
{/* * Creating a THP page is expensive so split it only if we * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { int err; if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; if (!folio_trylock(folio)) break; folio_get(folio); arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); start_pte = NULL; err = split_folio(folio); folio_unlock(folio); folio_put(folio); if (err) break; start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!start_pte) break; arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; } return 0; }if T3 and T4 swap-in same page, and they both do
swap_read_folio(). the first one of T3 and T4 who gets PTL will set pte, and the 2nd one will checkpte_same()and find pte has been changed by another thread, thus gotoout_nomapindo_swap_page.vm_fault_tdo_swap_page(struct vm_fault *vmf){ if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); page = &folio->page; if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); /* To provide entry to swap_read_folio() */ folio->swap = entry; swap_read_folio(folio, true, NULL); folio->private = NULL; } } else { } /* * Back out if somebody else already faulted in this pte. */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; swap_free(entry); pte = mk_pte(page, vma->vm_page_prot); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); return ret; }while T1 and T2 is working in parallel, T2 will split folio. this can run into race with T1's reclamation without splitting. T2 will split a large folio into a couple of normal pages and reclaim them.
If T3 finishes
swap_read_folioand gets PTL earlier than T4, it callsset_pteandswap_free. this will cause zRAM to free the slot. then t4 will get zero data inswap_read_folio()as the below zRAM code will fill zero for freed slots,static int zram_read_from_zspool(struct zram *zram, struct page *page,
u32 index) { ... handle = zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { unsigned long value; void *mem; value = handle ? zram_get_element(zram, index) : 0; mem = kmap_local_page(page); zram_fill_page(mem, PAGE_SIZE, value); kunmap_local(mem); return 0; } }usually, after t3 frees swap and does
set_pte, t4'spte_samebecomes false, it won't set pte again. So filling zero data into freed slot by zRAM driver is not a problem at all. but the race is that T1 and T2 might do set swap to ptes twice as t1 doesn't split but t2 splits (splitted normal folios are also added intoreclaim_list), thus, the corrupted zero data will get a chance to be set into PTE by t4 as t4 reads the new PTE which is set secondly and has the same swap entry as itsorig_pteafter T3 has swapped-in and free the swap entry.we have worked around this problem by preventing T4 from splitting large folios and letting it goto skip the large folios entirely in MADV PAGEOUT once we detect a concurrent reclamation for this large folio.
Quoted 8 lines
so my understanding is changing vmscan isn't sufficient to support large folio swap-out without splitting. we have to adjust madvise as well. we will have a fix for this problem in
[PATCH RFC 6/6] mm: madvise: don't split mTHP forMADV_PAGEOUT
https://lore.kernel.org/linux-mm/[email protected]/But i feel this patch should be a part of your swap-out patchset rather than the swap-in series of Chuanhua and me :-)
Hi Barry, Chuanhua,
Thanks for the very detailed bug report! I'm going to have to take some time to get my head around the details. But yes, I agree the fix needs to be part of the swap-out series.
Sorry I haven't progressed this series as I had hoped. I've been concentrating on getting the contpte series upstream. I'm hoping I will find some time to move this series along by the tail end of Feb (hoping to get it in shape for v6.10). Hopefully that doesn't cause you any big problems?
Thanks,
Ryan
Quoted 3 lines
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 179 lines · Ryan Roberts
Quoted 177 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
Review
Quoted 7 lines
(quoted diff) +7
-0+ /* + * __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, + * so indicate that we are scanning to synchronise with swapoff. + */ + si->flags += SWP_SCANNING; + ret = __scan_swap_map_try_ssd_cluster(si, &offset, &scan_base, order); + si->flags -= SWP_SCANNING;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise
MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end, structmm_walk*walk)
{/* * Creating a THP page is expensive so split it only if we * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { int err; if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; if (!folio_trylock(folio)) break; folio_get(folio); arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); start_pte = NULL; err = split_folio(folio); folio_unlock(folio); folio_put(folio); if (err) break; start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!start_pte) break; arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; } return 0; }if T3 and T4 swap-in same page, and they both do
swap_read_folio(). the first one of T3 and T4 who gets PTL will set pte, and the 2nd one will checkpte_same()and find pte has been changed by another thread, thus gotoout_nomapindo_swap_page.vm_fault_tdo_swap_page(struct vm_fault *vmf){ if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); page = &folio->page; if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); /* To provide entry to swap_read_folio() */ folio->swap = entry; swap_read_folio(folio, true, NULL); folio->private = NULL; } } else { } /* * Back out if somebody else already faulted in this pte. */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; swap_free(entry); pte = mk_pte(page, vma->vm_page_prot); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); return ret; }while T1 and T2 is working in parallel, T2 will split folio. this can run into race with T1's reclamation without splitting. T2 will split a large folio into a couple of normal pages and reclaim them.
If T3 finishes
swap_read_folioand gets PTL earlier than T4, it callsset_pteandswap_free. this will cause zRAM to free the slot. then t4 will get zero data inswap_read_folio()as the below zRAM code will fill zero for freed slots,static int zram_read_from_zspool(struct zram *zram, struct page *page,
u32 index) { ... handle = zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { unsigned long value; void *mem; value = handle ? zram_get_element(zram, index) : 0; mem = kmap_local_page(page); zram_fill_page(mem, PAGE_SIZE, value); kunmap_local(mem); return 0; } }usually, after t3 frees swap and does
set_pte, t4'spte_samebecomes false, it won't set pte again. So filling zero data into freed slot by zRAM driver is not a problem at all. but the race is that T1 and T2 might do set swap to ptes twice as t1 doesn't split but t2 splits (splitted normal folios are also added intoreclaim_list), thus, the corrupted zero data will get a chance to be set into PTE by t4 as t4 reads the new PTE which is set secondly and has the same swap entry as itsorig_pteafter T3 has swapped-in and free the swap entry.we have worked around this problem by preventing T4 from splitting large folios and letting it goto skip the large folios entirely in MADV PAGEOUT once we detect a concurrent reclamation for this large folio.
so my understanding is changing vmscan isn't sufficient to support large folio swap-out without splitting. we have to adjust madvise as well. we will have a fix for this problem in
[PATCH RFC 6/6] mm: madvise: don't split mTHP forMADV_PAGEOUT
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 9 lines
Quoted 2 lines
But i feel this patch should be a part of your swap-out patchset rather than the swap-in series of Chuanhua and me :-)
Hi Barry, Chuanhua,
Thanks for the very detailed bug report! I'm going to have to take some time to get my head around the details. But yes, I agree the fix needs to be part of the swap-out series.
Hi Ryan,
I am running into some races especially while enabling large folio swap-out and swap-in both. some of them, i am still struggling with the detailed timing how they
are happening.
but the below change can help remove those bugs which cause corrupted data.
index da2aab219c40..ef9cfbc84760 100644
mm/vmscan.c +0 -0
@@ -1953,6 +1953,16 @@ static unsigned int shrink_folio_list(struct
list_head *folio_list,
if (folio_test_pmd_mappable(folio))
flags |= TTU_SPLIT_HUGE_PMD;
+ /*
+ * make try_to_unmap_one hold ptl from the very first
+ * beginning if we are reclaiming a folio with multi-
+ * ptes. otherwise, we may only reclaim a part of the
+ * folio from the middle.
+ * for example, a parallel thread might temporarily
+ * set pte to none for various purposes.
+ */+ else if (folio_test_large(folio))
+ flags |= TTU_SYNC;
try_to_unmap(folio, flags);
if (folio_mapped(folio)) {While we are swapping-out a large folio, it has many ptes, we change those ptes to swap entries in try_to_unmap_one(). "while (page_vma_mapped_walk(&pvmw))" will iterate all ptes within the large folio. but it will only begin to acquire ptl when
it meets a valid pte as below /* xxxxxxx */
static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp)
{
pte_t ptent;
if (pvmw->flags & PVMW_SYNC) {
/* Use the stricter lookup */
pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd,
pvmw->address, &pvmw->ptl);
*ptlp = pvmw->ptl;
return !!pvmw->pte;
}
...
pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd,
pvmw->address, ptlp);
if (!pvmw->pte)
return false;
ptent = ptep_get(pvmw->pte);
if (pvmw->flags & PVMW_MIGRATION) {
if (!is_swap_pte(ptent))
return false;
} else if (is_swap_pte(ptent)) {
swp_entry_t entry;
...
entry = pte_to_swp_entry(ptent);
if (!is_device_private_entry(entry) &&
!is_device_exclusive_entry(entry))
return false;
} else if (!pte_present(ptent)) {
return false;
}
pvmw->ptl = *ptlp;
spin_lock(pvmw->ptl); /* xxxxxxx */
return true;
}for various reasons, for example, break-before-make for clearing access flags etc. pte can be set to none. since page_vma_mapped_walk() doesn't hold ptl from the beginning, it might only begin to set swap entries from the middle of a large folio.
For example, in case a large folio has 16 ptes, and 0,1,2 are somehow zero in the intermediate stage of a break-before-make, ptl will be held from the 3rd pte,
and swap entries will be set from 3rd pte as well. it seems not good as we are trying to swap out a large folio, but we are swapping out a part of them.
I am still struggling with all the timing of races, but using PVMW_SYNC to explicitly ask for ptl from the first pte seems a good thing for large folio regardless of those races. it can avoid try_to_unmap_one reading intermediate pte and further make the wrong decision since reclaiming pte-mapped large folios is atomic with just one pte.
Quoted 4 lines
Sorry I haven't progressed this series as I had hoped. I've been concentrating on getting the contpte series upstream. I'm hoping I will find some time to move this series along by the tail end of Feb (hoping to get it in shape for v6.10). Hopefully that doesn't cause you any big problems?
no worries. Anyway, we are already using your code to run various tests.
Quoted 3 lines
Thanks,
Ryan
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 190 lines · Barry Song
Quoted 179 lines · Ryan Roberts
Quoted 177 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
Review
Quoted 7 lines
(quoted diff) +7
-0+ /* + * __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, + * so indicate that we are scanning to synchronise with swapoff. + */ + si->flags += SWP_SCANNING; + ret = __scan_swap_map_try_ssd_cluster(si, &offset, &scan_base, order); + si->flags -= SWP_SCANNING;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise
MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end, structmm_walk*walk)
{/* * Creating a THP page is expensive so split it only if we * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { int err; if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; if (!folio_trylock(folio)) break; folio_get(folio); arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); start_pte = NULL; err = split_folio(folio); folio_unlock(folio); folio_put(folio); if (err) break; start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!start_pte) break; arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; } return 0; }if T3 and T4 swap-in same page, and they both do
swap_read_folio(). the first one of T3 and T4 who gets PTL will set pte, and the 2nd one will checkpte_same()and find pte has been changed by another thread, thus gotoout_nomapindo_swap_page.vm_fault_tdo_swap_page(struct vm_fault *vmf){ if (!folio) { if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && __swap_count(entry) == 1) { /* skip swapcache */ folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address, false); page = &folio->page; if (folio) { __folio_set_locked(folio); __folio_set_swapbacked(folio); /* To provide entry to swap_read_folio() */ folio->swap = entry; swap_read_folio(folio, true, NULL); folio->private = NULL; } } else { } /* * Back out if somebody else already faulted in this pte. */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; swap_free(entry); pte = mk_pte(page, vma->vm_page_prot); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); return ret; }while T1 and T2 is working in parallel, T2 will split folio. this can run into race with T1's reclamation without splitting. T2 will split a large folio into a couple of normal pages and reclaim them.
If T3 finishes
swap_read_folioand gets PTL earlier than T4, it callsset_pteandswap_free. this will cause zRAM to free the slot. then t4 will get zero data inswap_read_folio()as the below zRAM code will fill zero for freed slots,static int zram_read_from_zspool(struct zram *zram, struct page *page,
u32 index) { ... handle = zram_get_handle(zram, index); if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) { unsigned long value; void *mem; value = handle ? zram_get_element(zram, index) : 0; mem = kmap_local_page(page); zram_fill_page(mem, PAGE_SIZE, value); kunmap_local(mem); return 0; } }usually, after t3 frees swap and does
set_pte, t4'spte_samebecomes false, it won't set pte again. So filling zero data into freed slot by zRAM driver is not a problem at all. but the race is that T1 and T2 might do set swap to ptes twice as t1 doesn't split but t2 splits (splitted normal folios are also added intoreclaim_list), thus, the corrupted zero data will get a chance to be set into PTE by t4 as t4 reads the new PTE which is set secondly and has the same swap entry as itsorig_pteafter T3 has swapped-in and free the swap entry.we have worked around this problem by preventing T4 from splitting large folios and letting it goto skip the large folios entirely in MADV PAGEOUT once we detect a concurrent reclamation for this large folio.
so my understanding is changing vmscan isn't sufficient to support large folio swap-out without splitting. we have to adjust madvise as well. we will have a fix for this problem in
[PATCH RFC 6/6] mm: madvise: don't split mTHP forMADV_PAGEOUT
https://lore.kernel.org/linux-mm/[email protected]/Quoted 9 lines
Quoted 2 lines
But i feel this patch should be a part of your swap-out patchset rather than the swap-in series of Chuanhua and me :-)
Hi Barry, Chuanhua,
Thanks for the very detailed bug report! I'm going to have to take some time to get my head around the details. But yes, I agree the fix needs to be part of the swap-out series.
Quoted 6 lines
Hi Ryan,
I am running into some races especially while enabling large folio swap-out and swap-in both. some of them, i am still struggling with the detailed timing how they
are happening.
but the below change can help remove those bugs which cause corrupted data.
Thanks for the report! I'm out of office this week, but this is top of my todo list starting next week, so hopefully will knock these into shape and repost very soon.
Thanks,
Ryan
Quoted 99 lines
index
da2aab219c40..ef9cfbc84760100644mm/vmscan.c +10
-0@@ -1953,6 +1953,16 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (folio_test_pmd_mappable(folio)) flags |= TTU_SPLIT_HUGE_PMD; +/* +* make try_to_unmap_one hold ptl from the very first +* beginning if we are reclaiming a folio with multi- +* ptes. otherwise, we may only reclaim a part of the +* folio from the middle. +* for example, a parallel thread might temporarily +* set pte to none for various purposes. +*/ + else if (folio_test_large(folio)) + flags |= TTU_SYNC;try_to_unmap (folio ,flags );if (folio_mapped(folio)) {While we are swapping-out a large folio, it has many ptes, we change those ptes to swap entries in
try_to_unmap_one(). "while (page_vma_mapped_walk(&pvmw))" will iterate all ptes within the large folio. but it will only begin to acquire ptl when
it meets a valid pte as below /* xxxxxxx */static bool
map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp){ pte_t ptent; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, &pvmw->ptl); *ptlp = pvmw->ptl; return !!pvmw->pte; } ... pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, ptlp); if (!pvmw->pte) return false; ptent = ptep_get(pvmw->pte); if (pvmw->flags & PVMW_MIGRATION) { if (!is_swap_pte(ptent)) return false; } else if (is_swap_pte(ptent)) { swp_entry_t entry; ... entry = pte_to_swp_entry(ptent); if (!is_device_private_entry(entry) && !is_device_exclusive_entry(entry)) return false; } else if (!pte_present(ptent)) { return false; } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); /* xxxxxxx */ return true; }for various reasons, for example, break-before-make for clearing access flags etc. pte can be set to none. since
page_vma_mapped_walk()doesn't hold ptl from the beginning, it might only begin to set swap entries from the middle of a large folio.For example, in case a large folio has 16 ptes, and 0,1,2 are somehow zero in the intermediate stage of a break-before-make, ptl will be held from the 3rd pte,
and swap entries will be set from 3rd pte as well. it seems not good as we are trying to swap out a large folio, but we are swapping out a part of them.I am still struggling with all the timing of races, but using
PVMW_SYNCto explicitly ask for ptl from the first pte seems a good thing for large folio regardless of those races. it can avoidtry_to_unmap_onereading intermediate pte and further make the wrong decision since reclaiming pte-mapped large folios is atomic with just one pte.Quoted 4 lines
Sorry I haven't progressed this series as I had hoped. I've been concentrating on getting the contpte series upstream. I'm hoping I will find some time to move this series along by the tail end of Feb (hoping to get it in shape for v6.10). Hopefully that doesn't cause you any big problems?
no worries. Anyway, we are already using your code to run various tests.
Quoted 3 lines
Thanks,
RyanThanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Barry SongHi Barry,
Quoted 6 lines · Barry Song
Quoted 5 lines · Ryan Roberts
Quoted 3 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
[...]
Quoted 8 lines
Quoted 1 line
Hi Ryan,
I am running into some races especially while enabling large folio swap-out and swap-in both. some of them, i am still struggling with the detailed timing how they
are happening.
but the below change can help remove those bugs which cause corrupted data.
I'm getting quite confused with all the emails flying around on this topic. Here you were reporting a data corruption bug and your suggested fix below is the one you have now posted at [1]. But in the thread at [1] we concluded that it is not fixing a functional correctness issue, but is just an optimization in some corner cases. So does the corruption issue still manifest? Did you manage to root cause it? Is it a problem with my swap-out series or your swap-in series, or pre-existing?
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
Quoted 99 lines
index
da2aab219c40..ef9cfbc84760100644mm/vmscan.c +10
-0@@ -1953,6 +1953,16 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (folio_test_pmd_mappable(folio)) flags |= TTU_SPLIT_HUGE_PMD; +/* +* make try_to_unmap_one hold ptl from the very first +* beginning if we are reclaiming a folio with multi- +* ptes. otherwise, we may only reclaim a part of the +* folio from the middle. +* for example, a parallel thread might temporarily +* set pte to none for various purposes. +*/ + else if (folio_test_large(folio)) + flags |= TTU_SYNC;try_to_unmap (folio ,flags );if (folio_mapped(folio)) {While we are swapping-out a large folio, it has many ptes, we change those ptes to swap entries in
try_to_unmap_one(). "while (page_vma_mapped_walk(&pvmw))" will iterate all ptes within the large folio. but it will only begin to acquire ptl when
it meets a valid pte as below /* xxxxxxx */static bool
map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp){ pte_t ptent; if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, &pvmw->ptl); *ptlp = pvmw->ptl; return !!pvmw->pte; } ... pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, ptlp); if (!pvmw->pte) return false; ptent = ptep_get(pvmw->pte); if (pvmw->flags & PVMW_MIGRATION) { if (!is_swap_pte(ptent)) return false; } else if (is_swap_pte(ptent)) { swp_entry_t entry; ... entry = pte_to_swp_entry(ptent); if (!is_device_private_entry(entry) && !is_device_exclusive_entry(entry)) return false; } else if (!pte_present(ptent)) { return false; } pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); /* xxxxxxx */ return true; }for various reasons, for example, break-before-make for clearing access flags etc. pte can be set to none. since
page_vma_mapped_walk()doesn't hold ptl from the beginning, it might only begin to set swap entries from the middle of a large folio.For example, in case a large folio has 16 ptes, and 0,1,2 are somehow zero in the intermediate stage of a break-before-make, ptl will be held from the 3rd pte,
and swap entries will be set from 3rd pte as well. it seems not good as we are trying to swap out a large folio, but we are swapping out a part of them.I am still struggling with all the timing of races, but using
PVMW_SYNCto explicitly ask for ptl from the first pte seems a good thing for large folio regardless of those races. it can avoidtry_to_unmap_onereading intermediate pte and further make the wrong decision since reclaiming pte-mapped large folios is atomic with just one pte.Quoted 4 lines
Sorry I haven't progressed this series as I had hoped. I've been concentrating on getting the contpte series upstream. I'm hoping I will find some time to move this series along by the tail end of Feb (hoping to get it in shape for v6.10). Hopefully that doesn't cause you any big problems?
no worries. Anyway, we are already using your code to run various tests.
Quoted 3 lines
Thanks,
RyanThanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 20 lines · Ryan Roberts
Hi Barry,
Quoted 6 lines · Barry Song
Quoted 5 lines · Ryan Roberts
Quoted 3 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
[...]
Quoted 8 lines
Quoted 1 line
Hi Ryan,
I am running into some races especially while enabling large folio swap-out and swap-in both. some of them, i am still struggling with the detailed timing how they
are happening.
but the below change can help remove those bugs which cause corrupted data.
Quoted 7 lines
I'm getting quite confused with all the emails flying around on this topic. Here you were reporting a data corruption bug and your suggested fix below is the one you have now posted at [1]. But in the thread at [1] we concluded that it is not fixing a functional correctness issue, but is just an optimization in some corner cases. So does the corruption issue still manifest? Did you manage to root cause it? Is it a problem with my swap-out series or your swap-in series, or pre-existing?
Hi Ryan,
It is not a problem of your swap-out series, but a problem of my swap-in series. The bug in swap-in series is triggered by the skipped PTEs in the thread[1], but my swap-in code should still be able to cope with this situation and survive it - a large folio might be partially but not completely unmapped after try_to_unmap_one(). I actually replied to you and explained all the details here[2], but guess you missed it :-)
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/CAGsJ_4zdh5kOG7QP4UDaE-wmLFiTEJC2PX-_LxtOj=QrZSvkCA@mail.gmail.com/
apology this makes you confused.
Quoted 6 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
Ryan
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 29 lines · Barry Song
Quoted 20 lines · Ryan Roberts
Hi Barry,
Quoted 6 lines · Barry Song
Quoted 5 lines · Ryan Roberts
Quoted 3 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
[...]
Quoted 8 lines
Quoted 1 line
Hi Ryan,
I am running into some races especially while enabling large folio swap-out and swap-in both. some of them, i am still struggling with the detailed timing how they
are happening.
but the below change can help remove those bugs which cause corrupted data.Quoted 7 lines
I'm getting quite confused with all the emails flying around on this topic. Here you were reporting a data corruption bug and your suggested fix below is the one you have now posted at [1]. But in the thread at [1] we concluded that it is not fixing a functional correctness issue, but is just an optimization in some corner cases. So does the corruption issue still manifest? Did you manage to root cause it? Is it a problem with my swap-out series or your swap-in series, or pre-existing?
Quoted 7 lines
Hi Ryan,
It is not a problem of your swap-out series, but a problem of my swap-in series. The bug in swap-in series is triggered by the skipped PTEs in the thread[1], but my swap-in code should still be able to cope with this situation and survive it - a large folio might be partially but not completely unmapped after
try_to_unmap_one().
Ahh, understood, thanks!
Quoted 2 lines
I actually replied to you and explained all
the details here[2], but guess you missed it :-)
I did read that mail, but the first line "They are the same" made me think this was solving a functional problem. And I still have a very shaky understanding of parts of the code that I haven't directly worked on, so sometimes some of the details go over my head - I'll get there eventually!
Quoted 5 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/CAGsJ_4zdh5kOG7QP4UDaE-wmLFiTEJC2PX-_LxtOj=QrZSvkCA@mail.gmail.com/apology this makes you confused.
No need to apologise - I appreciate your taking the time to write it all down in detail. It helps me to learn these areas of the code.
Quoted 10 lines
Quoted 7 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
Thanks,
RyanThanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Barry SongQuoted 32 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1055
Quoted 7 lines
mm/swapfile.c +7
-0+/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 17 lines
mm/vmscan.c +6
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) && + !folio_entire_mapcount(folio) &&
Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 4 lines
mm/vmscan.c +0
-1split_folio_to_list(folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.
I can't claim to fully understand the problem yet (thanks for all the details - I'll keep reading it and looking at the code until I do), but I guess this problem should exist today for PMD-mappable folios? We already skip splitting those folios if they are pmd-mapped. Or does the problem only apply to pte-mapped folios?
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Barry SongQuoted 41 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1055
Quoted 7 lines
mm/swapfile.c +7
-0+/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.
Quoted 4 lines
in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise
MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,
Hi Barry,
Do you have a test case you can share that provokes this problem? And is this a separate problem to the race you solved with TTU_SYNC or is this solving the same problem?
Thanks,
Ryan
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 43 lines · Ryan Roberts
Quoted 41 lines · Barry Song
+Chris, Suren and Chuanhua
Hi Ryan,
mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1055
Quoted 7 lines
mm/swapfile.c +7
-0+/* +* __scan_swap_map_try_ssd_cluster() may drop si->lock during discard, +* so indicate that we are scanning to synchronise with swapoff. +*/ + si->flags +=SWP_SCANNING ; + ret =__scan_swap_map_try_ssd_cluster (si, &offset, &scan_base, order); + si->flags -=SWP_SCANNING ;nobody is using this
scan_baseafterwards. it seems a bit weird to pass a pointer.Quoted 11 lines
mm/vmscan.c +2
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list,
Quoted 10 lines
mm/vmscan.c +4
-2+* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked; --Chuanhua and I ran this patchset for a couple of days and found a race between reclamation and
split_folio. this might cause applications get wrong data 0 while swapping-in.
Quoted 5 lines
Quoted 4 lines
in case one thread(T1) is reclaiming a large folio by some means, still another thread is calling madvise
MADV_PGOUT(T2). and at the same time, we have two threads T3 and T4 to swap-in in parallel. T1 doesn't split and T2 does split as below,
Hi Ryan,
Quoted 5 lines
Hi Barry,
Do you have a test case you can share that provokes this problem? And is this a separate problem to the race you solved with
TTU_SYNCor is this solving the same problem?
They are the same.
After sending you the report about the races, I spent some time and finally figured
out what was happening, why corrupted data came while swapping in. it is absolutely
not your fault, but TTU_SYNC does somehow resolve my problem though it is not the root cause. this corrupted data only can reproduce after applying patch 4[1] of
swap-in series,
[1] [PATCH RFC 4/6] mm: support large folios swapin as a whole
https://lore.kernel.org/linux-mm/[email protected]/
In case we have a large folio with 16 PTEs as below, and after add_to_swap(), they get
swapoffset 0x10000, their PTEs are all present as they are still mapped.
PTE pte_stat
PTE0 present PTE1 present PTE2 present PTE3 present
...
PTE15 present
then we get to try_to_unmap_one, as try_to_unmap_one doesn't hold PTL from PTE0, while it scans PTEs, we might have
PTE pte_stat
PTE0 none (someone is writing PTE0 for various reasons) PTE1 present PTE2 present PTE3 present
...
PTE15 present
We hold PTL from PTE1.
after try_to_unmap_one, PTEs become
PTE pte_stat
PTE0 present (someone finished the write of PTE0) PTE1 swap 0x10001 PTE2 swap 0x10002 PTE3 swap 0x10003
...
...
PTE15 swap 0x1000F
Thus, after try_to_unmap_one, the large folio is still mapped. so its swapcache will still be there.
Now a parallel thread runs MADV_PAGEOUT, and it finds this large folio is not completely mapped, so it splits the folio into 16 small folios but their swap offsets are kept.
Now in swapcache, we have 16 folios with contiguous swap offsets. MADV_PAGEOUT will reclaim these 16 folios, after new 16 try_to_unmap_one,
PTE pte_stat
PTE0 swap 0x10000 SWAP_HAS_CACHE PTE1 swap 0x10001 SWAP_HAS_CACHE PTE2 swap 0x10002 SWAP_HAS_CACHE PTE3 swap 0x10003 SWAP_HAS_CACHE
...
PTE15 swap 0x1000F SWAP_HAS_CACHE
From this time, we can have various different cases for these 16 PTEs. for example,
PTE pte_stat
PTE0 swap 0x10000 SWAP_HAS_CACHE = 0 -> become false due to
finished pageout and remove_mapping
PTE1 swap 0x10001 SWAP_HAS_CACHE = 0 -> become false due to
finished pageout and remove_mapping
PTE2 swap 0x10002 SWAP_HAS_CACHE = 0 -> become false due to
concurrent swapin and swapout
PTE3 swap 0x10003 SWAP_HAS_CACHE = 1
...
PTE13 swap 0x1000D SWAP_HAS_CACHE = 1 PTE14 swap 0x1000E SWAP_HAS_CACHE = 1 PTE15 swap 0x1000F SWAP_HAS_CACHE = 1
but all of them have swp_count = 1 and different SWAP_HAS_CACHE. some of these small folios might be in swapcache, some others might not be in.
then we do_swap_page at one PTE whose SWAP_HAS_CACHE=0 and swap_count=1 (the folio is not in swapcache, thus has been written to swap), we do this check:
static bool pte_range_swap(pte_t *pte, int nr_pages)
{
int i;
swp_entry_t entry;
unsigned type;
pgoff_t start_offset;
entry = pte_to_swp_entry(ptep_get_lockless(pte));
if (non_swap_entry(entry))
return false;
start_offset = swp_offset(entry);
if (start_offset % nr_pages)
return false;
type = swp_type(entry);
for (i = 1; i < nr_pages; i++) {
entry = pte_to_swp_entry(ptep_get_lockless(pte + i));
if (non_swap_entry(entry))
return false;
if (swp_offset(entry) != start_offset + i)
return false;
if (swp_type(entry) != type)
return false;
}
return true;
}as those swp entries are contiguous, we will call swap_read_folio(). For those folios which are still in swapcache and haven't been written, we get zero-filled data from zRAM.
So the root cause is that pte_range_swap should check all 16 swap_map have the same SWAP_HAS_CACHE as false.
static bool is_pte_range_contig_swap(pte_t *pte, int nr_pages)
{
...
count = si->swap_map[start_offset];
for (i = 1; i < nr_pages; i++) {
entry = pte_to_swp_entry(ptep_get_lockless(pte + i));
if (non_swap_entry(entry))
return false;
if (swp_offset(entry) != start_offset + i)
return false;
if (swp_type(entry) != type)
return false;
/* fallback to small folios if SWAP_HAS_CACHE isn't same */
if (si->swap_map[start_offset + i] != count)
return false;
}
return true;
}but somehow TTU_SYNC "resolves" it by giving no chance to MADV_PAGEOUT to split this folio as the large folio are either entirely written by swap entries, or entirely keep present PTEs.
Though the bug is within the swap-in series, I am still a big fan of TTU_SYNC for large folio reclamation for at least three reasons,
1. We remove some possibility that large folios fail to be reclaimed, improving reclamation efficiency.
2. We avoid many strange cases and potential folio_split during reclamation. without TTU_SYNC, folios can be splitted later, or partially being set to swap entries while partially being still present
3. we don't increase PTL contention. My test shows try_to_unmap_one will always get PTL after it sometimes skips one or two PTEs because intermediate break-before-makes are short. Of course, most time try_to_unmap_one will get PTL from PTE0.
Quoted 4 lines
Thanks,
Ryan
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Ryan RobertsHi Ryan,
Quoted 12 lines
mm/vmscan.c +1
-3@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 10 lines
mm/vmscan.c +5
-1+* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ -if (!folio_entire_mapcount (folio) && +if (folio_test_pmd_mappable (folio) && + !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;
I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios, MADV_DONTNEED 4KB~64KB, and keep the first subpage 0~4KiB.
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcache
what i have done is printing folio->_nr_pages_mapped and dumping 16 swap_map[] at some specific stage
1. just after add_to_swap (swap slots are allocated)
2. before and after try_to_unmap (ptes are set to swap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and after remove_mapping
The below is the dumped info for a particular large folio,
1. after add_to_swap
[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40
as you can see,_nr_pages_mapped is 1 and all 16 swap_map are SWAP_HAS_CACHE (0x40)
2. before and after try_to_unmap
[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40
as you can see, one pte is set to swp_entry, and _nr_pages_mapped becomes 0 from 1. The 1st swp_map becomes 0x41, SWAP_HAS_CACHE + 1
3. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0
as you can see, obviously, we have done redundant I/O - 16 zram_write_page though 4~64KiB has been zap_pte_range before, we still write them to zRAM.
4. before and after remove_mapping
[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
as you can see, swp_map 1-15 becomes 0 and only the first swp_map is 1. all SWAP_HAS_CACHE has been removed. This is perfect and there is no swap slot leak at all!
Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrent do_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 20 lines · Barry Song
Hi Ryan,
Quoted 18 lines
mm/vmscan.c +5
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) &&
Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 4 lines
mm/vmscan.c +1
-0+ !folio_entire_mapcount(folio) && split_folio_to_list(folio, folio_list)) goto activate_locked;I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios,
MADV_DONTNEED4KB~64KB, and keep the first subpage 0~4KiB.
IOW, something that already happens with ordinary THP already IIRC.
Quoted 65 lines
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcachewhat i have done is printing
folio->_nr_pages_mappedand dumping 16 swap_map[] at some specific stage
1. just afteradd_to_swap(swap slots are allocated)
2. before and aftertry_to_unmap(ptes are set toswap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and afterremove_mappingThe below is the dumped info for a particular large folio,
1. after
add_to_swap[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see,
_nr_pages_mappedis 1 and all 16swap_mapareSWAP_HAS_CACHE(0x40)2. before and after
try_to_unmap[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see, one pte is set to
swp_entry, and_nr_pages_mappedbecomes 0 from 1. The 1stswp_mapbecomes0x41,SWAP_HAS_CACHE+ 13. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0as you can see, obviously, we have done redundant I/O - 16
zram_write_pagethough 4~64KiB has beenzap_pte_rangebefore, we still write them to zRAM.4. before and after
remove_mapping[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00as you can see,
swp_map1-15 becomes 0 and only the firstswp_mapis 1. allSWAP_HAS_CACHEhas been removed. This is perfect and there is no swap slot leak at all!
Quoted 7 lines
Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrentdo_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...
Just noting: I was running into something different in the past with THP. And it's effectively the same scenario, just swapout and MADV_DONTNEED reversed.
Imagine you swapped out a THP and the THP it still is in the swapcache.
Then you unmap/zap some PTEs, freeing up the swap slots.
In zap_pte_range(), we'll call free_swap_and_cache(). There, we run into the "!swap_page_trans_huge_swapped(p, entry)", and we won't be calling __try_to_reclaim_swap().
So we won't split the large folio that is in the swapcache and it will continue consuming "more memory" than intended until fully evicted.
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of _nr_pages_mapped if possible.
If we find that the folio is on the deferred split list, we might as well just split it right away, before swapping it out. That might be a reasonable optimization for the case you describe.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 110 lines · David Hildenbrand
Quoted 20 lines · Barry Song
Hi Ryan,
Quoted 18 lines
mm/vmscan.c +5
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) &&Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 4 lines
mm/vmscan.c +1
-0+ !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios,
MADV_DONTNEED4KB~64KB, and keep the first subpage 0~4KiB.IOW, something that already happens with ordinary THP already IIRC.
Quoted 65 lines
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcachewhat i have done is printing
folio->_nr_pages_mappedand dumping 16 swap_map[] at some specific stage
1. just afteradd_to_swap(swap slots are allocated)
2. before and aftertry_to_unmap(ptes are set toswap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and afterremove_mappingThe below is the dumped info for a particular large folio,
1. after
add_to_swap[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see,
_nr_pages_mappedis 1 and all 16swap_mapareSWAP_HAS_CACHE(0x40)2. before and after
try_to_unmap[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see, one pte is set to
swp_entry, and_nr_pages_mappedbecomes 0 from 1. The 1stswp_mapbecomes0x41,SWAP_HAS_CACHE+ 13. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0as you can see, obviously, we have done redundant I/O - 16
zram_write_pagethough 4~64KiB has beenzap_pte_rangebefore, we still write them to zRAM.4. before and after
remove_mapping[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00as you can see,
swp_map1-15 becomes 0 and only the firstswp_mapis 1. allSWAP_HAS_CACHEhas been removed. This is perfect and there is no swap slot leak at all!Quoted 7 lines
Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrentdo_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...Just noting: I was running into something different in the past with THP. And it's effectively the same scenario, just swapout and
MADV_DONTNEEDreversed.
Quoted 7 lines
Imagine you swapped out a THP and the THP it still is in the swapcache.
Then you unmap/zap some PTEs, freeing up the swap slots.
In
zap_pte_range(), we'll callfree_swap_and_cache(). There, we run into the "!swap_page_trans_huge_swapped(p, entry)", and we won't be calling__try_to_reclaim_swap().
I guess you mean swap_page_trans_huge_swapped(p, entry) not !swap_page_trans_huge_swapped(p, entry) ?
at that time, swap_page_trans_huge_swapped should be true as there are still some entries whose swap_map=0x41 or above (SWAP_HAS_CACHE and swap_count >= 1)
static bool swap_page_trans_huge_swapped(struct swap_info_struct *si,
swp_entry_t entry, unsigned int nr_pages)
{
...
for (i = 0; i < nr_pages; i++) {
if (swap_count(map[offset + i])) {
ret = true;
break;
}
}
unlock_out:
unlock_cluster_or_swap_info(si, ci);
return ret;
}So this will stop the swap free even for those ptes which have been zapped?
Another case I have reported[1] is that while reclaiming a large folio, in try_to_unmap_one, we are calling page_vma_mapped_walk(). as it only begins to hold PTL after it hits a valid pte, a paralel break-before-make might make 0nd, 1st and beginning PTEs zero, try_to_unmap_one can read intermediate ptes value, thus we can run into this below case. afte try_to_unmap_one,
pte 0 - untouched, present pte
pte 1 - untouched, present pte
pte 2 - swap entries
pte 3 - swap entries
...
pte n - swap entries
or
pte 0 - untouched, present pte
pte 1 - swap entries pte 2 - swap entries pte 3 - swap entries
...
pte n - swap entries
etc.
Thus, after try_to_unmap, the folio is still mapped with some ptes becoming swap entries, some PTEs are still present. it might be staying in swapcache for a long time with a broken CONT-PTE.
I also hate that and hope for a SYNC way to let large folio hold PTL from the 0nd pte, thus, it won't get intermediate PTEs from other break-before-make.
This also doesn't increase PTL contention as my test shows we will always get PTL for a large folio after skipping zero, one or two PTEs in try_to_unmap_one.
but skipping 1 or 2 is really bad and sad, breaking a large folio from the same whole to nr_pages different segments.
[1] https://lore.kernel.org/linux-mm/CAGsJ_4wo7BiJWSKb1K_WyAai30KmfckMQ3-mCJPXZ892CtXpyQ@mail.gmail.com/
Quoted 11 lines
So we won't split the large folio that is in the swapcache and it will continue consuming "more memory" than intended until fully evicted.
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.
Is _nr_pages_mapped < nr_pages a reasonable case to split as we have known the folio has at least some subpages zapped?
Quoted 3 lines
If we find that the folio is on the deferred split list, we might as well just split it right away, before swapping it out. That might be a reasonable optimization for the case you describe.
i tried to change Ryan's code as below
@@ -1905,11 +1922,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
* PMD map right away. Chances are some
* or all of the tail pages can be freed
* without IO.
+ * Similarly, split PTE-mapped folios if
+ * they have been alreadydeferred_split.
*/
- if (folio_test_pmd_mappable(folio) &&
- !folio_entire_mapcount(folio) &&
- split_folio_to_list(folio,
- folio_list))
+ if
(((folio_test_pmd_mappable(folio) && !folio_entire_mapcount(folio)) || +
(!folio_test_pmd_mappable(folio) && !list_empty(&folio->_deferred_list)))
+ &&split_folio_to_list(folio, folio_list))
goto activate_locked;
}
if (!add_to_swap(folio)) {It seems to work as expected. only one I/O is left for a large folio with 16 PTEs
but 15 of them have been zapped before.
Quoted 6 lines
--
Cheers,David / dhildenb
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 95 lines · Barry Song
Quoted 94 lines · David Hildenbrand
Quoted 20 lines · Barry Song
Hi Ryan,
Quoted 18 lines
mm/vmscan.c +5
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) &&Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 4 lines
mm/vmscan.c +1
-0+ !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios,
MADV_DONTNEED4KB~64KB, and keep the first subpage 0~4KiB.IOW, something that already happens with ordinary THP already IIRC.
Quoted 61 lines
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcachewhat i have done is printing
folio->_nr_pages_mappedand dumping 16 swap_map[] at some specific stage
1. just afteradd_to_swap(swap slots are allocated)
2. before and aftertry_to_unmap(ptes are set toswap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and afterremove_mappingThe below is the dumped info for a particular large folio,
1. after
add_to_swap[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see,
_nr_pages_mappedis 1 and all 16swap_mapareSWAP_HAS_CACHE(0x40)2. before and after
try_to_unmap[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see, one pte is set to
swp_entry, and_nr_pages_mappedbecomes 0 from 1. The 1stswp_mapbecomes0x41,SWAP_HAS_CACHE+ 13. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0as you can see, obviously, we have done redundant I/O - 16
zram_write_pagethough 4~64KiB has beenzap_pte_rangebefore, we still write them to zRAM.4. before and after
remove_mapping[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
as you can see,
swp_map1-15 becomes 0 and only the firstswp_mapis 1. allSWAP_HAS_CACHEhas been removed. This is perfect and there is no swap slot leak at all!Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.
So the cost of this is increased IO and swap storage, correct? Is this a big problem in practice? i.e. do you see a lot of partially mapped large folios in your workload? (I agree the proposed fix below is simple, so I think we should do it anyway - I'm just interested in the scale of the problem).
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrent
do_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...
Yes mine too. I would only expect the ptes that map the folio will get replaced with swap entries? So I would expect it to be safe. Although I understand the concern with the extra swap consumption.
[...]
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.Is
_nr_pages_mapped<nr_pagesa reasonable case to split as we have known the folio has at least some subpages zapped?
I'm not sure we need this - the folio's presence on the split list will tell us everything we need to know I think?
Quoted 4 lines
Quoted 4 lines
If we find that the folio is on the deferred split list, we might as well just split it right away, before swapping it out. That might be a reasonable optimization for the case you describe.
Yes, agreed. I think there is still chance of a race though; Some other thread could be munmapping in parallel. But in that case, I think we just end up with the increased IO and swap storage? That's not the end of the world if its a corner case.
Quoted 17 lines
i tried to change Ryan's code as below
(quoted diff) +2
-0@@ -1905,11 +1922,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * PMD map right away. Chances are some * or all of the tail pages can be freed * without IO. + * Similarly, split PTE-mapped folios if + * they have been already
deferred_split.*/(quoted diff) +1
-4- if (folio_test_pmd_mappable(folio) && - !folio_entire_mapcount(folio) && - split_folio_to_list(folio, - folio_list)) + if
Quoted 4 lines
(((
folio_test_pmd_mappable(folio)&& !folio_entire_mapcount(folio)) || +
(!folio_test_pmd_mappable(folio) && !list_empty(&folio->_deferred_list)))
I'm not sure we need the different tests for pmd_mappable vs !pmd_mappable. I think presence on the deferred list is a sufficient indicator that there are unmapped subpages?
I'll incorporate this into my next version.
Quoted 19 lines
+ &&
split_folio_to_list(folio, folio_list))goto activate_locked; } if (!add_to_swap(folio)) {It seems to work as expected. only one I/O is left for a large folio with 16 PTEs
but 15 of them have been zapped before.Quoted 6 lines
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 110 lines · Ryan Roberts
Quoted 95 lines · Barry Song
Quoted 94 lines · David Hildenbrand
Quoted 20 lines · Barry Song
Hi Ryan,
Quoted 18 lines
mm/vmscan.c +5
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) &&Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 4 lines
mm/vmscan.c +1
-0+ !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios,
MADV_DONTNEED4KB~64KB, and keep the first subpage 0~4KiB.IOW, something that already happens with ordinary THP already IIRC.
Quoted 61 lines
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcachewhat i have done is printing
folio->_nr_pages_mappedand dumping 16 swap_map[] at some specific stage
1. just afteradd_to_swap(swap slots are allocated)
2. before and aftertry_to_unmap(ptes are set toswap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and afterremove_mappingThe below is the dumped info for a particular large folio,
1. after
add_to_swap[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see,
_nr_pages_mappedis 1 and all 16swap_mapareSWAP_HAS_CACHE(0x40)2. before and after
try_to_unmap[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see, one pte is set to
swp_entry, and_nr_pages_mappedbecomes 0 from 1. The 1stswp_mapbecomes0x41,SWAP_HAS_CACHE+ 13. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0as you can see, obviously, we have done redundant I/O - 16
zram_write_pagethough 4~64KiB has beenzap_pte_rangebefore, we still write them to zRAM.4. before and after
remove_mapping[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
as you can see,
swp_map1-15 becomes 0 and only the firstswp_mapis 1. allSWAP_HAS_CACHEhas been removed. This is perfect and there is no swap slot leak at all!Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.So the cost of this is increased IO and swap storage, correct? Is this a big problem in practice? i.e. do you see a lot of partially mapped large folios in your workload? (I agree the proposed fix below is simple, so I think we should do it anyway - I'm just interested in the scale of the problem).
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrent
do_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...Yes mine too. I would only expect the ptes that map the folio will get replaced with swap entries? So I would expect it to be safe. Although I understand the concern with the extra swap consumption.
yes. it should still be safe. just more I/O and more swap spaces. but they will be removed while remove_mapping happens if try_to_unmap_one makes the folio unmapped.
but with the potential possibility even mapped PTEs can be skipped by try_to_unmap_one (reported intermediate PTEs issue - PTL is held till a valid PTE, some PTEs might be skipped by try_to_unmap without being set to swap entries), we could have the possibility folio_mapped() is still true after try_to_unmap_one. so we can't get to __remove_mapping() for a long time. but it still doesn't cause a crash.
Quoted 10 lines
[...]
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.
Quoted 5 lines
Quoted 2 lines
Is
_nr_pages_mapped<nr_pagesa reasonable case to split as we have known the folio has at least some subpages zapped?I'm not sure we need this - the folio's presence on the split list will tell us everything we need to know I think?
I agree, this is just one question to David, not my proposal. if deferred_list is sufficient,
I prefer we use deferred_list.
I actually don't quite understand why David dislikes _nr_pages_mapped being used though I do think _nr_pages_mapped cannot precisely reflect how a folio is mapped
by multi-processes. but _nr_pages_mapped < nr_pages seems be safe to tell the folio
is partially unmapped :-)
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
If we find that the folio is on the deferred split list, we might as well just split it right away, before swapping it out. That might be a reasonable optimization for the case you describe.
Yes, agreed. I think there is still chance of a race though; Some other thread could be munmapping in parallel. But in that case, I think we just end up with the increased IO and swap storage? That's not the end of the world if its a corner case.
I agree. btw, do we need a spinlock ds_queue->split_queue_lock for checking the list? deferred_split_folio(), for itself, has no spinlock while checking
if (!list_empty(&folio->_deferred_list)), but why? the read and write
need to be exclusive.....
void deferred_split_folio(struct folio *folio)
{
...
if (!list_empty(&folio->_deferred_list))
return;
spin_lock_irqsave(&ds_queue->split_queue_lock, flags);
if (list_empty(&folio->_deferred_list)) {
count_vm_event(THP_DEFERRED_SPLIT_PAGE);
list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
ds_queue->split_queue_len++;
#ifdef CONFIG_MEMCG
if (memcg)
set_shrinker_bit(memcg, folio_nid(folio),
deferred_split_shrinker->id);
#endif
}
spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags);
}Quoted 18 lines
Quoted 18 lines
i tried to change Ryan's code as below
(quoted diff) +2
-0@@ -1905,11 +1922,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * PMD map right away. Chances are some * or all of the tail pages can be freed * without IO. + * Similarly, split PTE-mapped folios if + * they have been already
deferred_split.*/(quoted diff) +1
-4- if (folio_test_pmd_mappable(folio) && - !folio_entire_mapcount(folio) && - split_folio_to_list(folio, - folio_list)) + if
Quoted 8 lines
Quoted 4 lines
(((
folio_test_pmd_mappable(folio)&& !folio_entire_mapcount(folio)) || +
(!folio_test_pmd_mappable(folio) && !list_empty(&folio->_deferred_list)))I'm not sure we need the different tests for
pmd_mappablevs !pmd_mappable. I think presence on the deferred list is a sufficient indicator that there are unmapped subpages?
I don't think there are fundamental differences for pmd and pte. i was testing pte-mapped folio at that time, so kept the behavior of pmd as is.
Quoted 2 lines
I'll incorporate this into my next version.
Great!
Quoted 12 lines
Quoted 12 lines
+ &&
split_folio_to_list(folio, folio_list))goto activate_locked; } if (!add_to_swap(folio)) {It seems to work as expected. only one I/O is left for a large folio with 16 PTEs
but 15 of them have been zapped before.Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
--
Cheers,David / dhildenb
Thanks
Barry
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.Quoted 5 lines
Quoted 2 lines
Is
_nr_pages_mapped<nr_pagesa reasonable case to split as we have known the folio has at least some subpages zapped?I'm not sure we need this - the folio's presence on the split list will tell us everything we need to know I think?
I agree, this is just one question to David, not my proposal. if
deferred_listis sufficient,
I prefer we usedeferred_list.
Quoted 6 lines
I actually don't quite understand why David dislikes
_nr_pages_mappedbeing used though I do think_nr_pages_mappedcannot precisely reflect how a folio is mapped
by multi-processes. but_nr_pages_mapped<nr_pagesseems be safe to tell the folio
is partially unmapped :-)
I'm hoping we can get rid of _nr_pages_mapped in some kernel configs in the future (that's what I am working on). So the less we depend on it the better.
With the total mapcount patch I'll revive shortly, _nr_pages_mapped will only be used inside rmap code. I'm hoping we won't have to introduce other users that will be harder to get rid of.
So please, if avoidable, no usage of _nr_pages_mapped outside of core rmap code.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
Quoted 30 lines · David Hildenbrand
Quoted 19 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.Quoted 5 lines
Quoted 2 lines
Is
_nr_pages_mapped<nr_pagesa reasonable case to split as we have known the folio has at least some subpages zapped?I'm not sure we need this - the folio's presence on the split list will tell us everything we need to know I think?
I agree, this is just one question to David, not my proposal. if
deferred_listis sufficient,
I prefer we usedeferred_list.Quoted 6 lines
I actually don't quite understand why David dislikes
_nr_pages_mappedbeing used though I do think_nr_pages_mappedcannot precisely reflect how a folio is mapped
by multi-processes. but_nr_pages_mapped<nr_pagesseems be safe to tell the folio
is partially unmapped :-)I'm hoping we can get rid of
_nr_pages_mappedin some kernel configs in the future (that's what I am working on). So the less we depend on it the better.
Quoted 6 lines
With the total mapcount patch I'll revive shortly,
_nr_pages_mappedwill only be used inside rmap code. I'm hoping we won't have to introduce other users that will be harder to get rid of.So please, if avoidable, no usage of
_nr_pages_mappedoutside of core rmap code.
Thanks for clarification on the plan. good to use deferred_list in this swap-out case.
Quoted 6 lines
--
Cheers,David / dhildenb
Thanks
Barry
Ryan Roberts
reviewerRe: [PATCH v3 4/4] mm: swap: Swap-out small-sized THP without splitting
In reply to Barry SongQuoted 169 lines · Barry Song
Quoted 110 lines · Ryan Roberts
Quoted 95 lines · Barry Song
Quoted 94 lines · David Hildenbrand
Quoted 20 lines · Barry Song
Hi Ryan,
Quoted 18 lines
mm/vmscan.c +5
-4@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!can_split_folio (folio,NULL )) goto activate_locked;/* -* Split folios without a PMD map right -* away. Chances are some or all of the -* tail pages can be freed without IO. +* Split PMD-mappable folios without a +* PMD map right away. Chances are some +* or all of the tail pages can be freed +* without IO. */ - if (!folio_entire_mapcount(folio) && + if (folio_test_pmd_mappable(folio) &&Quoted 8 lines
mm/vmscan.c static unsigned int shrink_folio_list(struct list_head *folio_list, · line 1224
Quoted 4 lines
mm/vmscan.c +1
-0+ !folio_entire_mapcount (folio) &&split_folio_to_list (folio, folio_list)) goto activate_locked;I ran a test to investigate what would happen while reclaiming a partially unmapped large folio. for example, for 64KiB large folios,
MADV_DONTNEED4KB~64KB, and keep the first subpage 0~4KiB.IOW, something that already happens with ordinary THP already IIRC.
Quoted 61 lines
My test wants to address my three concerns,
a. whether we will have leak on swap slots
b. whether we will have redundant I/O
c. whether we will cause races on swapcachewhat i have done is printing
folio->_nr_pages_mappedand dumping 16 swap_map[] at some specific stage
1. just afteradd_to_swap(swap slots are allocated)
2. before and aftertry_to_unmap(ptes are set toswap_entry)
3. before and after pageout (also add printk in zram driver to dump all I/O write)
4. before and afterremove_mappingThe below is the dumped info for a particular large folio,
1. after
add_to_swap[ 27.267357] vmscan: After add_to_swap shrink_folio_list 1947 mapnr:1 [ 27.267650] vmscan: offset:101b0 swp_map 40-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see,
_nr_pages_mappedis 1 and all 16swap_mapareSWAP_HAS_CACHE(0x40)2. before and after
try_to_unmap[ 27.268067] vmscan: before try to unmap shrink_folio_list 1991 mapnr:1 [ 27.268372] try_to_unmap_one address:ffff731f0000 pte:e8000103cd0b43 pte_p:ffff0000c36a8f80 [ 27.268854] vmscan: after try to unmap shrink_folio_list 1997 mapnr:0 [ 27.269180] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40as you can see, one pte is set to
swp_entry, and_nr_pages_mappedbecomes 0 from 1. The 1stswp_mapbecomes0x41,SWAP_HAS_CACHE+ 13. before and after pageout
[ 27.269602] vmscan: before pageout shrink_folio_list 2065 mapnr:0 [ 27.269880] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.270691] zram: zram_write_page page:fffffc00030f3400 index:101b0 [ 27.271061] zram: zram_write_page page:fffffc00030f3440 index:101b1 [ 27.271416] zram: zram_write_page page:fffffc00030f3480 index:101b2 [ 27.271751] zram: zram_write_page page:fffffc00030f34c0 index:101b3 [ 27.272046] zram: zram_write_page page:fffffc00030f3500 index:101b4 [ 27.272384] zram: zram_write_page page:fffffc00030f3540 index:101b5 [ 27.272746] zram: zram_write_page page:fffffc00030f3580 index:101b6 [ 27.273042] zram: zram_write_page page:fffffc00030f35c0 index:101b7 [ 27.273339] zram: zram_write_page page:fffffc00030f3600 index:101b8 [ 27.273676] zram: zram_write_page page:fffffc00030f3640 index:101b9 [ 27.274044] zram: zram_write_page page:fffffc00030f3680 index:101ba [ 27.274554] zram: zram_write_page page:fffffc00030f36c0 index:101bb [ 27.274870] zram: zram_write_page page:fffffc00030f3700 index:101bc [ 27.275166] zram: zram_write_page page:fffffc00030f3740 index:101bd [ 27.275463] zram: zram_write_page page:fffffc00030f3780 index:101be [ 27.275760] zram: zram_write_page page:fffffc00030f37c0 index:101bf [ 27.276102] vmscan: after pageout and before needs_release shrink_folio_list 2124 mapnr:0as you can see, obviously, we have done redundant I/O - 16
zram_write_pagethough 4~64KiB has beenzap_pte_rangebefore, we still write them to zRAM.4. before and after
remove_mapping[ 27.276428] vmscan: offset:101b0 swp_map 41-40-40-40-40-40-40-40-40-40-40-40-40-40-40-40 [ 27.277485] vmscan: after remove_mapping shrink_folio_list 2169 mapnr:0 offset:0 [ 27.277802] vmscan: offset:101b0 01-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
as you can see,
swp_map1-15 becomes 0 and only the firstswp_mapis 1. allSWAP_HAS_CACHEhas been removed. This is perfect and there is no swap slot leak at all!Thus, only two concerns are left for me,
1. as we don't split anyway, we have done 15 unnecessary I/O if a large folio is partially unmapped.So the cost of this is increased IO and swap storage, correct? Is this a big problem in practice? i.e. do you see a lot of partially mapped large folios in your workload? (I agree the proposed fix below is simple, so I think we should do it anyway - I'm just interested in the scale of the problem).
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
2. large folio is added as a whole as a swapcache covering the range whose part has been zapped. I am not quite sure if this will cause some problems while some concurrent
do_anon_page, swapin and swapout occurs between 3 and 4 on zapped subpage1~subpage15. still struggling.. my brain is exploding...Yes mine too. I would only expect the ptes that map the folio will get replaced with swap entries? So I would expect it to be safe. Although I understand the concern with the extra swap consumption.
yes. it should still be safe. just more I/O and more swap spaces. but they will be removed while
remove_mappinghappens iftry_to_unmap_onemakes the folio unmapped.but with the potential possibility even mapped PTEs can be skipped by
try_to_unmap_one(reported intermediate PTEs issue - PTL is held till a valid PTE, some PTEs might be skipped bytry_to_unmapwithout being set to swap entries), we could have the possibilityfolio_mapped()is still true aftertry_to_unmap_one. so we can't get to__remove_mapping()for a long time. but it still doesn't cause a crash.Quoted 10 lines
[...]
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
To me, it seems safer to split or do some other similar optimization if we find a large folio has partial map and unmap.
I'm hoping that we can avoid any new direct users of
_nr_pages_mappedif possible.Quoted 5 lines
Quoted 2 lines
Is
_nr_pages_mapped<nr_pagesa reasonable case to split as we have known the folio has at least some subpages zapped?I'm not sure we need this - the folio's presence on the split list will tell us everything we need to know I think?
I agree, this is just one question to David, not my proposal. if
deferred_listis sufficient,
I prefer we usedeferred_list.I actually don't quite understand why David dislikes
_nr_pages_mappedbeing used though I do think_nr_pages_mappedcannot precisely reflect how a folio is mapped
by multi-processes. but_nr_pages_mapped<nr_pagesseems be safe to tell the folio
is partially unmapped :-)Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
If we find that the folio is on the deferred split list, we might as well just split it right away, before swapping it out. That might be a reasonable optimization for the case you describe.
Yes, agreed. I think there is still chance of a race though; Some other thread could be munmapping in parallel. But in that case, I think we just end up with the increased IO and swap storage? That's not the end of the world if its a corner case.
Quoted 4 lines
I agree. btw, do we need a spinlock
ds_queue->split_queue_lockfor checking the list?deferred_split_folio(), for itself, has no spinlock while checking
if (!list_empty(&folio->_deferred_list)), but why? the read and write
need to be exclusive.....
I don't think so. It's safe to check if the folio is on the queue like this; but if it isn't then you need to recheck under the lock, as is done here. So for us, I think we can also do this safely. It is certainly preferable to avoid taking the lock.
The original change says this:
Before acquire split_queue_lock, check and bail out early if the THP head page is in the queue already. The checking without holding split_queue_lock could race with deferred_split_scan, but it doesn't impact the correctness here.
Quoted 78 lines
void
deferred_split_folio(struct folio *folio){ ... if (!list_empty(&folio->_deferred_list)) return; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); if (list_empty(&folio->_deferred_list)) { count_vm_event(THP_DEFERRED_SPLIT_PAGE); list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); ds_queue->split_queue_len++; #ifdef CONFIG_MEMCG if (memcg) set_shrinker_bit(memcg, folio_nid(folio), deferred_split_shrinker->id); #endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); }Quoted 18 lines
Quoted 18 lines
i tried to change Ryan's code as below
(quoted diff) +2
-0@@ -1905,11 +1922,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * PMD map right away. Chances are some * or all of the tail pages can be freed * without IO. + * Similarly, split PTE-mapped folios if + * they have been already
deferred_split.*/(quoted diff) +1
-4- if (folio_test_pmd_mappable(folio) && - !folio_entire_mapcount(folio) && - split_folio_to_list(folio, - folio_list)) + ifQuoted 8 lines
Quoted 4 lines
(((
folio_test_pmd_mappable(folio)&& !folio_entire_mapcount(folio)) || +
(!folio_test_pmd_mappable(folio) && !list_empty(&folio->_deferred_list)))I'm not sure we need the different tests for
pmd_mappablevs !pmd_mappable. I think presence on the deferred list is a sufficient indicator that there are unmapped subpages?I don't think there are fundamental differences for pmd and pte. i was testing pte-mapped folio at that time, so kept the behavior of pmd as is.
Quoted 2 lines
I'll incorporate this into my next version.
Great!
Quoted 12 lines
Quoted 12 lines
+ &&
split_folio_to_list(folio, folio_list))goto activate_locked; } if (!add_to_swap(folio)) {It seems to work as expected. only one I/O is left for a large folio with 16 PTEs
but 15 of them have been zapped before.Quoted 1 line
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
--
Cheers,David / dhildenb
Thanks
Barry
Quoted 52 lines
Hi All,
This is v3 of a series to add support for swapping out small-sized THP without needing to first split the large folio via
__split_huge_page(). It closely follows the approach already used by PMD-sized THP."Small-sized THP" is an upcoming feature that enables performance improvements by allocating large folios for anonymous memory, where the large folio size is smaller than the traditional PMD-size. See [3].
In some circumstances I've observed a performance regression (see patch 2 for details), and this series is an attempt to fix the regression in advance of merging small-sized THP support.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is probably sufficient.
The series applies against mm-unstable (
1a3c85fa684a)Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Thanks to Huang, Ying for the review feedback and suggestions!
Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).
Quoted 7 lines
Thanks,
Ryan
P.S. I know we agreed this is not a prerequisite for merging small-sized THP, but given Huang Ying had provided some review feedback, I wanted to progress it. All the actual prerequisites are either complete or being worked on by others.
Hi Ryan,
this is quite important to a phone and a must-have component, so is large-folio swapin, as i explained to you in another email.
Luckily, we are having Chuanhua Han(Cc-ed) to prepare a patchset of largefolio swapin on top of your this patchset, probably a port and cleanup of our do_swap_page[1] againest yours.
Another concern is that swapslots can be fragmented, if we place small/large folios in a swap device, since large folios always require contiguous swapslot, we can result in failure of getting slots even we still have many free slots which are not contiguous. To avoid this, [2] dynamic hugepage solution have two swap devices, one for basepage, the other one for CONTPTE. we have modified the priority-based selection of swap devices to choose swap devices based on small/large folios. i realize this approache is super ugly and might be very hard to find a way to upstream though, it seems not universal especially if you are a linux server (-_-)
two devices are not a nice approach though it works well for a real product, we might still need some decent way to address this problem while the problem is for sure not a stopper of your patchset.
[1] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/memory.c#L4648
[2] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/swapfile.c#L1129
Quoted 12 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Ryan Roberts (4):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap: Remove structpercpu_cluster
mm: swap: Simplify ssd behavior when scanner steals entry
mm: swap: Swap-out small-sized THP without splitting
Quoted 5 lines
include/linux/swap.h | 31 +++--- mm/huge_memory.c | 3 - mm/swapfile.c | 232 ++++++++++++++++++++++++------------------- mm/vmscan.c | 10 +- 4 files changed, 149 insertions(+), 127 deletions(-)
Thanks
Barry
Ryan Roberts
reviewerQuoted 60 lines · Barry Song
Quoted 52 lines
Hi All,
This is v3 of a series to add support for swapping out small-sized THP without needing to first split the large folio via
__split_huge_page(). It closely follows the approach already used by PMD-sized THP."Small-sized THP" is an upcoming feature that enables performance improvements by allocating large folios for anonymous memory, where the large folio size is smaller than the traditional PMD-size. See [3].
In some circumstances I've observed a performance regression (see patch 2 for details), and this series is an attempt to fix the regression in advance of merging small-sized THP support.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is probably sufficient.
The series applies against mm-unstable (
1a3c85fa684a)Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Thanks to Huang, Ying for the review feedback and suggestions!
Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).Quoted 7 lines
Thanks,
Ryan
P.S. I know we agreed this is not a prerequisite for merging small-sized THP, but given Huang Ying had provided some review feedback, I wanted to progress it. All the actual prerequisites are either complete or being worked on by others.
Quoted 4 lines
Hi Ryan,
this is quite important to a phone and a must-have component, so is large-folio swapin, as i explained to you in another email.
Yes understood; the "prerequisites" are just the things that must be merged *before* small-sized THP to ensure we don't regress existing behaviour or to ensure that small-size THP is correct/robust when enabled. Performance improvements can be merged after the initial small-sized series.
Quoted 3 lines
Luckily, we are having Chuanhua Han(Cc-ed) to prepare a patchset of largefolio swapin on top of your this patchset, probably a port and cleanup of our do_swap_page[1] againest yours.
That's great to hear - welcome aboard, Chuanhua Han! Feel free to reach out if you have questions.
I would guess that any large swap-in changes would be independent of this swap-out patch though? Wouldn't you just be looking for contiguous swap entries in the page table to determine a suitable folio order, then swap-in each of those entries into the folio? And if they happen to have contiguous swap offsets (enabled by this swap-out series) then you potentially get a batched disk access benefit.
That's just a guess though, perhaps you can describe your proposed approach?
Quoted 5 lines
Another concern is that swapslots can be fragmented, if we place small/large folios in a swap device, since large folios always require contiguous swapslot, we can result in failure of getting slots even we still have many free slots which are not contiguous.
This series tries to mitigate that problem by reserving a swap cluster per order. That works well until we run out of swap clusters; a cluster can't be freed until all contained swap entries are swapped back in and deallocated.
But I think we should start with the simple approach first, and only solve the problems as they arise through real testing.
To avoid this, [2] dynamic hugepage solution have two swap devices,
Quoted 8 lines
one for basepage, the other one for CONTPTE. we have modified the priority-based selection of swap devices to choose swap devices based on small/large folios. i realize this approache is super ugly and might be very hard to find a way to upstream though, it seems not universal especially if you are a linux server (-_-)
two devices are not a nice approach though it works well for a real product, we might still need some decent way to address this problem while the problem is for sure not a stopper of your patchset.
I guess that approach works for your case because A) you only have 2 sizes, and
B) your swap device is zRAM, which dynamically allocate RAM as it needs it.
The upstream small-sized THP solution can support multiple sizes, so you would need a swap device per size (I think 13 is the limit at the moment - PMD size for 64K base page). And if your swap device is a physical block device, you can't dynamically parition it the way you can with zRAM. Nether of those things scale particularly well IMHO.
Quoted 24 lines
[1] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/memory.c#L4648
[2] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/swapfile.c#L1129Quoted 12 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Ryan Roberts (4):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap: Remove structpercpu_cluster
mm: swap: Simplify ssd behavior when scanner steals entry
mm: swap: Swap-out small-sized THP without splittingQuoted 5 lines
include/linux/swap.h | 31 +++--- mm/huge_memory.c | 3 - mm/swapfile.c | 232 ++++++++++++++++++++++++------------------- mm/vmscan.c | 10 +- 4 files changed, 149 insertions(+), 127 deletions(-)Thanks
Barry
Quoted 64 lines · Ryan Roberts
Quoted 62 lines · Barry Song
Quoted 52 lines
Hi All,
This is v3 of a series to add support for swapping out small-sized THP without needing to first split the large folio via
__split_huge_page(). It closely follows the approach already used by PMD-sized THP."Small-sized THP" is an upcoming feature that enables performance improvements by allocating large folios for anonymous memory, where the large folio size is smaller than the traditional PMD-size. See [3].
In some circumstances I've observed a performance regression (see patch 2 for details), and this series is an attempt to fix the regression in advance of merging small-sized THP support.
I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is probably sufficient.
The series applies against mm-unstable (
1a3c85fa684a)Changes since v2 [2]
====================- Reuse
scan_swap_map_try_ssd_cluster()between order-0 and order > 0 allocation. This required some refactoring to make everything work nicely (new patches 2 and 3).
- Fix bug wherenr_swap_pageswould say there are pages available but the scanner would not be able to allocate them because they were reserved for the per-cpu allocator. We now allow stealing of order-0 entries from the high order per-cpu clusters (in addition to exisiting stealing from order-0 per-cpu clusters).Thanks to Huang, Ying for the review feedback and suggestions!
Changes since v1 [1]
====================- patch 1:
- Usecluster_set_count()instead ofcluster_set_count_flag()inswap_alloc_cluster()since we no longer have any flag to set. I was unable to killcluster_set_count_flag()as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array intopercpu_clusterto make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated becausePMD_ORDERis not compile-time constant for powerpc (fixes build error).Quoted 7 lines
Thanks,
Ryan
P.S. I know we agreed this is not a prerequisite for merging small-sized THP, but given Huang Ying had provided some review feedback, I wanted to progress it. All the actual prerequisites are either complete or being worked on by others.Hi Ryan,
Quoted 7 lines
Quoted 2 lines
this is quite important to a phone and a must-have component, so is large-folio swapin, as i explained to you in another email.
Yes understood; the "prerequisites" are just the things that must be merged *before* small-sized THP to ensure we don't regress existing behaviour or to ensure that small-size THP is correct/robust when enabled. Performance improvements can be merged after the initial small-sized series.
I completely agree. I didn't mean small-THP swap out as a whole should be a prerequisite for small-THP initial patchset, just describing how important it is to a phone :-)
And actually we have done much further than this on phones by optimizing zsmalloc/zram and allow a large folio compressed and decompressed as a whole, we have seen compressing/decompressing a whole large folio can significantly improve compression ratio and decrease CPU consumption.
so that means large folios can not only save memory but also decrease CPU consumption.
Quoted 8 lines
Quoted 4 lines
Luckily, we are having Chuanhua Han(Cc-ed) to prepare a patchset of largefolio swapin on top of your this patchset, probably a port and cleanup of our do_swap_page[1] againest yours.
That's great to hear - welcome aboard, Chuanhua Han! Feel free to reach out if you have questions.
Quoted 6 lines
I would guess that any large swap-in changes would be independent of this swap-out patch though? Wouldn't you just be looking for contiguous swap entries in the page table to determine a suitable folio order, then swap-in each of those entries into the folio? And if they happen to have contiguous swap offsets (enabled by this swap-out series) then you potentially get a batched disk access benefit.
I agree. Maybe we still need to check if the number of contiguous swap entries is one of those supported large folio sizes?
Quoted 2 lines
That's just a guess though, perhaps you can describe your proposed approach?
we have an ugly hack if we are swapping in from the dedicated zRAM for large folios,
we assume we have a chance to swapin as a whole, but we do also handle corner cases in which some entries might have been zap_pte_range()-ed.
My current proposal is as below,
A1. we get the number of contiguous swap entries with PTL and find it is a valid large folio size
A2. we allocate large folio without PTL
A3. after getting PTL again, we re-check PTEs if the situation in A1 have been changed,
if no other threads change those PTEs, we set_ptes and finish the swap-in
but we have a chance to fail in A2, so in this case we still need to fall back to basepage.
considering the MTE thread[1] I am handling, and MTE tag life cycle is the same with swap
entry life cycle. it seems we will still need a page-level arch_swap_restore even after
we support large folio swap-in for the below two reasons
1. contiguous PTEs might be partially dropped by madvise(DONTNEED) etc
2. we can still fall back to basepage for swap-in if we fail to get large folio even PTEs are all
contiguous swap entries
Of course, if we succeed in setting all PTEs for a large folio in A3, we can have
a folio-level arch_swap_restore.
To me, an universal folio-level arch_swap_restore seems not sensible to handle all kinds of
complex cases.
[1] [RFC V3 PATCH] arm64: mm: swap: save and restore mte tags for large folios
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 7 lines
Quoted 6 lines
Another concern is that swapslots can be fragmented, if we place small/large folios in a swap device, since large folios always require contiguous swapslot, we can result in failure of getting slots even we still have many free slots which are not contiguous.
Quoted 6 lines
This series tries to mitigate that problem by reserving a swap cluster per order. That works well until we run out of swap clusters; a cluster can't be freed until all contained swap entries are swapped back in and deallocated.
But I think we should start with the simple approach first, and only solve the problems as they arise through real testing.
I agree.
Quoted 11 lines
To avoid this, [2] dynamic hugepage solution have two swap devices,
Quoted 8 lines
one for basepage, the other one for CONTPTE. we have modified the priority-based selection of swap devices to choose swap devices based on small/large folios. i realize this approache is super ugly and might be very hard to find a way to upstream though, it seems not universal especially if you are a linux server (-_-)
two devices are not a nice approach though it works well for a real product, we might still need some decent way to address this problem while the problem is for sure not a stopper of your patchset.
Quoted 8 lines
I guess that approach works for your case because A) you only have 2 sizes, and
B) your swap device is zRAM, which dynamically allocate RAM as it needs it.The upstream small-sized THP solution can support multiple sizes, so you would need a swap device per size (I think 13 is the limit at the moment - PMD size for 64K base page). And if your swap device is a physical block device, you can't dynamically parition it the way you can with zRAM. Nether of those things scale particularly well IMHO.
right.
Quoted 17 lines
Quoted 17 lines
[1] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/memory.c#L4648
[2] https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/blob/oneplus/sm8550_u_14.0.0_oneplus11/mm/swapfile.c#L1129Quoted 12 lines
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/Ryan Roberts (4):
mm: swap: RemoveCLUSTER_FLAG_HUGEfrom swap_cluster_info:flags
mm: swap: Remove structpercpu_cluster
mm: swap: Simplify ssd behavior when scanner steals entry
mm: swap: Swap-out small-sized THP without splitting
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
include/linux/swap.h | 31 +++--- mm/huge_memory.c | 3 - mm/swapfile.c | 232 ++++++++++++++++++++++++------------------- mm/vmscan.c | 10 +- 4 files changed, 149 insertions(+), 127 deletions(-)
Thanks
Barry
cover mm: support large folios swap-in
In reply to Ryan RobertsOn an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
In theory, we don't need to rely on Ryan's swap out patchset[1]. That is to say, before swap-out, if some memory were normal pages, but when swapping in, we can also swap-in them as large folios. But this might require I/O happen at some random places in swap devices. So we limit the large folios swap-in to those areas which were large folios before swapping-out, aka, swaps are also contiguous in hardware. On the other hand, in OPPO's product, we've deployed anon large folios on millions of phones[2]. we enhanced zsmalloc and zRAM to compress and decompress large folios as a whole, which help improve compression ratio and decrease CPU consumption significantly. In zsmalloc and zRAM we can save large objects whose original size are 64KiB for example. So it is also a better choice for us to only swap-in large folios for those compressed large objects as a large folio can be decompressed all together.
Note I am moving my previous "arm64: mm: swap: support THP_SWAP on hardware with MTE" to this series as it might help review.
[1] [PATCH v3 0/4] Swap-out small-sized THP without splitting
https://lore.kernel.org/linux-mm/[email protected]/
[2] OnePlusOSS / android_kernel_oneplus_sm8550
https://github.com/OnePlusOSS/android_kernel_oneplus_sm8550/tree/oneplus/sm8550_u_14.0.0_oneplus11
Barry Song (2):
arm64: mm: swap: support THP_SWAP on hardware with MTE
mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Chuanhua Han (4):
mm: swap: introduce swap_nr_free() for batched swap_free()
mm: swap: make should_try_to_free_swap() support large-folio
mm: support large folios swapin as a whole
mm: madvise: don't split mTHP for MADV_PAGEOUT
arch/arm64/include/asm/pgtable.h | 21 ++---- arch/arm64/mm/mteswap.c | 42 ++++++++++++ include/asm-generic/tlb.h | 10 +++ include/linux/huge_mm.h | 12 ---- include/linux/pgtable.h | 62 ++++++++++++++++- include/linux/swap.h | 6 ++ mm/madvise.c | 48 ++++++++++++++ mm/memory.c | 110 ++++++++++++++++++++++++++----- mm/page_io.c | 2 +- mm/rmap.c | 5 +- mm/swap_slots.c | 2 +- mm/swapfile.c | 29 ++++++++ 12 files changed, 301 insertions(+), 48 deletions(-)
Signature · 1 line
2.34.1
RFC PATCH 1/6 arm64: mm: swap: support THP_SWAP on hardware with MTE
From: Barry Song <[email protected]>
Commit d0637c505f8a1 ("arm64: enable THP_SWAP for arm64") brings up THP_SWAP on ARM64, but it doesn't enable THP_SWP on hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page.
The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and THP_SWAP becomes more and more important.
This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more.
arch_prepare_to_swap() should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio.
As now we are restoring tags based-on folio, in arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache in do_swap_page(). In case a large folio has nr pages, do_swap_page() will only set the PTE of the particular page which is causing the page fault.
Thus do_swap_page() runs nr times, and each time, arch_swap_restore() will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2).
Once we support mapping large folios in do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as,
1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated;
2. users might use mprotect() to set MTEs on a part of a large folio.
arch_thp_swp_supported() is dropped since ARM64 MTE was the only one who needed it.
--- arch/arm64/include/asm/pgtable.h | 21 +++------------- arch/arm64/mm/mteswap.c | 42 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 49 insertions(+), 32 deletions(-)
arch/arm64/include/asm/pgtable.h +4 -17
@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page
@@ -1042,12 +1036,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap +extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)
@@ -1063,11 +1053,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore +extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */
arch/arm64/mm/mteswap.c +42 -0
51 lines not on this page · raw message
include/linux/huge_mm.h +0 -12
@@ -530,16 +530,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */
include/linux/pgtable.h +1 -1
@@ -925,7 +925,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }
mm/page_io.c +1 -1
@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);
mm/swap_slots.c +1 -1
@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; }
Signature · 1 line
2.34.1
Re: [PATCH RFC 1/6] arm64: mm: swap: support THP_SWAP on hardware with MTE
Quoted 68 lines · Barry Song
From: Barry Song <[email protected]>
Commit
d0637c505f8a1("arm64: enableTHP_SWAPfor arm64") brings upTHP_SWAPon ARM64, but it doesn't enableTHP_SWPon hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page.The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and
THP_SWAPbecomes more and more important.This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more.
arch_prepare_to_swap()should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio.As now we are restoring tags based-on folio, in
arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache indo_swap_page(). In case a large folio has nr pages,do_swap_page()will only set the PTE of the particular page which is causing the page fault.
Thusdo_swap_page()runs nr times, and each time,arch_swap_restore()will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2).Once we support mapping large folios in
do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as,
1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated;
2. users might usemprotect()to set MTEs on a part of a large folio.
arch_thp_swp_supported()is dropped since ARM64 MTE was the only one who needed it.Reviewed-by Steven Price <[email protected]>Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------- arch/arm64/mm/mteswap.c | 42 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 49 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +0
-6@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page
arch/arm64/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 1039
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +1
-6@@ -1042,12 +1036,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swap
This seems a noop, define "arch_prepare_to_swap" back to itself. What am I missing?
I see. Answer my own question, I guess you want to allow someone to overwrite the arch_prepare_to_swap.
Wouldn't testing against __HAVE_ARCH_PREPARE_TO_SWAP enough to support that?
Maybe I need to understand better how you want others to extend this code to make suggestions.
As it is, this looks strange.
Quoted 5 lines
arch/arm64/include/asm/pgtable.h +1
-0+extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)
arch/arm64/include/asm/pgtable.h static inline void arch_swap_invalidate_area(int type) · line 1056
Quoted 9 lines
arch/arm64/include/asm/pgtable.h +1
-5@@ -1063,11 +1053,8 @@ static inline void arch_swap_invalidate_area(int type) }#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restore
Same here.
Quoted 23 lines
arch/arm64/include/asm/pgtable.h +1
-0+extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +7
-0@@ -68,6 +68,13 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); + +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 10 lines
arch/arm64/mm/mteswap.c +7
-0@@ -83,3 +90,38 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) {
Very minor nitpick.
You can do
if (!system_supports_mte())
return 0;
Here and the for loop would have less indent. The function looks flatter.
Quoted 9 lines
arch/arm64/mm/mteswap.c +9
-0+long nr =folio_nr_pages (folio); + +for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ;
arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 118
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+ +out: +while (i--) +__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) {
Same here.
Looks good otherwise. None of the nitpicks are deal breakers.
Chris
Quoted 73 lines
arch/arm64/mm/mteswap.c +9
-0+long i, nr =folio_nr_pages (folio); + + entry.val -=swp_offset (entry) & (nr -1 ); +for (i =0 ; i < nr; i++) { +mte_restore_tags (entry,folio_page (folio, i)); + entry.val++; + } + } +}include/linux/huge_mm.h +0
-12@@ -530,16 +530,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -925,7 +925,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +1
-2@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; } --2.34.1
Re: [PATCH RFC 1/6] arm64: mm: swap: support THP_SWAP on hardware with MTE
Hi Chris,
Thanks for reviewing. sorry for the late reply as I’ve been getting a lot to do recently.
Quoted 84 lines · Chris Li
Quoted 68 lines · Barry Song
From: Barry Song <[email protected]>
Commit
d0637c505f8a1("arm64: enableTHP_SWAPfor arm64") brings upTHP_SWAPon ARM64, but it doesn't enableTHP_SWPon hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page.The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and
THP_SWAPbecomes more and more important.This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more.
arch_prepare_to_swap()should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio.As now we are restoring tags based-on folio, in
arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache indo_swap_page(). In case a large folio has nr pages,do_swap_page()will only set the PTE of the particular page which is causing the page fault.
Thusdo_swap_page()runs nr times, and each time,arch_swap_restore()will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2).Once we support mapping large folios in
do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as,
1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated;
2. users might usemprotect()to set MTEs on a part of a large folio.
arch_thp_swp_supported()is dropped since ARM64 MTE was the only one who needed it.Reviewed-by Steven Price <[email protected]>Signed-off-by Barry Song <[email protected]>--- arch/arm64/include/asm/pgtable.h | 21 +++------------- arch/arm64/mm/mteswap.c | 42 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 49 insertions(+), 32 deletions(-)arch/arm64/include/asm/pgtable.h +0
-6@@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported (void ) -{ -return !system_supports_mte (); -} -#define arch_thp_swp_supported arch_thp_swp_supported -/* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page arch/arm64/include/asm/pgtable.h static inline pmd_t pmdp_establish(struct vm_area_struct *vma, · line 1039
Quoted 10 lines
arch/arm64/include/asm/pgtable.h +1
-6@@ -1042,12 +1036,8 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,#ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) -{ -if (system_supports_mte ()) -return mte_save_tags (page); -return 0 ; -} +#define arch_prepare_to_swap arch_prepare_to_swapThis seems a noop, define "
arch_prepare_to_swap" back to itself. What am I missing?
Quoted 3 lines
I see. Answer my own question, I guess you want to allow someone to overwrite the
arch_prepare_to_swap.
Wouldn't testing against__HAVE_ARCH_PREPARE_TO_SWAPenough to support that?
you are right. i was blindly copying my previous code
static inline bool arch_thp_swp_supported(void)
{
return !system_supports_mte();
}#define arch_thp_swp_supported arch_thp_swp_supported
for arch_thp_swp_supported, there isn't a similar MACRO. so we are depending on arch_thp_swp_supported as a MACRO for itself in include/linux/huge_mm.h.
/*
* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to
* limitations in the implementation like arm64 MTE can override this to
* false
*/
#ifndef arch_thp_swp_supported
static inline bool arch_thp_swp_supported(void)
{
return true;
}#endif
Now the case is different, we do have __HAVE_ARCH_PREPARE_TO_SWAP instead.
Quoted 12 lines
Maybe I need to understand better how you want others to extend this code to make suggestions.
As it is, this looks strange.Quoted 7 lines
arch/arm64/include/asm/pgtable.h +1
-0+extern int arch_prepare_to_swap (struct folio *folio);#define __HAVE_ARCH_SWAP_INVALIDATEstatic inline void arch_swap_invalidate_page (int type,pgoff_t offset)@@ -1063,11 +1053,8 @@ static inline void arch_swap_invalidate_area(int type) }
Quoted 9 lines
arch/arm64/include/asm/pgtable.h static inline void arch_swap_invalidate_area(int type) · line 1056
Quoted 7 lines
arch/arm64/include/asm/pgtable.h +1
-5#define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore (swp_entry_t entry,struct folio *folio) -{ -if (system_supports_mte ()) -mte_restore_tags (entry, &folio->page); -} +#define arch_swap_restore arch_swap_restoreSame here.
you are right, again.
Quoted 36 lines
Quoted 24 lines
arch/arm64/include/asm/pgtable.h +1
-0+extern void arch_swap_restore (swp_entry_t entry,struct folio *folio);#endif /* CONFIG_ARM64_MTE */ arch/arm64/mm/mteswap.c +7
-0@@ -68,6 +68,13 @@ void mte_invalidate_tags(int type, pgoff_t offset)mte_free_tag_storage (tags); } +static inline void __mte_invalidate_tags (struct page *page) +{ +swp_entry_t entry =page_swap_entry (page); + +mte_invalidate_tags (swp_type (entry),swp_offset (entry)); +} +void mte_invalidate_tags_area (int type) {swp_entry_t entry =swp_entry (type,0 );arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 99
Quoted 10 lines
arch/arm64/mm/mteswap.c +7
-0@@ -83,3 +90,38 @@ void mte_invalidate_tags_area(int type) }xa_unlock (&mte_pages); } + +int arch_prepare_to_swap (struct folio *folio) +{ +int err; +long i; + +if (system_supports_mte ()) {Very minor nitpick.
Quoted 5 lines
You can do
if (!system_supports_mte())return 0;Here and the for loop would have less indent. The function looks flatter.
I agree.
Quoted 23 lines
Quoted 10 lines
arch/arm64/mm/mteswap.c +9
-0+long nr =folio_nr_pages (folio); + +for (i =0 ; i < nr; i++) { + err =mte_save_tags (folio_page (folio, i)); +if (err) + goto out; + } + } +return 0 ;arch/arm64/mm/mteswap.c void mte_invalidate_tags_area(int type) · line 118
Quoted 10 lines
arch/arm64/mm/mteswap.c +10
-0+ +out: +while (i--) +__mte_invalidate_tags (folio_page (folio, i)); +return err; +} + +void arch_swap_restore (swp_entry_t entry,struct folio *folio) +{ +if (system_supports_mte ()) {Same here.
Quoted 3 lines
Looks good otherwise. None of the nitpicks are deal breakers.
Acked-by Chris Li <[email protected]>
Thanks!
Quoted 67 lines
Chris
Quoted 63 lines
arch/arm64/mm/mteswap.c +9
-0+long i, nr =folio_nr_pages (folio); + + entry.val -=swp_offset (entry) & (nr -1 ); +for (i =0 ; i < nr; i++) { +mte_restore_tags (entry,folio_page (folio, i)); + entry.val++; + } + } +}include/linux/huge_mm.h +0
-12@@ -530,16 +530,4 @@ static inline int split_folio(struct folio *folio) return split_folio_to_list(folio, NULL); } -/* -* archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to -* limitations in the implementation like arm64 MTE can override this to -* false -*/ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported (void ) -{ -return true; -} -#endif -#endif /* _LINUX_HUGE_MM_H */ include/linux/pgtable.h +1
-1@@ -925,7 +925,7 @@ static inline int arch_unmap_one(struct mm_struct *mm,* prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap (struct page *page) +static inline int arch_prepare_to_swap (struct folio *folio) {return 0 ; }mm/page_io.c +1
-1@@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)* Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret =arch_prepare_to_swap (&folio->page); + ret =arch_prepare_to_swap (folio);if (ret) {folio_mark_dirty (folio);folio_unlock (folio);mm/swap_slots.c +0
-0@@ -307,7 +307,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val =0 ;
Quoted 9 lines
Quoted 9 lines
mm/swap_slots.c +1
-2if (folio_test_large (folio)) { -if (IS_ENABLED (CONFIG_THP_SWAP ) &&arch_thp_swp_supported ()) +if (IS_ENABLED (CONFIG_THP_SWAP ))get_swap_pages (1 , &entry,folio_nr_pages (folio)); goto out; } --2.34.1
Best Regards,
Barry
RFC PATCH 2/6 mm: swap: introduce swap_nr_free() for batched swap_free()
In reply to Barry SongFrom: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
--- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+)
include/linux/swap.h +6 -0
@@ -478,6 +478,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_nr_free (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern int free_swap_and_cache (swp_entry_t );int swap_type_of (dev_t device,sector_t offset);
@@ -553,6 +554,11 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_nr_free (swp_entry_t entry,int nr_pages) +{ + +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }
mm/swapfile.c +29 -0
@@ -1335,6 +1335,35 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); } +void swap_nr_free (swp_entry_t entry,int nr_pages) +{ +int i; +struct swap_cluster_info *ci; +struct swap_info_struct *p; +unsigned type =swp_type (entry); +unsigned long offset =swp_offset (entry); + DECLARE_BITMAP(usage, SWAPFILE_CLUSTER) = { 0 }; + +VM_BUG_ON (offset %SWAPFILE_CLUSTER + nr_pages >SWAPFILE_CLUSTER ); + +if (nr_pages ==1 ) { +swap_free (entry); +return ; + } + + p =_swap_info_get (entry); + + ci =lock_cluster (p, offset); +for (i =0 ; i < nr_pages; i++) { +if (__swap_entry_free_locked (p, offset + i,1 )) +__bitmap_set (usage, i,1 ); + } +unlock_cluster (ci); + +for_each_clear_bit (i, usage, nr_pages) +free_swap_slot (swp_entry (type, offset + i)); +} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */
Signature · 1 line
2.34.1
Chris Li
maintainerRe: [PATCH RFC 2/6] mm: swap: introduce swap_nr_free() for batched swap_free()
Quoted 47 lines · Barry Song
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+)include/linux/swap.h +6
-0@@ -478,6 +478,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_nr_free (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern int free_swap_and_cache (swp_entry_t );int swap_type_of (dev_t device,sector_t offset);@@ -553,6 +554,11 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_nr_free (swp_entry_t entry,int nr_pages) +{ + +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }mm/swapfile.c +0
-0@@ -1335,6 +1335,35 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); }
mm/swapfile.c void swap_free(swp_entry_t entry) · line 1347
Quoted 10 lines
mm/swapfile.c +10
-0+void swap_nr_free (swp_entry_t entry,int nr_pages) +{ +int i; +struct swap_cluster_info *ci; +struct swap_info_struct *p; +unsigned type =swp_type (entry); +unsigned long offset =swp_offset (entry); + DECLARE_BITMAP(usage, SWAPFILE_CLUSTER) = { 0 }; + +VM_BUG_ON (offset %SWAPFILE_CLUSTER + nr_pages >SWAPFILE_CLUSTER );
BUG_ON here seems a bit too developer originated. Maybe warn once and roll back to free one by one?
How big is your typical SWAPFILE_CUSTER and nr_pages typically in arm?
I ask this question because nr_ppages > 64, that is a totally different game, we can completely bypass the swap cache slots.
Quoted 7 lines
mm/swapfile.c +7
-0+ +if (nr_pages ==1 ) { +swap_free (entry); +return ; + } + + p =_swap_info_get (entry);
mm/swapfile.c void swap_free(swp_entry_t entry) · line 1364
Quoted 10 lines
mm/swapfile.c +10
-0+ + ci =lock_cluster (p, offset); +for (i =0 ; i < nr_pages; i++) { +if (__swap_entry_free_locked (p, offset + i,1 )) +__bitmap_set (usage, i,1 ); + } +unlock_cluster (ci); + +for_each_clear_bit (i, usage, nr_pages) +free_swap_slot (swp_entry (type, offset + i));
Notice that free_swap_slot() internal has per CPU cache batching as well. Every free_swap_slot will get some per_cpu swap slot cache and cache->lock. There is double batching here.
If the typical batch size here is bigger than 64 entries, we can go directly to batching swap_entry_free and avoid the free_swap_slot() batching altogether. Unlike free_swap_slot_entries(), here swap slots are all from one swap device, there is no need to sort and group the swap slot by swap devices.
Chris
Chris
Quoted 9 lines
mm/swapfile.c +2
-1+} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */ --2.34.1
Re: [PATCH RFC 2/6] mm: swap: introduce swap_nr_free() for batched swap_free()
Hi Chris,
Thanks!
Quoted 49 lines · Chris Li
Quoted 47 lines · Barry Song
From: Chuanhua Han <[email protected]>
While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free.
Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- include/linux/swap.h | 6 ++++++ mm/swapfile.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+)include/linux/swap.h +6
-0@@ -478,6 +478,7 @@ extern void swap_shmem_alloc(swp_entry_t);extern int swap_duplicate (swp_entry_t );extern int swapcache_prepare (swp_entry_t );extern void swap_free (swp_entry_t ); +extern void swap_nr_free (swp_entry_t entry,int nr_pages);extern void swapcache_free_entries (swp_entry_t *entries,int n);extern int free_swap_and_cache (swp_entry_t );int swap_type_of (dev_t device,sector_t offset);@@ -553,6 +554,11 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_nr_free (swp_entry_t entry,int nr_pages) +{ + +} +static inline void put_swap_folio (struct folio *folio,swp_entry_t swp) { }mm/swapfile.c +0
-0@@ -1335,6 +1335,35 @@ void swap_free(swp_entry_t entry)__swap_entry_free (p, entry); }
Quoted 13 lines
mm/swapfile.c void swap_free(swp_entry_t entry) · line 1347
Quoted 10 lines
mm/swapfile.c +10
-0+void swap_nr_free (swp_entry_t entry,int nr_pages) +{ +int i; +struct swap_cluster_info *ci; +struct swap_info_struct *p; +unsigned type =swp_type (entry); +unsigned long offset =swp_offset (entry); + DECLARE_BITMAP(usage, SWAPFILE_CLUSTER) = { 0 }; + +VM_BUG_ON (offset %SWAPFILE_CLUSTER + nr_pages >SWAPFILE_CLUSTER );
BUG_ONhere seems a bit too developer originated. Maybe warn once and roll back to free one by one?
The function is used only for the case we are quite sure we are freeing some contiguous swap entries within a cluster. if it is not the case, we will need an array of entries[]. will people be more comfortable to have a WARN_ON instead? but the problem is if that really happens, it is a bug, WARN isn't enough.
Quoted 2 lines
How big is your typical
SWAPFILE_CUSTERandnr_pagestypically in arm?
My case is SWAPFILE_CLUSTER = HPAGE_PMD_NR = 2MB/4KB = 512.
Quoted 4 lines
I ask this question because
nr_ppages> 64, that is a totally different game, we can completely bypass the swap cache slots.
I agree we have a chance to bypass slot cache if nr_pages is bigger than SWAP_SLOTS_CACHE_SIZE. on the other hand, even when nr_pages < 64, we still have a good chance to optimize free_swap_slot() by batching as there are many spin_lock and sort() for each single entry.
Quoted 18 lines
Quoted 7 lines
mm/swapfile.c +7
-0+ +if (nr_pages ==1 ) { +swap_free (entry); +return ; + } + + p =_swap_info_get (entry);mm/swapfile.c void swap_free(swp_entry_t entry) · line 1364
Quoted 10 lines
mm/swapfile.c +10
-0+ + ci =lock_cluster (p, offset); +for (i =0 ; i < nr_pages; i++) { +if (__swap_entry_free_locked (p, offset + i,1 )) +__bitmap_set (usage, i,1 ); + } +unlock_cluster (ci); + +for_each_clear_bit (i, usage, nr_pages) +free_swap_slot (swp_entry (type, offset + i));
Quoted 8 lines
Notice that
free_swap_slot()internal has per CPU cache batching as well. Everyfree_swap_slotwill get someper_cpuswap slot cache andcache->lock. There is double batching here.
If the typical batch size here is bigger than 64 entries, we can go directly to batchingswap_entry_freeand avoid thefree_swap_slot()batching altogether. Unlikefree_swap_slot_entries(), here swap slots are all from one swap device, there is no need to sort and group the swap slot by swap devices.
I agree. you are completely right!
However, to make the patchset smaller at the beginning, I prefer these optimizations to be deferred as a separate patchset after this one.
Quoted 6 lines
Chris
Chris
Quoted 7 lines
Quoted 7 lines
mm/swapfile.c +2
-1+} +/* * Called after dropping swapcache to decrease refcnt to swap entries. */ --2.34.1
Thanks
Barry
RFC PATCH 3/6 mm: swap: make should_try_to_free_swap() support large-folio
In reply to Barry SongFrom: Chuanhua Han <[email protected]>
should_try_to_free_swap() works with an assumption that swap-in is always done at normal page granularity, aka, folio_nr_pages = 1. To support large folio swap-in, this patch removes the assumption.
--- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/memory.c +1 -1
@@ -3714,7 +3714,7 @@ static inline bool should_try_to_free_swap(struct folio *folio,* reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && -folio_ref_count (folio) ==2 ; +folio_ref_count (folio) == (1 +folio_nr_pages (folio)); }static vm_fault_t pte_marker_clear (struct vm_fault *vmf)
Signature · 1 line
2.34.1
Re: [PATCH RFC 3/6] mm: swap: make should_try_to_free_swap() support large-folio
Chris
Quoted 31 lines · Barry Song
From: Chuanhua Han <[email protected]>
should_try_to_free_swap()works with an assumption that swap-in is always done at normal page granularity, aka,folio_nr_pages= 1. To support large folio swap-in, this patch removes the assumption.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/memory.c +1
-2@@ -3714,7 +3714,7 @@ static inline bool should_try_to_free_swap(struct folio *folio,* reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags &FAULT_FLAG_WRITE ) && !folio_test_ksm (folio) && -folio_ref_count (folio) ==2 ; +folio_ref_count (folio) == (1 +folio_nr_pages (folio)); }static vm_fault_t pte_marker_clear (struct vm_fault *vmf) --2.34.1
RFC PATCH 4/6 mm: support large folios swapin as a whole
In reply to Barry SongFrom: Chuanhua Han <[email protected]>
On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole.
On the other hand, the current implementation only covers the SWAP_SYCHRONOUS case. It doesn't support swapin_readahead as large folios yet.
Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in arch_swap_restore() while supporting MTE restore for folios rather than page.
--- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-)
mm/memory.c +94 -14
@@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map);static vm_fault_t do_fault (struct vm_fault *vmf);static vm_fault_t do_anonymous_page (struct vm_fault *vmf);static bool vmf_pte_changed (struct vm_fault *vmf); +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int ));/* * Return true if the original pte was a uffd-wp pte marker (so the pte was
@@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } +static bool pte_range_swap (pte_t *pte,int nr_pages) +{ +int i; +swp_entry_t entry; +unsigned type; +pgoff_t start_offset; + + entry =pte_to_swp_entry (ptep_get_lockless (pte)); +if (non_swap_entry (entry)) +return false; + start_offset =swp_offset (entry); +if (start_offset % nr_pages) +return false; + + type =swp_type (entry); +for (i =1 ; i < nr_pages; i++) { + entry =pte_to_swp_entry (ptep_get_lockless (pte + i)); +if (non_swap_entry (entry)) +return false; +if (swp_offset (entry) != start_offset + i) +return false; +if (swp_type (entry) != type) +return false; + } + +return true; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
@@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long start_address; +pte_t *start_pte;if (!pte_unmap_same (vmf)) goto out;
@@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , - vma, vmf->address, false); + folio =alloc_anon_folio (vmf, pte_range_swap); page = &folio->page;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio); +if (folio_test_large (folio)) { +unsigned long start_offset; + + nr_pages =folio_nr_pages (folio); + start_offset =swp_offset (entry) & ~(nr_pages -1 ); + entry =swp_entry (swp_type (entry), start_offset); + } +if (mem_cgroup_swapin_charge_folio (folio, vma->vm_mm,GFP_KERNEL , entry)) {
@@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address; + start_pte = vmf->pte; +if (folio_test_large (folio)) { +unsigned long nr =folio_nr_pages (folio); +unsigned long addr =ALIGN_DOWN (vmf->address, nr *PAGE_SIZE ); +pte_t *pte_t = vmf->pte - (vmf->address - addr) /PAGE_SIZE ; + +/* +* case 1: we are allocating large_folio, try to map it as a whole +* iff the swap entries are still entirely mapped; +* case 2: we hit a large folio in swapcache, and all swap entries +* are still entirely mapped, try to map a large folio as a whole. +* otherwise, map only the faulting page within the large folio +* which is swapcache +*/ +if (pte_range_swap (pte_t, nr)) { + start_address = addr; + start_pte = pte_t; +if (unlikely (folio == swapcache)) { +/* +* the below has been done before swap_read_folio() +* for case 1 +*/ + nr_pages = nr; + entry =pte_to_swp_entry (ptep_get (start_pte)); + page = &folio->page; + } + }else if (nr_pages >1 ) {/* ptes have changed for case 1 */ + goto out_nomap; + } + } +if (unlikely (!vmf->pte || !pte_same (ptep_get (vmf->pte), vmf->orig_pte))) goto out_nomap;
@@ -4047,12 +4120,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_nr_free (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); + pte =mk_pte (page, vma->vm_page_prot);/*
@@ -4062,14 +4137,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive ||folio_ref_count (folio) == nr_pages)) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte))
@@ -4081,14 +4156,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)folio_add_new_anon_rmap (folio, vma, vmf->address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, start_address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +set_ptes (vma->vm_mm, start_address, start_pte, pte, nr_pages); + +arch_do_swap_page (vma->vm_mm, vma, start_address, pte, vmf->orig_pte);folio_unlock (folio);if (folio != swapcache && swapcache) {
@@ -4105,6 +4181,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }if (vmf->flags &FAULT_FLAG_WRITE ) { +if (folio_test_large (folio) && nr_pages >1 ) + vmf->orig_pte =ptep_get (vmf->pte); + ret |=do_wp_page (vmf);if (ret &VM_FAULT_ERROR ) ret &=VM_FAULT_ERROR ;
@@ -4112,7 +4191,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, start_address, start_pte, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);
@@ -4148,7 +4227,8 @@ static bool pte_range_none(pte_t *pte, int nr_pages)return true; } -static struct folio *alloc_anon_folio (struct vm_fault *vmf) +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int )) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct vm_area_struct *vma = vmf->vma;
@@ -4190,7 +4270,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) order =highest_order (orders);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); -if (pte_range_none (pte +pte_index (addr),1 << order)) +if (pte_range_check (pte +pte_index (addr),1 << order))break ; order =next_order (&orders, order); }
@@ -4269,7 +4349,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (unlikely (anon_vma_prepare (vma))) goto oom;/* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ - folio =alloc_anon_folio (vmf); + folio =alloc_anon_folio (vmf, pte_range_none);if (IS_ERR (folio))return 0 ;if (!folio)
Signature · 1 line
2.34.1
Chris Li
maintainerRe: [PATCH RFC 4/6] mm: support large folios swapin as a whole
Quoted 35 lines · Barry Song
From: Chuanhua Han <[email protected]>
On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole.
On the other hand, the current implementation only covers the
SWAP_SYCHRONOUScase. It doesn't supportswapin_readaheadas large folios yet.Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in
arch_swap_restore()while supporting MTE restore for folios rather than page.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-)mm/memory.c +0
-0
mm/memory.c EXPORT_SYMBOL(mem_map); · line 111
Quoted 5 lines
mm/memory.c +2
-0@@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map);static vm_fault_t do_fault (struct vm_fault *vmf);static vm_fault_t do_anonymous_page (struct vm_fault *vmf);static bool vmf_pte_changed (struct vm_fault *vmf); +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int ));
Instead of returning "bool", the pte_range_check() can return the start of the swap entry of the large folio.
That will save some of the later code needed to get the start of the large folio.
mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3789
Quoted 8 lines
mm/memory.c +1
-0/* * Return true if the original pte was a uffd-wp pte marker (so the pte was @@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } +static bool pte_range_swap (pte_t *pte,int nr_pages)
This function name seems to suggest it will perform the range swap. That is not what it is doing.
Suggest change to some other name reflecting that it is only a condition test without actual swap action.
I am not very good at naming functions. Just think it out loud: e.g. pte_range_swap_check, pte_test_range_swap. You can come up with something better.
mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3801
Quoted 12 lines
mm/memory.c +12
-0+{ +int i; +swp_entry_t entry; +unsigned type; +pgoff_t start_offset; + + entry =pte_to_swp_entry (ptep_get_lockless (pte)); +if (non_swap_entry (entry)) +return false; + start_offset =swp_offset (entry); +if (start_offset % nr_pages) +return false;
This suggests the pte argument needs to point to the beginning of the large folio equivalent of swap entry(not sure what to call it. Let me call it "large folio swap" here.).
We might want to unify the terms for that.
Any way, might want to document this requirement, otherwise the caller might consider passing the current pte that generates the fault. From the function name it is not obvious which pte should pass it.
mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3804
Quoted 3 lines
mm/memory.c +3
-0+ + type =swp_type (entry); +for (i =1 ; i < nr_pages; i++) {
You might want to test the last page backwards, because if the entry is not the large folio swap, most likely it will have the last entry invalid. Some of the beginning swap entries might match due to batch allocation etc. The SSD likes to group the nearby swap entry write out together on the disk.
Quoted 27 lines
mm/memory.c +12
-0+ entry =pte_to_swp_entry (ptep_get_lockless (pte + i)); +if (non_swap_entry (entry)) +return false; +if (swp_offset (entry) != start_offset + i) +return false; +if (swp_type (entry) != type) +return false; + } + +return true; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
mm/memory.c +3
-0@@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long start_address; +pte_t *start_pte;if (!pte_unmap_same (vmf)) goto out;
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3904
Quoted 6 lines
mm/memory.c +1
-2@@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , - vma, vmf->address, false); + folio =alloc_anon_folio (vmf, pte_range_swap);
This function can call pte_range_swap() twice(), one here, another one in folio_test_large().
Consider caching the result so it does not need to walk the pte range swap twice.
I think alloc_anon_folio should either be told what is the size(prefered) or just figure out the right size. I don't think it needs to pass in the checking function as function callbacks. There are two call sites of alloc_anon_folio, they are all within this function. The call back seems a bit overkill here. Also duplicate the range swap walk.
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3914
Quoted 10 lines
mm/memory.c +5
-0page = &folio->page;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio); +if (folio_test_large (folio)) { +unsigned long start_offset; + + nr_pages =folio_nr_pages (folio); + start_offset =swp_offset (entry) & ~(nr_pages -1 );
Here is the first place place we roll up the start offset with folio size
Quoted 7 lines
mm/memory.c +3
-0+ entry =swp_entry (swp_type (entry), start_offset); + } +if (mem_cgroup_swapin_charge_folio (folio, vma->vm_mm,GFP_KERNEL , entry)) {
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4029
Quoted 10 lines
mm/memory.c +7
-0@@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address; + start_pte = vmf->pte; +if (folio_test_large (folio)) { +unsigned long nr =folio_nr_pages (folio); +unsigned long addr =ALIGN_DOWN (vmf->address, nr *PAGE_SIZE ); +pte_t *pte_t = vmf->pte - (vmf->address - addr) /PAGE_SIZE ;
Here is the second place we roll up the folio size. Maybe we can cache results and avoid repetition?
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4038
Quoted 9 lines
mm/memory.c +9
-0+ +/* +* case 1: we are allocating large_folio, try to map it as a whole +* iff the swap entries are still entirely mapped; +* case 2: we hit a large folio in swapcache, and all swap entries +* are still entirely mapped, try to map a large folio as a whole. +* otherwise, map only the faulting page within the large folio +* which is swapcache +*/
One question I have in mind is that the swap device is locked. We can't change the swap slot allocations.
It does not stop the pte entry getting changed right? Then we can have someone in the user pace racing to change the PTE vs we checking the pte there.
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4039
Quoted 1 line
mm/memory.c +1
-0+if (pte_range_swap (pte_t, nr)) {
After this pte_range_swap() check, some of the PTE entries get changed and now we don't have the full large page swap any more?
At least I can't conclude this possibility can't happen yet, please enlighten me.
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4048
Quoted 9 lines
mm/memory.c +9
-0+ start_address = addr; + start_pte = pte_t; +if (unlikely (folio == swapcache)) { +/* +* the below has been done before swap_read_folio() +* for case 1 +*/ + nr_pages = nr; + entry =pte_to_swp_entry (ptep_get (start_pte));
If we make pte_range_swap() return the entry, we can avoid refetching the swap entry here.
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4055
Quoted 7 lines
mm/memory.c +7
-0+ page = &folio->page; + } + }else if (nr_pages >1 ) {/* ptes have changed for case 1 */ + goto out_nomap; + } + } +
I rewrote the above to make the code indentation matching the execution flow. I did not add any functional change. Just rearrange the code to be a bit more streamlined. Get rid of the "else if goto".
if (!pte_range_swap(pte_t, nr)) {
if (nr_pages > 1) /* ptes have changed for case 1 */
goto out_nomap;
goto check_pte;
}
start_address = addr;
start_pte = pte_t;
if (unlikely(folio == swapcache)) {
/*
* the below has been done before swap_read_folio()
* for case 1
*/
nr_pages = nr;
entry = pte_to_swp_entry(ptep_get(start_pte));
page = &folio->page;
}
}check_pte:
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf)
Quoted 87 lines
mm/memory.c +17
-10if (unlikely (!vmf->pte || !pte_same (ptep_get (vmf->pte), vmf->orig_pte))) goto out_nomap;@@ -4047,12 +4120,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_nr_free (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); + pte =mk_pte (page, vma->vm_page_prot);/* @@ -4062,14 +4137,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive ||folio_ref_count (folio) == nr_pages)) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte))@@ -4081,14 +4156,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)folio_add_new_anon_rmap (folio, vma, vmf->address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, start_address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +set_ptes (vma->vm_mm, start_address, start_pte, pte, nr_pages); + +arch_do_swap_page (vma->vm_mm, vma, start_address, pte, vmf->orig_pte);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4105,6 +4181,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }if (vmf->flags &FAULT_FLAG_WRITE ) { +if (folio_test_large (folio) && nr_pages >1 ) + vmf->orig_pte =ptep_get (vmf->pte); + ret |=do_wp_page (vmf);if (ret &VM_FAULT_ERROR ) ret &=VM_FAULT_ERROR ;@@ -4112,7 +4191,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, start_address, start_pte, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);@@ -4148,7 +4227,8 @@ static bool pte_range_none(pte_t *pte, int nr_pages)return true; } -static struct folio *alloc_anon_folio (struct vm_fault *vmf) +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int )) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct vm_area_struct *vma = vmf->vma;@@ -4190,7 +4270,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)
About this patch context we have the following comments in the source code.
/*
* Find the highest order where the aligned range is completely
* pte_none(). Note that all remaining orders will be completely
* pte_none().
*/
mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4273
Quoted 5 lines
mm/memory.c +1
-1order =highest_order (orders);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); -if (pte_range_none (pte +pte_index (addr),1 << order)) +if (pte_range_check (pte +pte_index (addr),1 << order))
Again, I don't think we need to pass in the pte_range_check() as call back functions.
There are only two call sites, all within this file. This will totally invalide the above comments about pte_none(). In the worst case, just make it accept one argument: it is checking swap range or none range or not. Depending on the argument, do check none or swap range. We should make it blend in with alloc_anon_folio better. My gut feeling is that there should be a better way to make the range check blend in with alloc_anon_folio better. e.g. Maybe store some of the large swap context in the vmf and pass to different places etc. I need to spend more time thinking about it to come up with happier solutions.
Chris
Quoted 16 lines
mm/memory.c +1
-2break ; order =next_order (&orders, order); }@@ -4269,7 +4349,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (unlikely (anon_vma_prepare (vma))) goto oom;/* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ - folio =alloc_anon_folio (vmf); + folio =alloc_anon_folio (vmf, pte_range_none);if (IS_ERR (folio))return 0 ;if (!folio) --2.34.1
Re: [PATCH RFC 4/6] mm: support large folios swapin as a whole
Quoted 32 lines · Chris Li
Quoted 30 lines · Barry Song
From: Chuanhua Han <[email protected]>
On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole.
On the other hand, the current implementation only covers the
SWAP_SYCHRONOUScase. It doesn't supportswapin_readaheadas large folios yet.Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in
arch_swap_restore()while supporting MTE restore for folios rather than page.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-)
Quoted 15 lines
mm/memory.c EXPORT_SYMBOL(mem_map); · line 111
Quoted 10 lines
mm/memory.c +2
-0@@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map);static vm_fault_t do_fault (struct vm_fault *vmf);static vm_fault_t do_anonymous_page (struct vm_fault *vmf);static bool vmf_pte_changed (struct vm_fault *vmf); +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int ));Instead of returning "bool", the
pte_range_check()can return the start of the swap entry of the large folio.
That will save some of the later code needed to get the start of the large folio.
I am trying to reuse alloc_anon_folio() for both do_anon_page and do_swap_page. Unfortunately, this func returns a folio, no more place to return a swap entry unless we add a parameter. Getting start swap is quite cheap on the other hand.
Quoted 8 lines
Quoted 8 lines
mm/memory.c +0
-0/* * Return true if the original pte was a uffd-wp pte marker (so the pte was @@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; }
Quoted 9 lines
mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3789
Quoted 1 line
mm/memory.c +1
-0+static bool pte_range_swap (pte_t *pte,int nr_pages)This function name seems to suggest it will perform the range swap. That is not what it is doing.
Suggest change to some other name reflecting that it is only a condition test without actual swap action.
I am not very good at naming functions. Just think it out loud: e.g.pte_range_swap_check,pte_test_range_swap. You can come up with something better.
Ryan has a function named pte_range_none, which is checking the whole range is pte_none. Maybe we can have an is_pte_range_contig_swap which includes both swap and contiguous as we only need contiguous swap entries.
Quoted 15 lines
Quoted 4 lines
mm/memory.c +2
-0+{ +int i;mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3801
Quoted 10 lines
mm/memory.c +10
-0+swp_entry_t entry; +unsigned type; +pgoff_t start_offset; + + entry =pte_to_swp_entry (ptep_get_lockless (pte)); +if (non_swap_entry (entry)) +return false; + start_offset =swp_offset (entry); +if (start_offset % nr_pages) +return false;
Quoted 7 lines
This suggests the pte argument needs to point to the beginning of the large folio equivalent of swap entry(not sure what to call it. Let me call it "large folio swap" here.).
We might want to unify the terms for that.
Any way, might want to document this requirement, otherwise the caller might consider passing the current pte that generates the fault. From the function name it is not obvious which pte should pass it.
ok, Ryan's swap-out will allocate swap entries whose start offset is aligned with nr_pages. will add some doc to describe the first entry.
Quoted 10 lines
mm/memory.c static vm_fault_t handle_pte_marker(struct vm_fault *vmf) · line 3804
Quoted 4 lines
mm/memory.c +3
-0+ + type =swp_type (entry); +for (i =1 ; i < nr_pages; i++) {You might want to test the last page backwards, because if the entry is not the large folio swap, most likely it will have the last entry invalid. Some of the beginning swap entries might match due to batch allocation etc. The SSD likes to group the nearby swap entry write out together on the disk.
I am not sure I got your point. This is checking all pages within the range of a large folio, Ryan's patch allocates swap entries all together as a whole for a large folio while swapping out.
@@ -1073,14 +1133,13 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_size)
spin_unlock(&si->lock);
goto nextsi; }
- if (size == SWAPFILE_CLUSTER) {
- if (si->flags & SWP_BLKDEV)
- n_ret = swap_alloc_cluster(si, swp_entries);
+ if (size > 1) {
+ n_ret = swap_alloc_large(si, swp_entries, size);
} else
n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
n_goal, swp_entries);Quoted 42 lines
Quoted 30 lines
mm/memory.c +12
-0+ entry =pte_to_swp_entry (ptep_get_lockless (pte + i)); +if (non_swap_entry (entry)) +return false; +if (swp_offset (entry) != start_offset + i) +return false; +if (swp_type (entry) != type) +return false; + } + +return true; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked.
mm/memory.c +3
-0@@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long start_address; +pte_t *start_pte;if (!pte_unmap_same (vmf)) goto out;mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3904
Quoted 6 lines
mm/memory.c +1
-2@@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , - vma, vmf->address, false); + folio =alloc_anon_folio (vmf, pte_range_swap);This function can call
pte_range_swap()twice(), one here, another one infolio_test_large().
Consider caching the result so it does not need to walk the pte range swap twice.
Quoted 6 lines
I think
alloc_anon_folioshould either be told what is the size(prefered) or just figure out the right size. I don't think it needs to pass in the checking function as function callbacks. There are two call sites ofalloc_anon_folio, they are all within this function. The call back seems a bit overkill here. Also duplicate the range swap walk.
alloc_anon_folio is reusing the one for do_anon_page. in both cases, scanning PTEs to figure out the proper size is done. The other call site is within do_anonymous_page().
Quoted 25 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 3914
Quoted 11 lines
mm/memory.c +5
-0page = &folio->page;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio); +if (folio_test_large (folio)) { +unsigned long start_offset; + + nr_pages =folio_nr_pages (folio); + start_offset =swp_offset (entry) & ~(nr_pages -1 );Here is the first place place we roll up the start offset with folio size
Quoted 12 lines
mm/memory.c +5
-0+ entry =swp_entry (swp_type (entry), start_offset); + } +if (mem_cgroup_swapin_charge_folio (folio, vma->vm_mm,GFP_KERNEL , entry)) {@@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address;
Quoted 8 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4029
Quoted 5 lines
mm/memory.c +5
-0+ start_pte = vmf->pte; +if (folio_test_large (folio)) { +unsigned long nr =folio_nr_pages (folio); +unsigned long addr =ALIGN_DOWN (vmf->address, nr *PAGE_SIZE ); +pte_t *pte_t = vmf->pte - (vmf->address - addr) /PAGE_SIZE ;Here is the second place we roll up the folio size. Maybe we can cache results and avoid repetition?
We have two paths getting into large folios
1. we allocate a new large folio
2. we find a large folio in swapcache
We have rolled up the folio size for case 1 before, but here we need to take care of case 2 as well. so that is why we need both. let me think if we can have some way to remove some redundant code for case 1.
Quoted 17 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4038
Quoted 10 lines
mm/memory.c +9
-0+ +/* +* case 1: we are allocating large_folio, try to map it as a whole +* iff the swap entries are still entirely mapped; +* case 2: we hit a large folio in swapcache, and all swap entries +* are still entirely mapped, try to map a large folio as a whole. +* otherwise, map only the faulting page within the large folio +* which is swapcache +*/ One question I have in mind is that the swap device is locked. We can't change the swap slot allocations.
It does not stop the pte entry getting changed right? Then we can have someone in the user pace racing to change the PTE vs we checking the pte there.
Quoted 6 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4039
Quoted 1 line
mm/memory.c +1
-0+if (pte_range_swap (pte_t, nr)) {After this
pte_range_swap()check, some of the PTE entries get changed and now we don't have the full large page swap any more?
At least I can't conclude this possibility can't happen yet, please enlighten me.
This check is under PTL. no one else can change it as they have to hold PTL to change pte.
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);Quoted 11 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4048
Quoted 10 lines
mm/memory.c +9
-0+ start_address = addr; + start_pte = pte_t; +if (unlikely (folio == swapcache)) { +/* +* the below has been done before swap_read_folio() +* for case 1 +*/ + nr_pages = nr; + entry =pte_to_swp_entry (ptep_get (start_pte));
Quoted 2 lines
If we make
pte_range_swap()return the entry, we can avoid refetching the swap entry here.
we will have to add a parameter swp_entry_t *first_entry to return the entry. The difficulty is we will have to add this parameter in alloc_anon_folio() as well, that's a bit overkill for that function.
Quoted 21 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4055
Quoted 8 lines
mm/memory.c +7
-0+ page = &folio->page; + } + }else if (nr_pages >1 ) {/* ptes have changed for case 1 */ + goto out_nomap; + } + } +I rewrote the above to make the code indentation matching the execution flow. I did not add any functional change. Just rearrange the code to be a bit more streamlined. Get rid of the "else if goto".
if (!pte_range_swap(pte_t, nr)) { if (nr_pages > 1) /* ptes have changed for case 1 */ goto out_nomap; goto check_pte; } start_address = addr; start_pte = pte_t; if (unlikely(folio == swapcache)) { /*
Quoted 8 lines
* the below has been done before swap_read_folio()
* for case 1
*/
nr_pages = nr;
entry = pte_to_swp_entry(ptep_get(start_pte));
page = &folio->page;
}
}looks good to me.
Quoted 103 lines
check_pte:mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf)
Quoted 87 lines
mm/memory.c +17
-10if (unlikely (!vmf->pte || !pte_same (ptep_get (vmf->pte), vmf->orig_pte))) goto out_nomap;@@ -4047,12 +4120,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_nr_free (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); + pte =mk_pte (page, vma->vm_page_prot);/* @@ -4062,14 +4137,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive ||folio_ref_count (folio) == nr_pages)) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte))@@ -4081,14 +4156,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)folio_add_new_anon_rmap (folio, vma, vmf->address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, start_address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +set_ptes (vma->vm_mm, start_address, start_pte, pte, nr_pages); + +arch_do_swap_page (vma->vm_mm, vma, start_address, pte, vmf->orig_pte);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4105,6 +4181,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }if (vmf->flags &FAULT_FLAG_WRITE ) { +if (folio_test_large (folio) && nr_pages >1 ) + vmf->orig_pte =ptep_get (vmf->pte); + ret |=do_wp_page (vmf);if (ret &VM_FAULT_ERROR ) ret &=VM_FAULT_ERROR ;@@ -4112,7 +4191,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, start_address, start_pte, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);@@ -4148,7 +4227,8 @@ static bool pte_range_none(pte_t *pte, int nr_pages)return true; } -static struct folio *alloc_anon_folio (struct vm_fault *vmf) +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int )) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct vm_area_struct *vma = vmf->vma;@@ -4190,7 +4270,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf)About this patch context we have the following comments in the source code.
/*
* Find the highest order where the aligned range is completely
*pte_none(). Note that all remaining orders will be completely
*pte_none().
*/mm/memory.c static struct folio *alloc_anon_folio(struct vm_fault *vmf) · line 4273
Quoted 5 lines
mm/memory.c +1
-1order =highest_order (orders);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); -if (pte_range_none (pte +pte_index (addr),1 << order)) +if (pte_range_check (pte +pte_index (addr),1 << order))
Quoted 12 lines
Again, I don't think we need to pass in the
pte_range_check()as call back functions.
There are only two call sites, all within this file. This will totally invalide the above comments aboutpte_none(). In the worst case, just make it accept one argument: it is checking swap range or none range or not. Depending on the argument, do check none or swap range. We should make it blend in withalloc_anon_foliobetter. My gut feeling is that there should be a better way to make the range check blend in withalloc_anon_foliobetter. e.g. Maybe store some of the large swap context in the vmf and pass to different places etc. I need to spend more time thinking about it to come up with happier solutions.
could pass a type to hint pte_range_none or pte_range_swap. i'd like to avoid changing any global variable like vmf, as people will have to cross two or more functions to understand what is going on though the second function might be able to use the changed vmf value in the first function. but it really makes the code have more couples.
Quoted 18 lines
Chris
Quoted 13 lines
mm/memory.c +1
-2break ; order =next_order (&orders, order); }@@ -4269,7 +4349,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (unlikely (anon_vma_prepare (vma))) goto oom;/* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ - folio =alloc_anon_folio (vmf); + folio =alloc_anon_folio (vmf, pte_range_none);if (IS_ERR (folio))return 0 ;if (!folio) --Quoted 2 lines
2.34.1
Thanks
Barry
Chris Li
maintainerRe: [PATCH RFC 4/6] mm: support large folios swapin as a whole
In reply to Barry SongQuoted 112 lines · Barry Song
From: Chuanhua Han <[email protected]>
On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole.
On the other hand, the current implementation only covers the
SWAP_SYCHRONOUScase. It doesn't supportswapin_readaheadas large folios yet.Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in
arch_swap_restore()while supporting MTE restore for folios rather than page.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-)mm/memory.c +42
-2@@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map);static vm_fault_t do_fault (struct vm_fault *vmf);static vm_fault_t do_anonymous_page (struct vm_fault *vmf);static bool vmf_pte_changed (struct vm_fault *vmf); +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int ));/* * Return true if the original pte was a uffd-wp pte marker (so the pte was @@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } +static bool pte_range_swap (pte_t *pte,int nr_pages) +{ +int i; +swp_entry_t entry; +unsigned type; +pgoff_t start_offset; + + entry =pte_to_swp_entry (ptep_get_lockless (pte)); +if (non_swap_entry (entry)) +return false; + start_offset =swp_offset (entry); +if (start_offset % nr_pages) +return false; + + type =swp_type (entry); +for (i =1 ; i < nr_pages; i++) { + entry =pte_to_swp_entry (ptep_get_lockless (pte + i)); +if (non_swap_entry (entry)) +return false; +if (swp_offset (entry) != start_offset + i) +return false; +if (swp_type (entry) != type) +return false; + } + +return true; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long start_address; +pte_t *start_pte;if (!pte_unmap_same (vmf)) goto out;@@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , - vma, vmf->address, false); + folio =alloc_anon_folio (vmf, pte_range_swap); page = &folio->page;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio); +if (folio_test_large (folio)) { +unsigned long start_offset; + + nr_pages =folio_nr_pages (folio); + start_offset =swp_offset (entry) & ~(nr_pages -1 ); + entry =swp_entry (swp_type (entry), start_offset); + } +if (mem_cgroup_swapin_charge_folio (folio, vma->vm_mm,GFP_KERNEL , entry)) {
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4029
Quoted 10 lines
mm/memory.c +7
-0@@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address; + start_pte = vmf->pte; +if (folio_test_large (folio)) { +unsigned long nr =folio_nr_pages (folio); +unsigned long addr =ALIGN_DOWN (vmf->address, nr *PAGE_SIZE ); +pte_t *pte_t = vmf->pte - (vmf->address - addr) /PAGE_SIZE ;
I forgot about one comment here.
Please change the variable name other than "pte_t", it is a bit strange to use the typedef name as variable name here.
Chris
Quoted 134 lines
mm/memory.c +45
-13+ +/* +* case 1: we are allocating large_folio, try to map it as a whole +* iff the swap entries are still entirely mapped; +* case 2: we hit a large folio in swapcache, and all swap entries +* are still entirely mapped, try to map a large folio as a whole. +* otherwise, map only the faulting page within the large folio +* which is swapcache +*/ +if (pte_range_swap (pte_t, nr)) { + start_address = addr; + start_pte = pte_t; +if (unlikely (folio == swapcache)) { +/* +* the below has been done before swap_read_folio() +* for case 1 +*/ + nr_pages = nr; + entry =pte_to_swp_entry (ptep_get (start_pte)); + page = &folio->page; + } + }else if (nr_pages >1 ) {/* ptes have changed for case 1 */ + goto out_nomap; + } + } +if (unlikely (!vmf->pte || !pte_same (ptep_get (vmf->pte), vmf->orig_pte))) goto out_nomap;@@ -4047,12 +4120,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* We're already holding a reference on the page but haven't mapped it * yet. */ -swap_free (entry); +swap_nr_free (entry, nr_pages);if (should_try_to_free_swap (folio, vma, vmf->flags))folio_free_swap (folio); -inc_mm_counter (vma->vm_mm,MM_ANONPAGES ); -dec_mm_counter (vma->vm_mm,MM_SWAPENTS ); +folio_ref_add (folio, nr_pages -1 ); +add_mm_counter (vma->vm_mm,MM_ANONPAGES , nr_pages); +add_mm_counter (vma->vm_mm,MM_SWAPENTS , -nr_pages); + pte =mk_pte (page, vma->vm_page_prot);/* @@ -4062,14 +4137,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)* exclusivity. */ if (!folio_test_ksm (folio) && - (exclusive ||folio_ref_count (folio) ==1 )) { + (exclusive ||folio_ref_count (folio) == nr_pages)) {if (vmf->flags &FAULT_FLAG_WRITE ) { pte =maybe_mkwrite (pte_mkdirty (pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE ; } rmap_flags |=RMAP_EXCLUSIVE ; } -flush_icache_page (vma, page); +flush_icache_pages (vma, page, nr_pages);if (pte_swp_soft_dirty (vmf->orig_pte)) pte =pte_mksoft_dirty (pte);if (pte_swp_uffd_wp (vmf->orig_pte))@@ -4081,14 +4156,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)folio_add_new_anon_rmap (folio, vma, vmf->address);folio_add_lru_vma (folio, vma); }else { -folio_add_anon_rmap_pte (folio, page, vma, vmf->address, +folio_add_anon_rmap_ptes (folio, page, nr_pages, vma, start_address, rmap_flags); }VM_BUG_ON (!folio_test_anon (folio) || (pte_write (pte) && !PageAnonExclusive (page))); -set_pte_at (vma->vm_mm, vmf->address, vmf->pte, pte); -arch_do_swap_page (vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); +set_ptes (vma->vm_mm, start_address, start_pte, pte, nr_pages); + +arch_do_swap_page (vma->vm_mm, vma, start_address, pte, vmf->orig_pte);folio_unlock (folio);if (folio != swapcache && swapcache) {@@ -4105,6 +4181,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }if (vmf->flags &FAULT_FLAG_WRITE ) { +if (folio_test_large (folio) && nr_pages >1 ) + vmf->orig_pte =ptep_get (vmf->pte); + ret |=do_wp_page (vmf);if (ret &VM_FAULT_ERROR ) ret &=VM_FAULT_ERROR ;@@ -4112,7 +4191,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) }/* No need to invalidate - it was non-present before */ -update_mmu_cache_range (vmf, vma, vmf->address, vmf->pte,1 ); +update_mmu_cache_range (vmf, vma, start_address, start_pte, nr_pages); unlock:if (vmf->pte)pte_unmap_unlock (vmf->pte, vmf->ptl);@@ -4148,7 +4227,8 @@ static bool pte_range_none(pte_t *pte, int nr_pages)return true; } -static struct folio *alloc_anon_folio (struct vm_fault *vmf) +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int )) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE struct vm_area_struct *vma = vmf->vma;@@ -4190,7 +4270,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) order =highest_order (orders);while (orders) { addr =ALIGN_DOWN (vmf->address,PAGE_SIZE << order); -if (pte_range_none (pte +pte_index (addr),1 << order)) +if (pte_range_check (pte +pte_index (addr),1 << order))break ; order =next_order (&orders, order); }@@ -4269,7 +4349,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)if (unlikely (anon_vma_prepare (vma))) goto oom;/* Returns NULL on OOM or ERR_PTR(-EAGAIN) if we must retry the fault */ - folio =alloc_anon_folio (vmf); + folio =alloc_anon_folio (vmf, pte_range_none);if (IS_ERR (folio))return 0 ;if (!folio) --2.34.1
Re: [PATCH RFC 4/6] mm: support large folios swapin as a whole
Quoted 120 lines · Chris Li
Quoted 118 lines · Barry Song
From: Chuanhua Han <[email protected]>
On an embedded system like Android, more than half of anon memory is actually in swap devices such as zRAM. For example, while an app is switched to back- ground, its most memory might be swapped-out.
Now we have mTHP features, unfortunately, if we don't support large folios swap-in, once those large folios are swapped-out, we immediately lose the performance gain we can get through large folios and hardware optimization such as CONT-PTE.
This patch brings up mTHP swap-in support. Right now, we limit mTHP swap-in to those contiguous swaps which were likely swapped out from mTHP as a whole.
On the other hand, the current implementation only covers the
SWAP_SYCHRONOUScase. It doesn't supportswapin_readaheadas large folios yet.Right now, we are re-faulting large folios which are still in swapcache as a whole, this can effectively decrease extra loops and early-exitings which we have increased in
arch_swap_restore()while supporting MTE restore for folios rather than page.Signed-off-by Chuanhua Han <[email protected]>Co-developed-by Barry Song <[email protected]>Signed-off-by Barry Song <[email protected]>--- mm/memory.c | 108 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 94 insertions(+), 14 deletions(-)mm/memory.c +45
-2@@ -107,6 +107,8 @@ EXPORT_SYMBOL(mem_map);static vm_fault_t do_fault (struct vm_fault *vmf);static vm_fault_t do_anonymous_page (struct vm_fault *vmf);static bool vmf_pte_changed (struct vm_fault *vmf); +static struct folio *alloc_anon_folio (struct vm_fault *vmf, +bool (*pte_range_check)(pte_t *,int ));/* * Return true if the original pte was a uffd-wp pte marker (so the pte was @@ -3784,6 +3786,34 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)return VM_FAULT_SIGBUS ; } +static bool pte_range_swap (pte_t *pte,int nr_pages) +{ +int i; +swp_entry_t entry; +unsigned type; +pgoff_t start_offset; + + entry =pte_to_swp_entry (ptep_get_lockless (pte)); +if (non_swap_entry (entry)) +return false; + start_offset =swp_offset (entry); +if (start_offset % nr_pages) +return false; + + type =swp_type (entry); +for (i =1 ; i < nr_pages; i++) { + entry =pte_to_swp_entry (ptep_get_lockless (pte + i)); +if (non_swap_entry (entry)) +return false; +if (swp_offset (entry) != start_offset + i) +return false; +if (swp_type (entry) != type) +return false; + } + +return true; +} +/* * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults), and pte mapped but not yet locked. @@ -3804,6 +3834,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)pte_t pte;vm_fault_t ret =0 ;void *shadow =NULL ; +int nr_pages =1 ; +unsigned long start_address; +pte_t *start_pte;if (!pte_unmap_same (vmf)) goto out;@@ -3868,13 +3901,20 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)if (data_race (si->flags &SWP_SYNCHRONOUS_IO ) &&__swap_count (entry) ==1 ) {/* skip swapcache */ - folio =vma_alloc_folio (GFP_HIGHUSER_MOVABLE ,0 , - vma, vmf->address, false); + folio =alloc_anon_folio (vmf, pte_range_swap); page = &folio->page;if (folio) {__folio_set_locked (folio);__folio_set_swapbacked (folio); +if (folio_test_large (folio)) { +unsigned long start_offset; + + nr_pages =folio_nr_pages (folio); + start_offset =swp_offset (entry) & ~(nr_pages -1 ); + entry =swp_entry (swp_type (entry), start_offset); + } +if (mem_cgroup_swapin_charge_folio (folio, vma->vm_mm,GFP_KERNEL , entry)) {@@ -3980,6 +4020,39 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)*/ vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + start_address = vmf->address; + start_pte = vmf->pte;
Quoted 9 lines
mm/memory.c vm_fault_t do_swap_page(struct vm_fault *vmf) · line 4029
Quoted 4 lines
mm/memory.c +4
-0+if (folio_test_large (folio)) { +unsigned long nr =folio_nr_pages (folio); +unsigned long addr =ALIGN_DOWN (vmf->address, nr *PAGE_SIZE ); +pte_t *pte_t = vmf->pte - (vmf->address - addr) /PAGE_SIZE ;I forgot about one comment here.
Please change the variable name other than "pte_t", it is a bit strange to use the typedef name as variable name here.
make sense!
Quoted 1 line
Chris
Thanks
Barry
RFC PATCH 5/6 mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
In reply to Barry SongFrom: Barry Song <[email protected]>
In do_swap_page(), while supporting large folio swap-in, we are using the helper folio_add_anon_rmap_ptes. This is triggerring a WARN_ON in __folio_add_anon_rmap. We can make the warning quiet by two ways
1. in do_swap_page, we call folio_add_new_anon_rmap() if we are sure the large folio is new allocated one; we call folio_add_anon_rmap_ptes() if we find the large folio in swapcache.
2. we always call folio_add_anon_rmap_ptes() in do_swap_page but weaken the WARN_ON in __folio_add_anon_rmap() by letting the WARN_ON less sensitive.
Option 2 seems to be better for do_swap_page() as it can use unified code for all cases.
--- mm/rmap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/rmap.c +4 -1
@@ -1304,7 +1304,10 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio,* page. */ VM_WARN_ON_FOLIO (folio_test_large (folio) && - level !=RMAP_LEVEL_PMD , folio); + level !=RMAP_LEVEL_PMD && + (!IS_ALIGNED (address, nr_pages *PAGE_SIZE ) || + (folio_test_swapcache (folio) && !IS_ALIGNED (folio->index, nr_pages)) || + page != &folio->page), folio);__folio_set_anon (folio, vma, address, !!(flags &RMAP_EXCLUSIVE )); }else if (likely (!folio_test_ksm (folio))) {
Signature · 1 line
2.34.1
Re: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Quoted 26 lines · Barry Song
From: Barry Song <[email protected]>
In
do_swap_page(), while supporting large folio swap-in, we are using the helperfolio_add_anon_rmap_ptes. This is triggerring aWARN_ONin__folio_add_anon_rmap. We can make the warning quiet by two ways
1. indo_swap_page, we callfolio_add_new_anon_rmap()if we are sure the large folio is new allocated one; we callfolio_add_anon_rmap_ptes()if we find the large folio in swapcache.
2. we always callfolio_add_anon_rmap_ptes()indo_swap_pagebut weaken theWARN_ONin__folio_add_anon_rmap()by letting theWARN_ONless sensitive.Option 2 seems to be better for
do_swap_page()as it can use unified code for all cases.Signed-off-by Barry Song <[email protected]>Tested-by Chuanhua Han <[email protected]>--- mm/rmap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)mm/rmap.c +0
-0@@ -1304,7 +1304,10 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio,* page.
mm/rmap.c static __always_inline void __folio_add_anon_rmap(struct folio *folio, · line 1313
Quoted 10 lines
mm/rmap.c +4
-1*/ VM_WARN_ON_FOLIO (folio_test_large (folio) && - level !=RMAP_LEVEL_PMD , folio); + level !=RMAP_LEVEL_PMD && + (!IS_ALIGNED (address, nr_pages *PAGE_SIZE ) || + (folio_test_swapcache (folio) && !IS_ALIGNED (folio->index, nr_pages)) || + page != &folio->page), folio);__folio_set_anon (folio, vma, address, !!(flags &RMAP_EXCLUSIVE )); }else if (likely (!folio_test_ksm (folio))) {
I have on my todo list to move all that !anon handling out of folio_add_anon_rmap_ptes(), and instead make swapin code call add folio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).
That's the cleaner approach.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Quoted 45 lines · David Hildenbrand
Quoted 26 lines · Barry Song
From: Barry Song <[email protected]>
In
do_swap_page(), while supporting large folio swap-in, we are using the helperfolio_add_anon_rmap_ptes. This is triggerring aWARN_ONin__folio_add_anon_rmap. We can make the warning quiet by two ways
1. indo_swap_page, we callfolio_add_new_anon_rmap()if we are sure the large folio is new allocated one; we callfolio_add_anon_rmap_ptes()if we find the large folio in swapcache.
2. we always callfolio_add_anon_rmap_ptes()indo_swap_pagebut weaken theWARN_ONin__folio_add_anon_rmap()by letting theWARN_ONless sensitive.Option 2 seems to be better for
do_swap_page()as it can use unified code for all cases.Signed-off-by Barry Song <[email protected]>Tested-by Chuanhua Han <[email protected]>--- mm/rmap.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)mm/rmap.c +0
-0@@ -1304,7 +1304,10 @@ static __always_inline void __folio_add_anon_rmap(struct folio *folio,* page. mm/rmap.c static __always_inline void __folio_add_anon_rmap(struct folio *folio, · line 1313
Quoted 10 lines
mm/rmap.c +4
-1*/ VM_WARN_ON_FOLIO (folio_test_large (folio) && - level !=RMAP_LEVEL_PMD , folio); + level !=RMAP_LEVEL_PMD && + (!IS_ALIGNED (address, nr_pages *PAGE_SIZE ) || + (folio_test_swapcache (folio) && !IS_ALIGNED (folio->index, nr_pages)) || + page != &folio->page), folio);__folio_set_anon (folio, vma, address, !!(flags &RMAP_EXCLUSIVE )); }else if (likely (!folio_test_ksm (folio))) {I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).
Quoted 2 lines
That's the cleaner approach.
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
call folio_add_new_anon_rmap.
especially when we want to support swapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.
is it an acceptable way to do the below in do_swap_page?
if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()
Quoted 5 lines
--
Cheers,David / dhildenb
Thanks
Barry
Chris Li
maintainerRe: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Hi David and Barry,
Quoted 11 lines · Barry Song
Quoted 10 lines
I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).That's the cleaner approach.
Quoted 6 lines
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
callfolio_add_new_anon_rmap.
especially when we want to supportswapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.
I think Barry has a point. Two tasks might race to swap in the folio then race to perform the rmap.folio_add_new_anon_rmap() should only call a folio that is absolutely "new", not shared. The sharing in swap cache disqualifies that condition.
Quoted 5 lines
is it an acceptable way to do the below in
do_swap_page? if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()
I am curious to know the answer as well.
BTW, that test might have a race as well. By the time the task got !anon result, this result might get changed by another task. We need to make sure in the caller context this race can't happen. Otherwise we can't do the above safely.
Chris.
Re: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Quoted 21 lines · Chris Li
Hi David and Barry,
Quoted 11 lines · Barry Song
Quoted 10 lines
I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).That's the cleaner approach.
Quoted 6 lines
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
callfolio_add_new_anon_rmap.
especially when we want to supportswapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.
Quoted 5 lines
I think Barry has a point. Two tasks might race to swap in the folio then race to perform the rmap.
folio_add_new_anon_rmap()should only call a folio that is absolutely "new", not shared. The sharing in swap cache disqualifies that condition.
We have to hold the folio lock. So only one task at a time might do the folio_add_anon_rmap_ptes() right now, and the folio_add_new_shared_anon_rmap() in the future [below].
Also observe how folio_add_anon_rmap_ptes() states that one must hold the page lock, because otherwise this would all be completely racy.
From the pte swp exclusive flags, we know for sure whether we are dealing with exclusive vs. shared. I think patch #6 does not properly check that all entries are actually the same in that regard (all exclusive vs all shared). That likely needs fixing.
[I have converting per-page PageAnonExclusive flags to a single per-folio flag on my todo list. I suspect that we'll keep the per-swp-pte exlusive bits, but the question is rather what we can actually make work, because swap and migration just make it much more complicated. Anyhow, future work]
Quoted 8 lines
Quoted 6 lines
is it an acceptable way to do the below in
do_swap_page? if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()I am curious to know the answer as well.
Yes, the end code should likely be something like:
/* ksm created a completely new copy */
if (unlikely(folio != swapcache && swapcache)) {
folio_add_new_anon_rmap(folio, vma, vmf->address);
folio_add_lru_vma(folio, vma);
} else if (folio_test_anon(folio)) { folio_add_anon_rmap_ptes(rmap_flags)
} else {
folio_add_new_anon_rmap(rmap_flags)
}
Maybe we want to avoid teaching all existing folio_add_new_anon_rmap() callers about a new flag, and just have a new folio_add_new_shared_anon_rmap() instead. TBD.
Quoted 5 lines
BTW, that test might have a race as well. By the time the task got !anon result, this result might get changed by another task. We need to make sure in the caller context this race can't happen. Otherwise we can't do the above safely.
Again, folio lock. Observe the folio_lock_or_retry() call that covers our existing folio_add_new_anon_rmap/folio_add_anon_rmap_pte calls.
Signature · 3 lines
Cheers, David / dhildenb
Chris Li
maintainerRe: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Quoted 29 lines · David Hildenbrand
Quoted 21 lines · Chris Li
Hi David and Barry,
Quoted 11 lines · Barry Song
Quoted 10 lines
I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).That's the cleaner approach.
Quoted 6 lines
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
callfolio_add_new_anon_rmap.
especially when we want to supportswapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.Quoted 5 lines
I think Barry has a point. Two tasks might race to swap in the folio then race to perform the rmap.
folio_add_new_anon_rmap()should only call a folio that is absolutely "new", not shared. The sharing in swap cache disqualifies that condition.
Quoted 4 lines
We have to hold the folio lock. So only one task at a time might do the
folio_add_anon_rmap_ptes()right now, and thefolio_add_new_shared_anon_rmap()in the future [below].
Ah, I see. The folio_lock() is the answer I am looking for.
Quoted 31 lines
Also observe how
folio_add_anon_rmap_ptes()states that one must hold the page lock, because otherwise this would all be completely racy.From the pte swp exclusive flags, we know for sure whether we are
dealing with exclusive vs. shared. I think patch #6 does not properly check that all entries are actually the same in that regard (all exclusive vs all shared). That likely needs fixing.[I have converting per-page PageAnonExclusive flags to a single per-folio flag on my todo list. I suspect that we'll keep the per-swp-pte exlusive bits, but the question is rather what we can actually make work, because swap and migration just make it much more complicated. Anyhow, future work]
Quoted 8 lines
Quoted 6 lines
is it an acceptable way to do the below in
do_swap_page? if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()I am curious to know the answer as well.
Yes, the end code should likely be something like:
/* ksm created a completely new copy */
if (unlikely(folio != swapcache && swapcache)) { folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else if (folio_test_anon(folio)) {
Quoted 8 lines
folio_add_anon_rmap_ptes(rmap_flags)
} else {
folio_add_new_anon_rmap(rmap_flags)
}Maybe we want to avoid teaching all existing
folio_add_new_anon_rmap()callers about a new flag, and just have a newfolio_add_new_shared_anon_rmap()instead. TBD.
There is more than one caller needed to perform that dance around folio_test_anon() then decide which function to call. It would be nice to have a wrapper function folio_add_new_shared_anon_rmap() to abstract this behavior.
Quoted 8 lines
Quoted 6 lines
BTW, that test might have a race as well. By the time the task got !anon result, this result might get changed by another task. We need to make sure in the caller context this race can't happen. Otherwise we can't do the above safely.
Again, folio lock. Observe the
folio_lock_or_retry()call that covers our existing folio_add_new_anon_rmap/folio_add_anon_rmap_pte calls.
Ack. Thanks for the explanation.
Chris
Re: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
Quoted 31 lines · Chris Li
Quoted 29 lines · David Hildenbrand
Quoted 26 lines · Chris Li
Hi David and Barry,
Quoted 11 lines · Barry Song
Quoted 10 lines
I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).That's the cleaner approach.
Quoted 6 lines
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
callfolio_add_new_anon_rmap.
especially when we want to supportswapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.I think Barry has a point. Two tasks might race to swap in the folio then race to perform the rmap.
folio_add_new_anon_rmap()should only call a folio that is absolutely "new", not shared. The sharing in swap cache disqualifies that condition.
Quoted 46 lines
Quoted 4 lines
We have to hold the folio lock. So only one task at a time might do the
folio_add_anon_rmap_ptes()right now, and thefolio_add_new_shared_anon_rmap()in the future [below].Ah, I see. The
folio_lock()is the answer I am looking for.Quoted 39 lines
Also observe how
folio_add_anon_rmap_ptes()states that one must hold the page lock, because otherwise this would all be completely racy.From the pte swp exclusive flags, we know for sure whether we are
dealing with exclusive vs. shared. I think patch #6 does not properly check that all entries are actually the same in that regard (all exclusive vs all shared). That likely needs fixing.[I have converting per-page PageAnonExclusive flags to a single per-folio flag on my todo list. I suspect that we'll keep the per-swp-pte exlusive bits, but the question is rather what we can actually make work, because swap and migration just make it much more complicated. Anyhow, future work]
Quoted 8 lines
Quoted 6 lines
is it an acceptable way to do the below in
do_swap_page? if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()I am curious to know the answer as well.
Yes, the end code should likely be something like:
/* ksm created a completely new copy */
if (unlikely(folio != swapcache && swapcache)) { folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else if (folio_test_anon(folio)) {
folio_add_anon_rmap_ptes(rmap_flags)
} else {
folio_add_new_anon_rmap(rmap_flags)
}Maybe we want to avoid teaching all existing
folio_add_new_anon_rmap()callers about a new flag, and just have a newfolio_add_new_shared_anon_rmap()instead. TBD.
if we have to add a wrapper like folio_add_new_shared_anon_rmap() to avoid "if (folio_test_anon(folio))" and "else" everywhere, why not we just do it in folio_add_anon_rmap_ptes() ?
folio_add_anon_rmap_ptes()
{
if (!folio_test_anon(folio))
return folio_add_new_anon_rmap();
}Anyway, I am going to change the patch 4/6 to if(folio_test_anon)/else first and drop this 5/6.
we may figure out if we need a wrapper later.
Quoted 13 lines
There is more than one caller needed to perform that dance around
folio_test_anon()then decide which function to call. It would be nice to have a wrapper functionfolio_add_new_shared_anon_rmap()to abstract this behavior.Quoted 6 lines
Quoted 6 lines
BTW, that test might have a race as well. By the time the task got !anon result, this result might get changed by another task. We need to make sure in the caller context this race can't happen. Otherwise we can't do the above safely.
Quoted 6 lines
Quoted 2 lines
Again, folio lock. Observe the
folio_lock_or_retry()call that covers our existing folio_add_new_anon_rmap/folio_add_anon_rmap_pte calls.Ack. Thanks for the explanation.
Chris
Thanks
Barry
Re: [PATCH RFC 5/6] mm: rmap: weaken the WARN_ON in __folio_add_anon_rmap()
In reply to David HildenbrandQuoted 64 lines · David Hildenbrand
Quoted 21 lines · Chris Li
Hi David and Barry,
Quoted 11 lines · Barry Song
Quoted 10 lines
I have on my todo list to move all that !anon handling out of
folio_add_anon_rmap_ptes(), and instead make swapin code call addfolio_add_new_anon_rmap(), where we'll have to pass an exclusive flag then (-> whole new folio exclusive).That's the cleaner approach.
Quoted 6 lines
one tricky thing is that sometimes it is hard to know who is the first one to add rmap and thus should
callfolio_add_new_anon_rmap.
especially when we want to supportswapin_readahead(), the one who allocated large filio might not
be that one who firstly does rmap.Quoted 5 lines
I think Barry has a point. Two tasks might race to swap in the folio then race to perform the rmap.
folio_add_new_anon_rmap()should only call a folio that is absolutely "new", not shared. The sharing in swap cache disqualifies that condition.We have to hold the folio lock. So only one task at a time might do the
folio_add_anon_rmap_ptes()right now, and thefolio_add_new_shared_anon_rmap()in the future [below].Also observe how
folio_add_anon_rmap_ptes()states that one must hold the page lock, because otherwise this would all be completely racy.From the pte swp exclusive flags, we know for sure whether we are
dealing with exclusive vs. shared. I think patch #6 does not properly check that all entries are actually the same in that regard (all exclusive vs all shared). That likely needs fixing.[I have converting per-page PageAnonExclusive flags to a single per-folio flag on my todo list. I suspect that we'll keep the per-swp-pte exlusive bits, but the question is rather what we can actually make work, because swap and migration just make it much more complicated. Anyhow, future work]
Quoted 8 lines
Quoted 6 lines
is it an acceptable way to do the below in
do_swap_page? if (!folio_test_anon(folio))
folio_add_new_anon_rmap()
else
folio_add_anon_rmap_ptes()I am curious to know the answer as well.
Yes, the end code should likely be something like:
/* ksm created a completely new copy */
if (unlikely(folio != swapcache && swapcache)) { folio_add_new_anon_rmap(folio, vma, vmf->address); folio_add_lru_vma(folio, vma); } else if (folio_test_anon(folio)) {
Quoted 8 lines
folio_add_anon_rmap_ptes(rmap_flags)
} else {
folio_add_new_anon_rmap(rmap_flags)
}Maybe we want to avoid teaching all existing
folio_add_new_anon_rmap()callers about a new flag, and just have a newfolio_add_new_shared_anon_rmap()instead. TBD.
right.
We need to clarify that the new anon_folio might not necessarily be exclusive. Unlike folio_add_new_anon_rmap, which assumes the new folio is exclusive, folio_add_anon_rmap_ptes is capable of handling both exclusive and non-exclusive new anon folios.
The code would be like:
if (unlikely(folio != swapcache && swapcache)) {
folio_add_new_anon_rmap(folio, vma, vmf->address);
folio_add_lru_vma(folio, vma);
} else if (!folio_test_anon(folio)) {
folio_add_anon_rmap_ptes(rmap_flags);
} else {
if (exclusive)
folio_add_new_anon_rmap();
else
folio_add_new_shared_anon_rmap();}
It appears a bit lengthy?
Quoted 6 lines
Quoted 6 lines
BTW, that test might have a race as well. By the time the task got !anon result, this result might get changed by another task. We need to make sure in the caller context this race can't happen. Otherwise we can't do the above safely.
Quoted 7 lines
Again, folio lock. Observe the
folio_lock_or_retry()call that covers our existing folio_add_new_anon_rmap/folio_add_anon_rmap_pte calls.--
Cheers,David / dhildenb
Thanks
Barry