Skip to contentSkip to conversation

Swap-out small-sized THP without splitting

PATCH Superseded Ryan Roberts 2 participants 8 messages Oct 17, 2023 – Oct 20, 2023

Subsystems (5)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ryan Roberts

author reviewer
#0 lore raw

v2 cover Swap-out small-sized THP without splitting

Hi All,

This is v2 of a series to add support for swapping out small-sized THP without needing to first split the large folio via __split_huge_page(). It closely follows the approach already used by PMD-sized THP.

"Small-sized THP" is an upcoming feature that enables performance improvements by allocating large folios for anonymous memory, where the large folio size is smaller than the traditional PMD-size. See [2].

In some circumstances I've observed a performance regression (see patch 2 for details), and this series is an attempt to fix the regression in advance of merging small-sized THP support.

I've done what I thought was the smallest change possible, and as a result, this approach is only employed when the swap is backed by a non-rotating block device (just as PMD-sized THP is supported today). Discussion against the RFC concluded that this is probably sufficient.

The series applies against mm-unstable (3fb06e6d0a6f)

Changes since v1 [1]
====================

- patch 1:
- Use cluster_set_count() instead of cluster_set_count_flag() in swap_alloc_cluster() since we no longer have any flag to set. I was unable to kill cluster_set_count_flag() as proposed against v1 as other call sites depend explicitly setting flags to 0.
- patch 2:
- Moved large_next[] array into percpu_cluster to make it per-cpu (recommended by Huang, Ying).
- large_next[] array is dynamically allocated because PMD_ORDER is not compile-time constant for powerpc (fixes build error).

Thanks,
Ryan

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/

Ryan Roberts (2):
mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags
mm: swap: Swap-out small-sized THP without splitting

 include/linux/swap.h |  16 +++---
 mm/huge_memory.c     |   3 --
 mm/swapfile.c        | 119 ++++++++++++++++++++++++-------------------
 mm/vmscan.c          |  10 ++--
 4 files changed, 78 insertions(+), 70 deletions(-)

--
2.25.1

author reviewer
#1 lore raw

PATCH v2 1/2 mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags

As preparation for supporting small-sized THP in the swap-out path, without first needing to split to order-0, Remove the CLUSTER_FLAG_HUGE, which, when present, always implies PMD-sized THP, which is the same as the cluster size.

The only use of the flag was to determine whether a swap entry refers to a single page or a PMD-sized THP in swap_page_trans_huge_swapped(). Instead of relying on the flag, we now pass in nr_pages, which originates from the folio's number of pages. This allows the logic to work for folios of any order.

The one snag is that one of the swap_page_trans_huge_swapped() call sites does not have the folio. But it was only being called there to avoid bothering to call __try_to_reclaim_swap() in some cases. __try_to_reclaim_swap() gets the folio and (via some other functions) calls swap_page_trans_huge_swapped(). So I've removed the problematic call site and believe the new logic should be equivalent.

Removing CLUSTER_FLAG_HUGE also means we can remove split_swap_cluster() which used to be called during folio splitting, since split_swap_cluster()'s only job was to remove the flag.

---
 include/linux/swap.h | 10 ----------
 mm/huge_memory.c     |  3 ---
 mm/swapfile.c        | 47 ++++++++------------------------------------
 3 files changed, 8 insertions(+), 52 deletions(-)
include/linux/swap.h +0 -10
index 19f30a29e1f1..a073366a227c 100644
@@ -259,7 +259,6 @@ struct swap_cluster_info {
 };
 #define CLUSTER_FLAG_FREE 1 /* This cluster is free */
 #define CLUSTER_FLAG_NEXT_NULL 2 /* This cluster has no next cluster */
-#define CLUSTER_FLAG_HUGE 4 /* This cluster is backing a transparent huge page */
 
 /*
  * We assign a cluster to each CPU, so each CPU can allocate swap entry from
@@ -595,15 +594,6 @@ static inline int add_swap_extent(struct swap_info_struct *sis,
 }
 #endif /* CONFIG_SWAP */
 
-#ifdef CONFIG_THP_SWAP
-extern int split_swap_cluster(swp_entry_t entry);
-#else
-static inline int split_swap_cluster(swp_entry_t entry)
-{
-	return 0;
-}
-#endif
-
 #ifdef CONFIG_MEMCG
 static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg)
 {
mm/huge_memory.c +0 -3
index c9cbcbf6697e..46b3fb943207 100644
@@ -2597,9 +2597,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,
 		shmem_uncharge(head->mapping->host, nr_dropped);
 	remap_page(folio, nr);
 
-	if (folio_test_swapcache(folio))
-		split_swap_cluster(folio->swap);
-
 	for (i = 0; i < nr; i++) {
 		struct page *subpage = head + i;
 		if (subpage == page)
mm/swapfile.c +8 -39

107 lines not on this page · raw message

Signature · 1 line
2.25.1
author reviewer
#2 lore raw

PATCH v2 2/2 mm: swap: Swap-out small-sized THP without splitting

In reply to Ryan Roberts

The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.

Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.

Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.

The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [4, (1 << PMD_ORDER)] (THP cannot support order-1 folios). This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).

The per-order current clusters are maintained per-cpu using the existing percpu_cluster infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.

As far as I can tell, this should not cause any extra fragmentation concerns, given how similar it is to the existing PMD-sized THP allocation mechanism. There could be up to (PMD_ORDER-2) * nr_cpus clusters in concurrent use though, which in a pathalogical case (cluster set aside for every order for every cpu and only one huge entry allocated from it) would tie up ~12MiB of unused swap entries for these high orders (assuming PMD_ORDER=9). In practice, the number of orders in use will be small and the amount of swap space reserved is very small compared to a typical swap file.

Note that PMD_ORDER is not compile-time constant on powerpc, so we have to allocate the large_next[] array at runtime.

I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run usemem from vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean and stdev:

Mean Performance Improvement vs 4K/baseline

| alloc size |            baseline |       + this series |
|            |  v6.6-rc4+anonfolio |                     |
|:-----------|--------------------:|--------------------:|
| 4K Page    |                0.0% |                1.1% |
| 64K THP    |              -44.1% |                0.9% |
| 2M THP     |               56.0% |               56.4% |

So with this change, the regression for 64K swap performance goes away. Both 4K and 64K benhcmarks are now bottlenecked on TLBI performance from try_to_unmap_flush_dirty(), on arm64 at least. When using fewer cpus in the test, I see upto 2x performance of 64K THP swapping compared to 4K.

---
 include/linux/swap.h |  6 ++++
 mm/swapfile.c        | 74 +++++++++++++++++++++++++++++++++++---------
 mm/vmscan.c          | 10 +++---
 3 files changed, 71 insertions(+), 19 deletions(-)
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +59 -15

134 lines not on this page · raw message

mm/vmscan.c +6 -4
index c16e2b1ea8ae..5984d2ae4547 100644
@@ -1212,11 +1212,13 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 					if (!can_split_folio(folio, NULL))
 						goto activate_locked;
 					/*
-					 * Split folios without a PMD map right
-					 * away. Chances are some or all of the
-					 * tail pages can be freed without IO.
+					 * Split PMD-mappable folios without a
+					 * PMD map right away. Chances are some
+					 * or all of the tail pages can be freed
+					 * without IO.
 					 */
-					if (!folio_entire_mapcount(folio) &&
+					if (folio_test_pmd_mappable(folio) &&
+					    !folio_entire_mapcount(folio) &&
 					    split_folio_to_list(folio,
 								folio_list))
 						goto activate_locked;

--
2.25.1

#3 lore raw

Re: [PATCH v2 2/2] mm: swap: Swap-out small-sized THP without splitting

Quoted 96 lines · Ryan Roberts

The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.

Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.

Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.

The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [4, (1 << PMD_ORDER)] (THP cannot support order-1 folios). This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).

The per-order current clusters are maintained per-cpu using the existing percpu_cluster infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.

As far as I can tell, this should not cause any extra fragmentation concerns, given how similar it is to the existing PMD-sized THP allocation mechanism. There could be up to (PMD_ORDER-2) * nr_cpus clusters in concurrent use though, which in a pathalogical case (cluster set aside for every order for every cpu and only one huge entry allocated from it) would tie up ~12MiB of unused swap entries for these high orders (assuming PMD_ORDER=9). In practice, the number of orders in use will be small and the amount of swap space reserved is very small compared to a typical swap file.

Note that PMD_ORDER is not compile-time constant on powerpc, so we have to allocate the large_next[] array at runtime.

I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run usemem from vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean and stdev:

Mean Performance Improvement vs 4K/baseline

| alloc size |            baseline |       + this series |
|            |  v6.6-rc4+anonfolio |                     |
|:-----------|--------------------:|--------------------:|
| 4K Page    |                0.0% |                1.1% |
| 64K THP    |              -44.1% |                0.9% |
| 2M THP     |               56.0% |               56.4% |

So with this change, the regression for 64K swap performance goes away. Both 4K and 64K benhcmarks are now bottlenecked on TLBI performance from try_to_unmap_flush_dirty(), on arm64 at least. When using fewer cpus in the test, I see upto 2x performance of 64K THP swapping compared to 4K.

---
 include/linux/swap.h |  6 ++++
 mm/swapfile.c        | 74 +++++++++++++++++++++++++++++++++++---------
 mm/vmscan.c          | 10 +++---
 3 files changed, 71 insertions(+), 19 deletions(-)
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +0 -0
index b83ad77e04c0..625964e53c22 100644

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 991

Quoted 6 lines
mm/swapfile.c +2 -1
@@ -987,35 +987,70 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 	return n_ret;
 }
 
-static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
+			    unsigned int nr_pages)

This looks hacky. IMO, we should put the allocation logic inside percpu_cluster framework. If percpu_cluster framework doesn't work for you, just refactor it firstly.

Quoted 45 lines
mm/swapfile.c +21 -8
 {
+	int order_idx;
 	unsigned long idx;
 	struct swap_cluster_info *ci;
+	struct percpu_cluster *cluster;
 	unsigned long offset;
 
 	/*
 	 * Should not even be attempting cluster allocations when huge
 	 * page swap is disabled.  Warn and fail the allocation.
 	 */
-	if (!IS_ENABLED(CONFIG_THP_SWAP)) {
+	if (!IS_ENABLED(CONFIG_THP_SWAP) ||
+	    nr_pages < 4 || nr_pages > SWAPFILE_CLUSTER ||
+	    !is_power_of_2(nr_pages)) {
 		VM_WARN_ON_ONCE(1);
 		return 0;
 	}
 
-	if (cluster_list_empty(&si->free_clusters))
+	/*
+	 * Not using clusters so unable to allocate large entries.
+	 */
+	if (!si->cluster_info)
 		return 0;
 
-	idx = cluster_list_first(&si->free_clusters);
-	offset = idx * SWAPFILE_CLUSTER;
-	ci = lock_cluster(si, offset);
-	alloc_cluster(si, idx);
-	cluster_set_count(ci, SWAPFILE_CLUSTER);
+	order_idx = ilog2(nr_pages) - 2;
+	cluster = this_cpu_ptr(si->percpu_cluster);
+	offset = cluster->large_next[order_idx];
+
+	if (offset == UINT_MAX) {
+		if (cluster_list_empty(&si->free_clusters))
+			return 0;
+
+		idx = cluster_list_first(&si->free_clusters);
+		offset = idx * SWAPFILE_CLUSTER;
 
-	memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER);
+		ci = lock_cluster(si, offset);
+		alloc_cluster(si, idx);

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1038

Quoted 10 lines
mm/swapfile.c +10 -0
+		cluster_set_count(ci, SWAPFILE_CLUSTER);
+
+		/*
+		 * If scan_swap_map_slots() can't find a free cluster, it will
+		 * check si->swap_map directly. To make sure this standby
+		 * cluster isn't taken by scan_swap_map_slots(), mark the swap
+		 * entries bad (occupied). (same approach as discard).
+		 */
+		memset(si->swap_map + offset + nr_pages, SWAP_MAP_BAD,
+			SWAPFILE_CLUSTER - nr_pages);

There's an issue with this solution. If the free space of swap device runs low, it's possible that

- some cluster are put in the percpu_cluster of some CPUs the swap entries there are marked as used

- no free swap entries elsewhere

- nr_swap_pages isn't 0

So, we will still scan LRU, but swap allocation fails, although there's still free swap space.

I think that we should follow the method we used for the original percpu_cluster. That is, if all free swap entries are in percpu_cluster, we will start to allocate from percpu_cluster.

Quoted 9 lines
mm/swapfile.c +7 -1
+	} else {
+		idx = offset / SWAPFILE_CLUSTER;
+		ci = lock_cluster(si, offset);
+	}
+
+	memset(si->swap_map + offset, SWAP_HAS_CACHE, nr_pages);
 	unlock_cluster(ci);
-	swap_range_alloc(si, offset, SWAPFILE_CLUSTER);
+	swap_range_alloc(si, offset, nr_pages);

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1056

Quoted 10 lines
mm/swapfile.c +5 -0
 	*slot = swp_entry(si->type, offset);
 
+	offset += nr_pages;
+	if (idx != offset / SWAPFILE_CLUSTER)
+		offset = UINT_MAX;
+	cluster->large_next[order_idx] = offset;
+
 	return 1;
 }
 

[snip]

--
Best Regards,
Huang, Ying

author reviewer
#4 lore raw

Re: [PATCH v2 2/2] mm: swap: Swap-out small-sized THP without splitting

Quoted 101 lines · Ying
Quoted 99 lines · Ryan Roberts

The upcoming anonymous small-sized THP feature enables performance improvements by allocating large folios for anonymous memory. However I've observed that on an arm64 system running a parallel workload (e.g. kernel compilation) across many cores, under high memory pressure, the speed regresses. This is due to bottlenecking on the increased number of TLBIs added due to all the extra folio splitting.

Therefore, solve this regression by adding support for swapping out small-sized THP without needing to split the folio, just like is already done for PMD-sized THP. This change only applies when CONFIG_THP_SWAP is enabled, and when the swap backing store is a non-rotating block device. These are the same constraints as for the existing PMD-sized THP swap-out support.

Note that no attempt is made to swap-in THP here - this is still done page-by-page, like for PMD-sized THP.

The main change here is to improve the swap entry allocator so that it can allocate any power-of-2 number of contiguous entries between [4, (1 << PMD_ORDER)] (THP cannot support order-1 folios). This is done by allocating a cluster for each distinct order and allocating sequentially from it until the cluster is full. This ensures that we don't need to search the map and we get no fragmentation due to alignment padding for different orders in the cluster. If there is no current cluster for a given order, we attempt to allocate a free cluster from the list. If there are no free clusters, we fail the allocation and the caller falls back to splitting the folio and allocates individual entries (as per existing PMD-sized THP fallback).

The per-order current clusters are maintained per-cpu using the existing percpu_cluster infrastructure. This is done to avoid interleving pages from different tasks, which would prevent IO being batched. This is already done for the order-0 allocations so we follow the same pattern.

As far as I can tell, this should not cause any extra fragmentation concerns, given how similar it is to the existing PMD-sized THP allocation mechanism. There could be up to (PMD_ORDER-2) * nr_cpus clusters in concurrent use though, which in a pathalogical case (cluster set aside for every order for every cpu and only one huge entry allocated from it) would tie up ~12MiB of unused swap entries for these high orders (assuming PMD_ORDER=9). In practice, the number of orders in use will be small and the amount of swap space reserved is very small compared to a typical swap file.

Note that PMD_ORDER is not compile-time constant on powerpc, so we have to allocate the large_next[] array at runtime.

I've run the tests on Ampere Altra (arm64), set up with a 35G block ram device as the swap device and from inside a memcg limited to 40G memory. I've then run usemem from vm-scalability with 70 processes (each has its own core), each allocating and writing 1G of memory. I've repeated everything 5 times and taken the mean and stdev:

Mean Performance Improvement vs 4K/baseline

| alloc size |            baseline |       + this series |
|            |  v6.6-rc4+anonfolio |                     |
|:-----------|--------------------:|--------------------:|
| 4K Page    |                0.0% |                1.1% |
| 64K THP    |              -44.1% |                0.9% |
| 2M THP     |               56.0% |               56.4% |

So with this change, the regression for 64K swap performance goes away. Both 4K and 64K benhcmarks are now bottlenecked on TLBI performance from try_to_unmap_flush_dirty(), on arm64 at least. When using fewer cpus in the test, I see upto 2x performance of 64K THP swapping compared to 4K.

---
 include/linux/swap.h |  6 ++++
 mm/swapfile.c        | 74 +++++++++++++++++++++++++++++++++++---------
 mm/vmscan.c          | 10 +++---
 3 files changed, 71 insertions(+), 19 deletions(-)
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +0 -0
index b83ad77e04c0..625964e53c22 100644
@@ -987,35 +987,70 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 	return n_ret;
 }
 
Quoted 7 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 991

Quoted 3 lines
mm/swapfile.c +2 -1
-static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
+			    unsigned int nr_pages)

This looks hacky. IMO, we should put the allocation logic inside percpu_cluster framework. If percpu_cluster framework doesn't work for you, just refactor it firstly.

I'm not sure I really understand what you are suggesting - could you elaborate? What "framework"? I only see a per-cpu data structure and scan_swap_map_try_ssd_cluster(), which is very much geared towards order-0 allocations.

Are you suggesting you want to allocate large entries (> order-0) from the same cluster that is used for small (order-0) entries? The problem with this approach is that there may not be enough space left in the current cluster for the large entry that you are trying to allocate. Then you would need to take a new cluster from the free list and presumably leave the previous cluster with unused entries (which will now only be accessible by scanning). That unused space could be considerable.

That's why I am currently reserving a "current cluster" per order; that way, all allocations are the same order, they are all naturally aligned and there is no waste.

Perhaps I could implement "stealing" between cpus so that a cpu trying to allocate a specific order, but which doesn't have a current cluster for that order and the free list is empty, could allocate from another cpu's current cluster? I don't think it's a good idea to mix different orders in the same cluster though.

I guess if really low, I could remove a current cluster from a cpu and allow it to be scanned for order-0 allocations at least?

Any opinions gratefully received!

Quoted 63 lines
Quoted 46 lines
mm/swapfile.c +21 -8
 {
+	int order_idx;
 	unsigned long idx;
 	struct swap_cluster_info *ci;
+	struct percpu_cluster *cluster;
 	unsigned long offset;
 
 	/*
 	 * Should not even be attempting cluster allocations when huge
 	 * page swap is disabled.  Warn and fail the allocation.
 	 */
-	if (!IS_ENABLED(CONFIG_THP_SWAP)) {
+	if (!IS_ENABLED(CONFIG_THP_SWAP) ||
+	    nr_pages < 4 || nr_pages > SWAPFILE_CLUSTER ||
+	    !is_power_of_2(nr_pages)) {
 		VM_WARN_ON_ONCE(1);
 		return 0;
 	}
 
-	if (cluster_list_empty(&si->free_clusters))
+	/*
+	 * Not using clusters so unable to allocate large entries.
+	 */
+	if (!si->cluster_info)
 		return 0;
 
-	idx = cluster_list_first(&si->free_clusters);
-	offset = idx * SWAPFILE_CLUSTER;
-	ci = lock_cluster(si, offset);
-	alloc_cluster(si, idx);
-	cluster_set_count(ci, SWAPFILE_CLUSTER);
+	order_idx = ilog2(nr_pages) - 2;
+	cluster = this_cpu_ptr(si->percpu_cluster);
+	offset = cluster->large_next[order_idx];
+
+	if (offset == UINT_MAX) {
+		if (cluster_list_empty(&si->free_clusters))
+			return 0;
+
+		idx = cluster_list_first(&si->free_clusters);
+		offset = idx * SWAPFILE_CLUSTER;
 
-	memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER);
+		ci = lock_cluster(si, offset);
+		alloc_cluster(si, idx);

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1038

Quoted 10 lines
mm/swapfile.c +10 -0
+		cluster_set_count(ci, SWAPFILE_CLUSTER);
+
+		/*
+		 * If scan_swap_map_slots() can't find a free cluster, it will
+		 * check si->swap_map directly. To make sure this standby
+		 * cluster isn't taken by scan_swap_map_slots(), mark the swap
+		 * entries bad (occupied). (same approach as discard).
+		 */
+		memset(si->swap_map + offset + nr_pages, SWAP_MAP_BAD,
+			SWAPFILE_CLUSTER - nr_pages);

There's an issue with this solution. If the free space of swap device runs low, it's possible that

- some cluster are put in the percpu_cluster of some CPUs the swap entries there are marked as used

Quoted 6 lines

- no free swap entries elsewhere

- nr_swap_pages isn't 0

So, we will still scan LRU, but swap allocation fails, although there's still free swap space.

Ahh yes, good spot.

Quoted 4 lines

I think that we should follow the method we used for the original percpu_cluster. That is, if all free swap entries are in percpu_cluster, we will start to allocate from percpu_cluster.

As i suggested above, I think I could do this by removing a cpu's current cluster for a given order from the percpu_cluster and making it generally available for scanning. Does that work for you?

Quoted 26 lines
Quoted 10 lines
mm/swapfile.c +7 -1
+	} else {
+		idx = offset / SWAPFILE_CLUSTER;
+		ci = lock_cluster(si, offset);
+	}
+
+	memset(si->swap_map + offset, SWAP_HAS_CACHE, nr_pages);
 	unlock_cluster(ci);
-	swap_range_alloc(si, offset, SWAPFILE_CLUSTER);
+	swap_range_alloc(si, offset, nr_pages);

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1056

Quoted 10 lines
mm/swapfile.c +5 -0
 	*slot = swp_entry(si->type, offset);
 
+	offset += nr_pages;
+	if (idx != offset / SWAPFILE_CLUSTER)
+		offset = UINT_MAX;
+	cluster->large_next[order_idx] = offset;
+
 	return 1;
 }
 

[snip]

--
Best Regards,
Huang, Ying

#5 lore raw

Re: [PATCH v2 2/2] mm: swap: Swap-out small-sized THP without splitting

Quoted 3 lines · Ryan Roberts
Quoted 2 lines · Ying

Ryan Roberts <[email protected]> writes:

[snip]

Quoted 33 lines
Quoted 25 lines
Quoted 25 lines
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +0 -0
index b83ad77e04c0..625964e53c22 100644
@@ -987,35 +987,70 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 	return n_ret;
 }
 
Quoted 7 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 991

Quoted 3 lines
mm/swapfile.c +2 -1
-static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
+			    unsigned int nr_pages)

This looks hacky. IMO, we should put the allocation logic inside percpu_cluster framework. If percpu_cluster framework doesn't work for you, just refactor it firstly.

Quoted 4 lines

I'm not sure I really understand what you are suggesting - could you elaborate? What "framework"? I only see a per-cpu data structure and scan_swap_map_try_ssd_cluster(), which is very much geared towards order-0 allocations.

I suggest to share as much code as possible between order-0 and order > 0 swap entry allocation. I think that we can make scan_swap_map_try_ssd_cluster() works for order > 0 swap entry allocation.

Quoted 11 lines

Are you suggesting you want to allocate large entries (> order-0) from the same cluster that is used for small (order-0) entries? The problem with this approach is that there may not be enough space left in the current cluster for the large entry that you are trying to allocate. Then you would need to take a new cluster from the free list and presumably leave the previous cluster with unused entries (which will now only be accessible by scanning). That unused space could be considerable.

That's why I am currently reserving a "current cluster" per order; that way, all allocations are the same order, they are all naturally aligned and there is no waste.

I am fine to use one swap cluster per order per CPU. I just think that we should share code.

Quoted 5 lines

Perhaps I could implement "stealing" between cpus so that a cpu trying to allocate a specific order, but which doesn't have a current cluster for that order and the free list is empty, could allocate from another cpu's current cluster? I don't think it's a good idea to mix different orders in the same cluster though.

I think we can start from a simple solution, that is, just fall back to split the large folio. Then, we can optimize it step by step.

Quoted 2 lines

I guess if really low, I could remove a current cluster from a cpu and allow it to be scanned for order-0 allocations at least?

Better to have same behavior between order- and order > 0. Perhaps begin with the current solution (allow swap entries in per-CPU cluster to be stolen). Then we can optimize based on this.

Not directly related to this patchset. Maybe we can replace swap_slots_cache with per-CPU cluster in the future. This will reduce the code complexity.

Quoted 1 line

Any opinions gratefully received!

Thanks!

Quoted 73 lines
Quoted 63 lines
Quoted 46 lines
mm/swapfile.c +21 -8
 {
+	int order_idx;
 	unsigned long idx;
 	struct swap_cluster_info *ci;
+	struct percpu_cluster *cluster;
 	unsigned long offset;
 
 	/*
 	 * Should not even be attempting cluster allocations when huge
 	 * page swap is disabled.  Warn and fail the allocation.
 	 */
-	if (!IS_ENABLED(CONFIG_THP_SWAP)) {
+	if (!IS_ENABLED(CONFIG_THP_SWAP) ||
+	    nr_pages < 4 || nr_pages > SWAPFILE_CLUSTER ||
+	    !is_power_of_2(nr_pages)) {
 		VM_WARN_ON_ONCE(1);
 		return 0;
 	}
 
-	if (cluster_list_empty(&si->free_clusters))
+	/*
+	 * Not using clusters so unable to allocate large entries.
+	 */
+	if (!si->cluster_info)
 		return 0;
 
-	idx = cluster_list_first(&si->free_clusters);
-	offset = idx * SWAPFILE_CLUSTER;
-	ci = lock_cluster(si, offset);
-	alloc_cluster(si, idx);
-	cluster_set_count(ci, SWAPFILE_CLUSTER);
+	order_idx = ilog2(nr_pages) - 2;
+	cluster = this_cpu_ptr(si->percpu_cluster);
+	offset = cluster->large_next[order_idx];
+
+	if (offset == UINT_MAX) {
+		if (cluster_list_empty(&si->free_clusters))
+			return 0;
+
+		idx = cluster_list_first(&si->free_clusters);
+		offset = idx * SWAPFILE_CLUSTER;
 
-	memset(si->swap_map + offset, SWAP_HAS_CACHE, SWAPFILE_CLUSTER);
+		ci = lock_cluster(si, offset);
+		alloc_cluster(si, idx);

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1038

Quoted 10 lines
mm/swapfile.c +10 -0
+		cluster_set_count(ci, SWAPFILE_CLUSTER);
+
+		/*
+		 * If scan_swap_map_slots() can't find a free cluster, it will
+		 * check si->swap_map directly. To make sure this standby
+		 * cluster isn't taken by scan_swap_map_slots(), mark the swap
+		 * entries bad (occupied). (same approach as discard).
+		 */
+		memset(si->swap_map + offset + nr_pages, SWAP_MAP_BAD,
+			SWAPFILE_CLUSTER - nr_pages);

There's an issue with this solution. If the free space of swap device runs low, it's possible that

- some cluster are put in the percpu_cluster of some CPUs the swap entries there are marked as used

Quoted 6 lines

- no free swap entries elsewhere

- nr_swap_pages isn't 0

So, we will still scan LRU, but swap allocation fails, although there's still free swap space.

Ahh yes, good spot.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

I think that we should follow the method we used for the original percpu_cluster. That is, if all free swap entries are in percpu_cluster, we will start to allocate from percpu_cluster.

As i suggested above, I think I could do this by removing a cpu's current cluster for a given order from the percpu_cluster and making it generally available for scanning. Does that work for you?

replied above.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
mm/swapfile.c +7 -1
+	} else {
+		idx = offset / SWAPFILE_CLUSTER;
+		ci = lock_cluster(si, offset);
+	}
+
+	memset(si->swap_map + offset, SWAP_HAS_CACHE, nr_pages);
 	unlock_cluster(ci);
-	swap_range_alloc(si, offset, SWAPFILE_CLUSTER);
+	swap_range_alloc(si, offset, nr_pages);
 	*slot = swp_entry(si->type, offset);
 
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines
mm/swapfile.c +5 -0
+	offset += nr_pages;
+	if (idx != offset / SWAPFILE_CLUSTER)
+		offset = UINT_MAX;
+	cluster->large_next[order_idx] = offset;
+
 	return 1;
 }
 

[snip]

--
Best Regards,
Huang, Ying

author reviewer
#6 lore raw

Re: [PATCH v2 2/2] mm: swap: Swap-out small-sized THP without splitting

Quoted 50 lines · Ying
Quoted 3 lines · Ryan Roberts
Quoted 2 lines · Ying

Ryan Roberts <[email protected]> writes:

[snip]

Quoted 33 lines
Quoted 25 lines
Quoted 25 lines
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +0 -0
index b83ad77e04c0..625964e53c22 100644
@@ -987,35 +987,70 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 	return n_ret;
 }
 
Quoted 7 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 991

Quoted 3 lines
mm/swapfile.c +2 -1
-static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
+			    unsigned int nr_pages)

This looks hacky. IMO, we should put the allocation logic inside percpu_cluster framework. If percpu_cluster framework doesn't work for you, just refactor it firstly.

Quoted 4 lines

I'm not sure I really understand what you are suggesting - could you elaborate? What "framework"? I only see a per-cpu data structure and scan_swap_map_try_ssd_cluster(), which is very much geared towards order-0 allocations.

I suggest to share as much code as possible between order-0 and order > 0 swap entry allocation. I think that we can make scan_swap_map_try_ssd_cluster() works for order > 0 swap entry allocation.

[...]

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1038

Quoted 8 lines
mm/swapfile.c +8 -0
+		/*
+		 * If scan_swap_map_slots() can't find a free cluster, it will
+		 * check si->swap_map directly. To make sure this standby
+		 * cluster isn't taken by scan_swap_map_slots(), mark the swap
+		 * entries bad (occupied). (same approach as discard).
+		 */
+		memset(si->swap_map + offset + nr_pages, SWAP_MAP_BAD,
+			SWAPFILE_CLUSTER - nr_pages);

There's an issue with this solution. If the free space of swap device runs low, it's possible that

- some cluster are put in the percpu_cluster of some CPUs the swap entries there are marked as used

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- no free swap entries elsewhere

- nr_swap_pages isn't 0

So, we will still scan LRU, but swap allocation fails, although there's still free swap space.

I'd like to decide how best to solve this problem before I can figure out how much code sharing I can do for the order-0 vs order > 0 allocators.

I see a couple of potential options:

1) Manipulate nr_swap_pages to include the entries that are marked SWAP_MAP_BAD, so when reserving a new per-order/per-cpu cluster, subtract SWAPFILE_CLUSTER, and then add nr_pages for each allocation from that cluster.

2) Don't mark the entries in the reserved cluster as SWAP_MAP_BAD, which would allow the scanner to steal (order-0) entries. The scanner could set a flag in the cluster info to mark it as having been allocated from by the scanner, so the next attempt to allocate a high order from it would cause discarding it as the cpu's current cluster and trying to get a fresh cluster from the free list.

While option 2 is a bit more complex, I prefer it as a solution. What do you think?

#7 lore raw

Re: [PATCH v2 2/2] mm: swap: Swap-out small-sized THP without splitting

Quoted 85 lines · Ryan Roberts
Quoted 50 lines · Ying
Quoted 3 lines · Ryan Roberts
Quoted 2 lines · Ying

Ryan Roberts <[email protected]> writes:

[snip]

Quoted 33 lines
Quoted 25 lines
Quoted 25 lines
include/linux/swap.h +6 -0
index a073366a227c..35cbbe6509a9 100644
@@ -268,6 +268,12 @@ struct swap_cluster_info {
 struct percpu_cluster {
 	struct swap_cluster_info index; /* Current cluster index */
 	unsigned int next; /* Likely next allocation offset */
+	unsigned int large_next[];	/*
+					 * next free offset within current
+					 * allocation cluster for large folios,
+					 * or UINT_MAX if no current cluster.
+					 * Index is (order - 1).
+					 */
 };
 
 struct swap_cluster_list {
mm/swapfile.c +0 -0
index b83ad77e04c0..625964e53c22 100644
@@ -987,35 +987,70 @@ static int scan_swap_map_slots(struct swap_info_struct *si,
 	return n_ret;
 }
 
Quoted 7 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 991

Quoted 3 lines
mm/swapfile.c +2 -1
-static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+static int swap_alloc_large(struct swap_info_struct *si, swp_entry_t *slot,
+			    unsigned int nr_pages)

This looks hacky. IMO, we should put the allocation logic inside percpu_cluster framework. If percpu_cluster framework doesn't work for you, just refactor it firstly.

Quoted 4 lines

I'm not sure I really understand what you are suggesting - could you elaborate? What "framework"? I only see a per-cpu data structure and scan_swap_map_try_ssd_cluster(), which is very much geared towards order-0 allocations.

I suggest to share as much code as possible between order-0 and order > 0 swap entry allocation. I think that we can make scan_swap_map_try_ssd_cluster() works for order > 0 swap entry allocation.

[...]

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines

mm/swapfile.c static int scan_swap_map_slots(struct swap_info_struct *si, · line 1038

Quoted 8 lines
mm/swapfile.c +8 -0
+		/*
+		 * If scan_swap_map_slots() can't find a free cluster, it will
+		 * check si->swap_map directly. To make sure this standby
+		 * cluster isn't taken by scan_swap_map_slots(), mark the swap
+		 * entries bad (occupied). (same approach as discard).
+		 */
+		memset(si->swap_map + offset + nr_pages, SWAP_MAP_BAD,
+			SWAPFILE_CLUSTER - nr_pages);

There's an issue with this solution. If the free space of swap device runs low, it's possible that

- some cluster are put in the percpu_cluster of some CPUs the swap entries there are marked as used

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

- no free swap entries elsewhere

- nr_swap_pages isn't 0

So, we will still scan LRU, but swap allocation fails, although there's still free swap space.

I'd like to decide how best to solve this problem before I can figure out how much code sharing I can do for the order-0 vs order > 0 allocators.

I see a couple of potential options:

1) Manipulate nr_swap_pages to include the entries that are marked SWAP_MAP_BAD, so when reserving a new per-order/per-cpu cluster, subtract SWAPFILE_CLUSTER, and then add nr_pages for each allocation from that cluster.

Quoted 7 lines

2) Don't mark the entries in the reserved cluster as SWAP_MAP_BAD, which would allow the scanner to steal (order-0) entries. The scanner could set a flag in the cluster info to mark it as having been allocated from by the scanner, so the next attempt to allocate a high order from it would cause discarding it as the cpu's current cluster and trying to get a fresh cluster from the free list.

While option 2 is a bit more complex, I prefer it as a solution. What do you think?

I think that this is a good choice to start with. We may build more optimization on top of it if necessary.

--
Best Regards,
Huang, Ying