mm, swap: Enable THP SWAP for PowerPC Book3S64

On PowerPC Book3S64, MMU is selected at runtime, so macros like PMD_SHIFT are effectively runtime variables in the Book3S64 code. THP

swap code uses these macros to size some of its array data structures based on PMD_ORDER e.g. SWAPFILE_CLUSTER macro is used for this very purpose.
Hence this patch makes the users of SWAPFILE_CLUSTER to use this macro value at runtime and also modifies swap_table and swap_memcg_table which were earlier using this macro for defining the number of table entries.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
 mm/swap_table.h |  6 ++----
 mm/swapfile.c   | 17 ++++++++++++-----
 2 files changed, 14 insertions(+), 9 deletions(-)

mm/swap_table.h +2 -4

index e6613e62f8d0..90e2a7852300 100644
@@ -8,16 +8,14 @@
 
 /* A typical flat array in each cluster as swap table */
 struct swap_table {
-       atomic_long_t entries[SWAPFILE_CLUSTER];
+       DECLARE_FLEX_ARRAY(atomic_long_t, entries);
 };
 
 /* For storing memcg private id */
 struct swap_memcg_table {
-       unsigned short id[SWAPFILE_CLUSTER];
+       DECLARE_FLEX_ARRAY(unsigned short, id);
 };
 
-#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)
-
 /*
  * A swap table entry represents the status of a swap slot on a swap
  * (physical or virtual) device. The swap table in each cluster is a

mm/swapfile.c +0 -0

index 78b49b0658ad..4bf11c5b87eb 100644

Quoted 3 lines · Ritesh Harjani (IBM)

On PowerPC Book3S64, MMU is selected at runtime, so macros like PMD_SHIFT are effectively runtime variables in the Book3S64 code. THP

Not an expert on Book3S64—could you explain the runtime variables in more detail? Does enabling THP_SWAP on PowerPC cause any build issues?

Quoted 44 lines

swap code uses these macros to size some of its array data structures based on PMD_ORDER e.g. SWAPFILE_CLUSTER macro is used for this very purpose.
Hence this patch makes the users of SWAPFILE_CLUSTER to use this macro value at runtime and also modifies swap_table and swap_memcg_table which were earlier using this macro for defining the number of table entries.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
 mm/swap_table.h |  6 ++----
 mm/swapfile.c   | 17 ++++++++++++-----
 2 files changed, 14 insertions(+), 9 deletions(-)

mm/swap_table.h +2 -4

index e6613e62f8d0..90e2a7852300 100644
@@ -8,16 +8,14 @@
 
 /* A typical flat array in each cluster as swap table */
 struct swap_table {
-       atomic_long_t entries[SWAPFILE_CLUSTER];
+       DECLARE_FLEX_ARRAY(atomic_long_t, entries);
 };
 
 /* For storing memcg private id */
 struct swap_memcg_table {
-       unsigned short id[SWAPFILE_CLUSTER];
+       DECLARE_FLEX_ARRAY(unsigned short, id);
 };
 
-#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)
-
 /*
  * A swap table entry represents the status of a swap slot on a swap
  * (physical or virtual) device. The swap table in each cluster is a

mm/swapfile.c +0 -0

index 78b49b0658ad..4bf11c5b87eb 100644
@@ -129,6 +129,8 @@ static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = {
        .lock = INIT_LOCAL_LOCK(),
 };
 

mm/swapfile.c static DEFINE_PER_CPU(struct percpu_swap_cluster, percpu_swap_cluster) = { · line 132

Quoted 1 line

mm/swapfile.c +1 -0

+static bool swap_table_use_page __ro_after_init;

Does a static key help here?

Best Regards
Barry

On PowerPC Book3S64, MMU is selected at runtime, so macros like PMD_SHIFT are effectively runtime variables in the Book3S64 code. THP swap code uses these macros to size some of its array data structures based on PMD_ORDER e.g. SWAPFILE_CLUSTER macro is used for this very purpose.
Hence this patch makes the users of SWAPFILE_CLUSTER to use this macro value at runtime and also modifies swap_table and swap_memcg_table which were earlier using this macro for defining the number of table entries.

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

[1]: https://lore.kernel.org/linuxppc-dev/[email protected]/

Quoted 11 lines · Ritesh Harjani (IBM)

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

Quoted 9 lines

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

Do we know the maximum PMD size?

On arm64 with a 64 KB base page,
a PMD can be as large as 512 MB:
https://docs.kernel.org/arch/arm64/hugetlbpage.html

One concern we have is that performing I/O on such a large folio could incur significant latency before reclaiming any memory. For this reason, on arm64 we initially enabled THP_SWAPOUT only for 4 KB base pages:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0637c505f

Quoted 3 lines

[1]: https://lore.kernel.org/linuxppc-dev/[email protected]/

Best Regards
Barry

Quoted 23 lines · Barry Song

Quoted 11 lines · Ritesh Harjani (IBM)

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

Quoted 9 lines

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

Quoted 1 line

Do we know the maximum PMD size?

ARCH_MAX_PMD_ORDER will be 8 on PowerPC book3s64 with 64K pagesize. PowerPC Hash MMU with 64K default pagesize supports PMD size of 16MB.

Quoted 4 lines

On arm64 with a 64 KB base page,
a PMD can be as large as 512 MB:
https://docs.kernel.org/arch/arm64/hugetlbpage.html

Quoted 6 lines

One concern we have is that performing I/O on such a large folio could incur significant latency before reclaiming any memory. For this reason, on arm64 we initially enabled THP_SWAPOUT only for 4 KB base pages:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0637c505f

That's not the case on PowerPC. Max PMD size for Hash will be 16MB.

Also we still need this patch since we can at runtime choose Hash or Radix MMU. So, the main problem this patch is trying to solve on PowerPC Book3s64 is enabling this feature w/o impacting any other architecture. W/O this patch series, we can't enable it, since it gives build errors.

Quoted 35 lines · Ritesh Harjani

Quoted 23 lines · Barry Song

Quoted 11 lines · Ritesh Harjani (IBM)

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

Quoted 9 lines

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

Quoted 1 line

Do we know the maximum PMD size?

ARCH_MAX_PMD_ORDER will be 8 on PowerPC book3s64 with 64K pagesize. PowerPC Hash MMU with 64K default pagesize supports PMD size of 16MB.

Quoted 4 lines

On arm64 with a 64 KB base page,
a PMD can be as large as 512 MB:
https://docs.kernel.org/arch/arm64/hugetlbpage.html

Quoted 8 lines

Quoted 6 lines

One concern we have is that performing I/O on such a large folio could incur significant latency before reclaiming any memory. For this reason, on arm64 we initially enabled THP_SWAPOUT only for 4 KB base pages:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0637c505f

That's not the case on PowerPC. Max PMD size for Hash will be 16MB.

Yep. A 16 MB folio might be fine, although I'm not sure whether splitting a 16 MB folio into eight 2 MB folios would help much.

For 512 MB PMD-sized pages on arm64, one possible approach might be to split them into 256 × 2 MB folios rather than all the way down to 4 KB pages. That could provide a better balance between I/O latency and swap performance.

Quoted 4 lines

Also we still need this patch since we can at runtime choose Hash or Radix MMU. So, the main problem this patch is trying to solve on PowerPC Book3s64 is enabling this feature w/o impacting any other architecture. W/O this patch series, we can't enable it, since it gives build errors.

I see. If possible, please mention in the changelog that the maximum PMD size on your platform is 16 MB.

In that case, the I/O latency concerns I raised may not really apply.

w/ that, please free feel to add:

Reviewed-by Barry Song <[email protected]>

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

[1]: https://lore.kernel.org/linuxppc-dev/[email protected]/

Suggested-by YoungJun Park <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
 arch/powerpc/include/asm/book3s/64/pgtable.h |  7 +++++++
 include/linux/swap.h                         | 12 +++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

arch/powerpc/include/asm/book3s/64/pgtable.h +0 -0

index e67e64ac6e8c..7f22d5d5fbdf 100644

Quoted 35 lines · Ritesh Harjani (IBM)

SWAP_NR_ORDERS sizes a few small bounded arrays inside THP swap allocator code (nofull/frag cluster lists, percpu_swap_cluster's si/offset arrays, next array for rotational device). This currently expands to PMD_ORDER+1, which only works when PMD_ORDER is a compile time constant.

However on architecture like PowerPC Book3S64, PMD_ORDER is a runtime variable which depends upon which MMU is selected (Radix / Hash), so in that case, PMD_ORDER cannot be used to size the static arrays.

This patch provides an optional ARCH_MAX_PMD_ORDER (upper-bound) override for such architectures. The memory overhead on enabling this override is negligible. Even if we make SWAP_NR_ORDERS runtime alloc, default slab padding could cause some memory waste. Also we lose the per-cpu cacheline benefits (for percpu_swap_cluster) because it might cost an extra cacheline indirection overhead in swap_alloc_fast() for fetching si[order]/offset[order]. Note that a fully runtime SWAP_NR_ORDERS was considered in previous version but was dropped for this reason [1]

[1]: https://lore.kernel.org/linuxppc-dev/[email protected]/

Suggested-by YoungJun Park <[email protected]>

Signed-off-by Ritesh Harjani (IBM) <[email protected]>

---
 arch/powerpc/include/asm/book3s/64/pgtable.h |  7 +++++++
 include/linux/swap.h                         | 12 +++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

arch/powerpc/include/asm/book3s/64/pgtable.h +0 -0

index e67e64ac6e8c..7f22d5d5fbdf 100644

arch/powerpc/include/asm/book3s/64/pgtable.h extern unsigned long __pmd_frag_size_shift; · line 212

Quoted 9 lines

arch/powerpc/include/asm/book3s/64/pgtable.h +6 -0

@@ -204,6 +204,13 @@ extern unsigned long __pmd_frag_size_shift;
 #define MAX_PTRS_PER_PGD       (1 << (H_PGD_INDEX_SIZE > RADIX_PGD_INDEX_SIZE ? \
                                       H_PGD_INDEX_SIZE : RADIX_PGD_INDEX_SIZE))
 
+/*
+ * Compile-time upper bound on PMD_ORDER across hash and radix MMUs.
+ * Used by THP SWAP code. Check include/linux/swap.h
+ */
+#define ARCH_MAX_PMD_ORDER ((H_PTE_INDEX_SIZE > RADIX_PTE_INDEX_SIZE) ? \
+                               H_PTE_INDEX_SIZE : RADIX_PTE_INDEX_SIZE)

Hi Ritesh

So swap is the only user of this macro? Will there by any other users?

I see that due to the percpu cluster design, it's hard to use a flexible array here. We will probabaly get rid of the fixed percpu cluster design in the future. By then should we be able to get rid of this macro?

I'm OK with this approach though. This current design has no negative effect on other archs so no reason to block it,

just wondering if this can be made simpler in the future :)

THP_SWAP avoids splitting of a transparent huge folio into 32 smaller 64K folios (Radix-64K pagesize / 2M PMD) or into 256 smaller 64K folios (Hash-64K pagesize / 16M PMD), during swapout. This improves the swapping performance since all the bookking & I/O submission happens once per large folio. More details at [1].

PowerPC Book3S64 could not enable this before because PMD_ORDER is selected at runtime depending upon the chosen MMU. The earlier patches in this series turn SWAPFILE_CLUSTER into a runtime value and introduce an ARCH_MAX_PMD_ORDER upperbound override for SWAP_NR_ORDERS. With those changes, we can now enable THP SWAP for Book3S64.

This increases bandwidth throughput with zram backend for swapout by 40-50% with Radix and 100-130% with Hash (Tested by Sayali)

[1]: https://lore.kernel.org/all/[email protected]/

Quoted 13 lines · Ritesh Harjani (IBM)

THP_SWAP avoids splitting of a transparent huge folio into 32 smaller 64K folios (Radix-64K pagesize / 2M PMD) or into 256 smaller 64K folios (Hash-64K pagesize / 16M PMD), during swapout. This improves the swapping performance since all the bookking & I/O submission happens once per large folio. More details at [1].

PowerPC Book3S64 could not enable this before because PMD_ORDER is selected at runtime depending upon the chosen MMU. The earlier patches in this series turn SWAPFILE_CLUSTER into a runtime value and introduce an ARCH_MAX_PMD_ORDER upperbound override for SWAP_NR_ORDERS. With those changes, we can now enable THP SWAP for Book3S64.

Quoted 2 lines

This increases bandwidth throughput with zram backend for swapout by 40-50% with Radix and 100-130% with Hash (Tested by Sayali)

Thanks!

I am curious about the contents of the anonymous memory being tested and the compression algorithm used by zram.

Quoted 2 lines

[1]: https://lore.kernel.org/all/[email protected]/

Best Regards
Barry

Quoted 15 lines · Barry Song

Quoted 13 lines · Ritesh Harjani (IBM)

THP_SWAP avoids splitting of a transparent huge folio into 32 smaller 64K folios (Radix-64K pagesize / 2M PMD) or into 256 smaller 64K folios (Hash-64K pagesize / 16M PMD), during swapout. This improves the swapping performance since all the bookking & I/O submission happens once per large folio. More details at [1].

PowerPC Book3S64 could not enable this before because PMD_ORDER is selected at runtime depending upon the chosen MMU. The earlier patches in this series turn SWAPFILE_CLUSTER into a runtime value and introduce an ARCH_MAX_PMD_ORDER upperbound override for SWAP_NR_ORDERS. With those changes, we can now enable THP SWAP for Book3S64.

Quoted 8 lines

Quoted 2 lines

This increases bandwidth throughput with zram backend for swapout by 40-50% with Radix and 100-130% with Hash (Tested by Sayali)

Thanks!

I am curious about the contents of the anonymous memory being tested and the compression algorithm used by zram.

I am sure it was derived from your microbenchmark itself which you had shared here (so repetitive pattern) with default zram compression algorithm. Thanks for that :)

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0637c505f

I think I got your point - I can mention that it was a microbenchmark similar to yours and not a real world workload test. Is this what you meant here?

Quoted 31 lines · Ritesh Harjani

Quoted 15 lines · Barry Song

Quoted 13 lines · Ritesh Harjani (IBM)

THP_SWAP avoids splitting of a transparent huge folio into 32 smaller 64K folios (Radix-64K pagesize / 2M PMD) or into 256 smaller 64K folios (Hash-64K pagesize / 16M PMD), during swapout. This improves the swapping performance since all the bookking & I/O submission happens once per large folio. More details at [1].

PowerPC Book3S64 could not enable this before because PMD_ORDER is selected at runtime depending upon the chosen MMU. The earlier patches in this series turn SWAPFILE_CLUSTER into a runtime value and introduce an ARCH_MAX_PMD_ORDER upperbound override for SWAP_NR_ORDERS. With those changes, we can now enable THP SWAP for Book3S64.

Quoted 8 lines

Quoted 2 lines

This increases bandwidth throughput with zram backend for swapout by 40-50% with Radix and 100-130% with Hash (Tested by Sayali)

Thanks!

I am curious about the contents of the anonymous memory being tested and the compression algorithm used by zram.

I am sure it was derived from your microbenchmark itself which you had shared here (so repetitive pattern) with default zram compression algorithm. Thanks for that :)

Quoted 5 lines

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d0637c505f

I think I got your point - I can mention that it was a microbenchmark similar to yours and not a real world workload test. Is this what you meant here?

Yep. Please make it clear in the changelog what kind of workload was used, as different data can result in completely different compression ratios and compression/decompression costs. Consequently, the reported swap-out and swap-in performance improvements can vary significantly as well.

w/ that, please feel free to add:

Reviewed-by Barry Song <[email protected]>