Skip to contentSkip to conversation

mm: swap: introduce swap tier infrastructure

PATCH Superseded Youngjun Park 6 participants 15 messages May 27 – Jun 1

Subsystems (7)

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Youngjun Park

author reviewer
#0 lore raw

v7 cover mm: swap: introduce swap tier infrastructure

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

I revisited the design (see Design Rationale). Since our use case fits best with a memcg-based model, the implementation remains within memcg and preserves its resource accounting semantics.

Alternatives considered:

1. A separate sysfs interface under swap. (Workable. But, it would still need to reference memcg paths, and fully decoupling it would add swap-layer logic to manage memcgs, making it secondary option.)

2. Making the feature non-default.

Other interfaces were also reviewed. Aside from sysfs and BPF, the options involve trade-offs and are largely design choices. BPF was excluded due to possible disablement on our embedded platform, though future extension remains possible.

Overview
========

Swap Tiers group swap devices into performance classes (e.g. NVMe, HDD, Network) and allow per-memcg selection of which tiers to use. This mechanism was suggested by Chris Li.

Design Rationale
================

Swap tier selection is attached to memcg. A child cgroup may select a subset of the parent's allowed tiers.

This
- Preserves cgroup inheritance semantics (boundary at parent, refinement at child).
- Reuses memcg, which already groups processes and enforces hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.

Placing tier control outside memcg (e.g., via BPF, syscalls, or madvise) would allow swap preference to diverge from the memcg hierarchy. Integrating it into memcg keeps the swap policy consistent with existing memory ownership semantics. There are also real use cases built around memcg.

In the future, this can be extended to other interfaces to cover additional use cases.

I believe a memcg-based swap control is a good starting point before such extensions.

Use Cases
=========

#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)

The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers according to latency requirements.

This prevents latency-sensitive workloads from being swapped to slow devices used by background workloads.

#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)

The parent (job) has access to both zswap and SSD tiers. The child (VMM guest memory) selects SSD as its swap tier via memory.swap.tiers. In this deployment, swap device selection happens at the child level from the parent's available set.

#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)

Each child uses a different tier. Since swap paths are separated per tier, synchronization overhead between the two workloads is reduced.

Future extension
================

#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a round-robin fashion. Per-tier policy files under /sys/kernel/mm/swap/tiers/ can control how devices within a tier are selected (e.g. round-robin, weighted).

#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single tier. The current interface defines only tier assignment; it does not yet define when or how pages move between tiers. Two triggering models are possible:

(a) User-triggered: userspace explicitly initiates migration between tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at appropriate points such as reclaim or refault.

#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process swap. Or we can use it as BPF program.

Experimentation
===============

Tested on our internal platform using NBD as a separate swap tier. Our first production's simple usecase.

Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications

Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)

Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks

Change log
===========

v7
- Collect Baoquan's review tag
- Baoquan's feedback on fixing improper comment
- Minor code adjustments per Baoquan's feedback.
- Rebase on recent mm-new
- v6 link: https://lore.kernel.org/linux-mm/[email protected]/

v6
- Sashiko AI review fixes
- Fix batch parsing error path to restore snapshot before exit
- Reject overlong tier names to prevent truncated duplicates
- Avoid restoring raw list_head via memcpy (stale pointer risk)
- Ensure early parse errors do not skip DEF_SWAP_PRIO validation
- Use (1U << TIER_DEFAULT_IDX) to avoid signed shift UB
- Defer tier mask inheritance to css_online() to close race window
- Add READ_ONCE()/WRITE_ONCE() for tier mask accesses
- Other fixes
- Fix build error reintroduced due to missing v5 change (sorry for that..)
- Fix WARNING in folio_tier_effective_mask by adding rcu_read_lock() (syzbot CI fix)
- default number of swap tier max (change to 32->31, for reserving last bit)
- commit message refinement.
- rebased on recently mm-new
- v5 link: https://lore.kernel.org/linux-mm/[email protected]/

v5
- Fixed build errors reported in v4
- rebased on up to date mm-new
- Minor cleanups
- Design docs with validation (by Shakeel Butt discussion)
- v4 link : https://lore.kernel.org/linux-mm/[email protected]/

v4
- Simplified control flow and indentation
- Added CONFIG option for MAX_SWAPTIER (default: 4)
- Added memory.swap.tiers.effective interface
- Reworked save/restore logic into snapshot/rollback model
- Removed tier priority modification support (deferred)
- Improved validation and fixed edge cases
- Rebased onto latest mm-new
- RFC v3 link: https://lore.kernel.org/linux-mm/[email protected]/

RFC v1 ~ v3
- Change the direction after discussion with Chris-Li
- apply some LPC feedback.
- RFC v2 - https://lore.kernel.org/linux-mm/[email protected]/
- RFC v1 - https://lore.kernel.org/linux-mm/[email protected]/

Earlier Approach (per cgroup swap priority)
- v1: https://lore.kernel.org/linux-mm/[email protected]/
- RFC: https://lore.kernel.org/linux-mm/aEvLjEInMQC7hEyh@yjaykim-PowerEdge-T330/T/#mbbb6a5e9e30843097e1f5f65fb98f31d582b973d

Youngjun Park (4):
mm: swap: introduce swap tier infrastructure
mm: swap: associate swap devices with tiers
mm: memcontrol: add interfaces for swap tier selection
mm: swap: filter swap allocation by memcg tier mask

 Documentation/admin-guide/cgroup-v2.rst |  29 ++
 Documentation/mm/index.rst              |   1 +
 Documentation/mm/swap-tier.rst          | 159 ++++++++
 MAINTAINERS                             |   3 +
 include/linux/memcontrol.h              |   5 +
 include/linux/swap.h                    |   1 +
 mm/Kconfig                              |  12 +
 mm/Makefile                             |   2 +-
 mm/memcontrol.c                         |  96 +++++
 mm/swap.h                               |   4 +
 mm/swap_state.c                         |  75 ++++
 mm/swap_tier.c                          | 482 ++++++++++++++++++++++++
 mm/swap_tier.h                          |  75 ++++
 mm/swapfile.c                           |  20 +-
 14 files changed, 959 insertions(+), 5 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h

base-commit: 938bf00744a1b82cefd551f848a927cc24d5fb2f

Signature · 1 line
2.34.1
author reviewer
#1 lore raw

PATCH v7 1/4 mm: swap: introduce swap tier infrastructure

This patch introduces the "Swap tier" concept, which serves as an abstraction layer for managing swap devices based on their performance characteristics (e.g., NVMe, HDD, Network swap).

Swap tiers are user-named groups representing priority ranges. Tier names must consist of alphanumeric characters and underscores. These tiers collectively cover the entire priority space from -1 (DEF_SWAP_PRIO) to SHRT_MAX.

To configure tiers, a new sysfs interface is exposed at /sys/kernel/mm/swap/tiers. The input parser evaluates commands from left to right and supports batch input, allowing users to add or remove multiple tiers in a single write operation.

Tier management enforces continuous priority ranges anchored by start priorities. Operations trigger range splitting or merging, but overwriting start priorities is forbidden. Merging expands lower tiers upwards to preserve configured start priorities, except when removing DEF_SWAP_PRIO, which merges downwards.

---
 MAINTAINERS     |   2 +
 mm/Kconfig      |  12 ++
 mm/Makefile     |   2 +-
 mm/swap.h       |   4 +
 mm/swap_state.c |  74 ++++++++++++
 mm/swap_tier.c  | 302 ++++++++++++++++++++++++++++++++++++++++++++++++
 mm/swap_tier.h  |  20 ++++
 mm/swapfile.c   |   8 +-
 8 files changed, 420 insertions(+), 4 deletions(-)
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h
MAINTAINERS +2 -0
index e3ee97f5474e..e3cbfedbaa5f 100644
@@ -17049,6 +17049,8 @@ F:	mm/swap.c
 F:	mm/swap.h
 F:	mm/swap_table.h
 F:	mm/swap_state.c
+F:	mm/swap_tier.c
+F:	mm/swap_tier.h
 F:	mm/swapfile.c
 
 MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
mm/Kconfig +12 -0
index 776b67c66e82..5343937f3da9 100644
@@ -19,6 +19,18 @@ menuconfig SWAP
 	  used to provide more virtual memory than the actual RAM present
 	  in your computer.  If unsure say Y.
 
+config NR_SWAP_TIERS
+        int "Number of swap device tiers"
+        depends on SWAP
+        default 4
+        range 1 31
+        help
+          Sets the number of swap device tiers. Swap devices are
+          grouped into tiers based on their priority, allowing the
+          system to prefer faster devices over slower ones.
+
+          If unsure, say 4.
+
 config ZSWAP
 	bool "Compressed cache for swap pages"
 	depends on SWAP
mm/Makefile +1 -1
index eff9f9e7e061..29cb1e778285 100644
@@ -75,7 +75,7 @@ ifdef CONFIG_MMU
 	obj-$(CONFIG_ADVISE_SYSCALLS)	+= madvise.o
 endif
 
-obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o swap_tier.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
mm/swap.h +4 -0
index 900a539c63f0..067f0afd7a3e 100644
@@ -34,6 +34,10 @@ extern int page_cluster;
 #define swap_entry_order(order)	0
 #endif
 
+#define DEF_SWAP_PRIO  -1
+
+extern spinlock_t swap_lock;
+extern struct plist_head swap_active_head;
 extern struct swap_info_struct *swap_info[];
 
 /*
mm/swap_state.c +74 -0

88 lines not on this page · raw message

mm/swap_tier.c new file +302 -0

302 lines not on this page · raw message

mm/swap_tier.h new file +20 -0
new file mode 100644
index 000000000000..a1395ec02c24
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+
+extern spinlock_t swap_tier_lock;
+
+/* Initialization and application */
+void swap_tiers_init(void);
+ssize_t swap_tiers_sysfs_show(char *buf);
+
+int swap_tiers_add(const char *name, int prio);
+int swap_tiers_remove(const char *name);
+
+void swap_tiers_snapshot(void);
+void swap_tiers_snapshot_restore(void);
+bool swap_tiers_validate(void);
+#endif /* _SWAP_TIER_H */
mm/swapfile.c +5 -3
index e3d126602a1e..3f7225dbc6cd 100644
@@ -48,6 +48,7 @@
 #include "swap_table.h"
 #include "internal.h"
 #include "swap.h"
+#include "swap_tier.h"
 
 static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
@@ -63,7 +64,8 @@ static void move_cluster(struct swap_info_struct *si,
  *
  * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
  */
-static DEFINE_SPINLOCK(swap_lock);
+DEFINE_SPINLOCK(swap_lock);
+
 static unsigned int nr_swapfiles;
 atomic_long_t nr_swap_pages;
 /*
@@ -74,7 +76,6 @@ atomic_long_t nr_swap_pages;
 EXPORT_SYMBOL_GPL(nr_swap_pages);
 /* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
 long total_swap_pages;
-#define DEF_SWAP_PRIO  -1
 unsigned long swapfile_maximum_size;
 #ifdef CONFIG_MIGRATION
 bool swap_migration_ad_supported;
@@ -87,7 +88,7 @@ static const char Bad_offset[] = "Bad swap offset entry ";
  * all active swap_info_structs
  * protected with swap_lock, and ordered by priority.
  */
-static PLIST_HEAD(swap_active_head);
+PLIST_HEAD(swap_active_head);
 
 /*
  * all available (active, not full) swap_info_structs
@@ -3988,6 +3989,7 @@ static int __init swapfile_init(void)
 		swap_migration_ad_supported = true;
 #endif	/* CONFIG_MIGRATION */
 
+	swap_tiers_init();
 	return 0;
 }
 subsys_initcall(swapfile_init);
Signature · 1 line
2.34.1
author reviewer
#2 lore raw

PATCH v7 2/4 mm: swap: associate swap devices with tiers

In reply to Youngjun Park

This patch connects swap devices to the swap tier infrastructure, ensuring that devices are correctly assigned to tiers based on their priority.

A tier_mask is added to identify the tier membership of swap devices. Although tier-based allocation logic is not yet implemented, this mapping is necessary to track which tier a device belongs to. Upon activation, the device is assigned to a tier by matching its priority against the configured tier ranges.

The infrastructure allows dynamic modification of tiers, such as splitting or merging ranges. These operations are permitted provided that the tier assignment of already configured swap devices remains unchanged.

This patch also adds the documentation for the swap tier feature, covering the core concepts, sysfs interface usage, and configuration details.

---
 Documentation/mm/index.rst     |   1 +
 Documentation/mm/swap-tier.rst | 159 +++++++++++++++++++++++++++++++++
 MAINTAINERS                    |   1 +
 include/linux/swap.h           |   1 +
 mm/swap_state.c                |   2 +-
 mm/swap_tier.c                 | 101 ++++++++++++++++++---
 mm/swap_tier.h                 |  13 ++-
 mm/swapfile.c                  |   2 +
 8 files changed, 266 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
Documentation/mm/index.rst +1 -0
index 7aa2a8886908..a0d1447c5569 100644
@@ -21,6 +21,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
    page_reclaim
    swap
    swap-table
+   swap-tier
    page_cache
    shmfs
    oom
Documentation/mm/swap-tier.rst new file +159 -0

159 lines not on this page · raw message

MAINTAINERS +1 -0
index e3cbfedbaa5f..c008014663b3 100644
@@ -17041,6 +17041,7 @@ R:	Youngjun Park <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	Documentation/mm/swap-table.rst
+F:	Documentation/mm/swap-tier.rst
 F:	include/linux/swap.h
 F:	include/linux/swapfile.h
 F:	include/linux/swapops.h
include/linux/swap.h +1 -0
index 6d72778e6cc3..21286945770a 100644
@@ -250,6 +250,7 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
+	int tier_mask;			/* swap tier mask */
 	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
mm/swap_state.c +1 -1
index e609bbdf7e13..de285b36e31c 100644
@@ -1053,7 +1053,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 		}
 	}
 
-	if (!swap_tiers_validate()) {
+	if (!swap_tiers_update()) {
 		ret = -EINVAL;
 		goto restore;
 	}
mm/swap_tier.c +89 -12

165 lines not on this page · raw message

mm/swap_tier.h +12 -1
index a1395ec02c24..3e355f857363 100644
@@ -5,8 +5,15 @@
 #include <linux/types.h>
 #include <linux/spinlock.h>
 
+/* Forward declarations */
+struct swap_info_struct;
+
 extern spinlock_t swap_tier_lock;
 
+#define TIER_ALL_MASK		(~0)
+#define TIER_DEFAULT_IDX	(31)
+#define TIER_DEFAULT_MASK	(1U << TIER_DEFAULT_IDX)
+
 /* Initialization and application */
 void swap_tiers_init(void);
 ssize_t swap_tiers_sysfs_show(char *buf);
@@ -16,5 +23,9 @@ int swap_tiers_remove(const char *name);
 
 void swap_tiers_snapshot(void);
 void swap_tiers_snapshot_restore(void);
-bool swap_tiers_validate(void);
+bool swap_tiers_update(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+
 #endif /* _SWAP_TIER_H */
mm/swapfile.c +2 -0
index 3f7225dbc6cd..9a86ebe992f4 100644
@@ -3036,6 +3036,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 
 	/* Add back to available list */
 	add_to_avail_list(si, true);
+
+	swap_tiers_assign_dev(si);
 }
 
 /*
Signature · 1 line
2.34.1
author reviewer
#3 lore raw

PATCH v7 3/4 mm: memcontrol: add interfaces for swap tier selection

In reply to Youngjun Park

Integrate swap tier infrastructure with cgroup to allow selecting specific swap devices per cgroup.

Introduce memory.swap.tiers for configuring allowed tiers, and memory.swap.tiers.effective for exposing the effective tiers. The effective tiers are the intersection of the configured tiers and the parent's effective tiers.

Note that cgroups do not pin swap tiers, similar to cpuset and CPU hotplug, allowing configuration changes regardless of usage.

---
 Documentation/admin-guide/cgroup-v2.rst |  29 +++++++
 include/linux/memcontrol.h              |   5 ++
 mm/memcontrol.c                         |  96 +++++++++++++++++++++
 mm/swap_state.c                         |   5 +-
 mm/swap_tier.c                          | 107 +++++++++++++++++++++++-
 mm/swap_tier.h                          |  56 +++++++++++--
 6 files changed, 288 insertions(+), 10 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +29 -0
index 6efd0095ed99..08253072a252 100644
@@ -1850,6 +1850,35 @@ The following nested keys are defined.
 	Swap usage hard limit.  If a cgroup's swap usage reaches this
 	limit, anonymous memory of the cgroup will not be swapped out.
 
+  memory.swap.tiers
+        A read-write file which exists on non-root cgroups.
+        Format is similar to cgroup.subtree_control.
+
+        Controls which swap tiers this cgroup is allowed to swap
+        out to. All tiers are enabled by default.
+
+        ::
+
+            (-|+)TIER [(-|+)TIER ...]
+
+        "-" disables a tier, "+" re-enables it.
+        Entries are whitespace-delimited.
+
+        Changes here are combined with parent restrictions to
+        compute memory.swap.tiers.effective.
+
+        If a tier is removed from /sys/kernel/mm/swap/tiers,
+        any prior disable for that tier is invalidated.
+
+  memory.swap.tiers.effective
+        A read-only file which exists on non-root cgroups.
+
+        Shows the tiers this cgroup can actually swap out to.
+        This is the intersection of the parent's effective tiers
+        and this cgroup's own memory.swap.tiers configuration.
+        A child cannot enable a tier that is disabled in its
+        parent.
+
   memory.swap.events
 	A read-only flat-keyed file which exists on non-root cgroups.
 	The following entries are defined.  Unless specified
include/linux/memcontrol.h +5 -0
index bf1a6e131eca..eb33c8e30c9e 100644
@@ -287,6 +287,11 @@ struct mem_cgroup {
 	struct lru_gen_mm_list mm_list;
 #endif
 
+#ifdef CONFIG_SWAP
+	int tier_mask;
+	int tier_effective_mask;
+#endif
+
 #ifdef CONFIG_MEMCG_V1
 	/* Legacy consumer-oriented counters */
 	struct page_counter kmem;		/* v1 only */
mm/memcontrol.c +96 -0

120 lines not on this page · raw message

mm/swap_state.c +3 -2
index de285b36e31c..2fda6b61e2de 100644
@@ -1011,6 +1011,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 	char *p, *token, *name, *tmp;
 	int ret = 0;
 	short prio;
+	int mask = 0;
 
 	tmp = kstrdup(buf, GFP_KERNEL);
 	if (!tmp)
@@ -1043,7 +1044,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 				goto restore;
 			break;
 		case '-':
-			ret = swap_tiers_remove(token + 1);
+			ret = swap_tiers_remove(token + 1, &mask);
 			if (ret)
 				goto restore;
 			break;
@@ -1053,7 +1054,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 		}
 	}
 
-	if (!swap_tiers_update()) {
+	if (!swap_tiers_update(mask)) {
 		ret = -EINVAL;
 		goto restore;
 	}
mm/swap_tier.c +105 -2

130 lines not on this page · raw message

mm/swap_tier.h +50 -6

72 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#4 lore raw

PATCH v7 4/4 mm: swap: filter swap allocation by memcg tier mask

In reply to Youngjun Park

Apply memcg tier effective mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's effective mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's effective mask.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations:

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work.

---
 mm/swapfile.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
mm/swapfile.c +9 -1
index 9a86ebe992f4..1a2d29735b71 100644
@@ -1365,14 +1365,18 @@ static bool swap_alloc_fast(struct folio *folio)
 	struct swap_cluster_info *ci;
 	struct swap_info_struct *si;
 	unsigned int offset;
+	int mask = folio_tier_effective_mask(folio);
 
 	/*
 	 * Once allocated, swap_info_struct will never be completely freed,
 	 * so checking it's liveness by get_swap_device_info is enough.
 	 */
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
+	if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+		return false;
+
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!offset || !get_swap_device_info(si))
 		return false;
 
 	ci = swap_cluster_lock(si, offset);
@@ -1392,10 +1396,14 @@ static bool swap_alloc_fast(struct folio *folio)
 static void swap_alloc_slow(struct folio *folio)
 {
 	struct swap_info_struct *si, *next;
+	int mask = folio_tier_effective_mask(folio);
 
 	spin_lock(&swap_avail_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+		if (!swap_tiers_mask_test(si->tier_mask, mask))
+			continue;
+
 		/* Rotate the device and switch to a new cluster */
 		plist_requeue(&si->avail_list, &swap_avail_head);
 		spin_unlock(&swap_avail_lock);
Signature · 1 line
2.34.1
reviewer Reviewed-by on v7 4/4
#5 lore raw

Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask

Quoted 24 lines · Youngjun Park

Apply memcg tier effective mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's effective mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's effective mask.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations:

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

Quoted 8 lines

These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work.

---
 mm/swapfile.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

LGTM,

Quoted 44 lines
mm/swapfile.c +9 -1
index 9a86ebe992f4..1a2d29735b71 100644
@@ -1365,14 +1365,18 @@ static bool swap_alloc_fast(struct folio *folio)
 	struct swap_cluster_info *ci;
 	struct swap_info_struct *si;
 	unsigned int offset;
+	int mask = folio_tier_effective_mask(folio);
 
 	/*
 	 * Once allocated, swap_info_struct will never be completely freed,
 	 * so checking it's liveness by get_swap_device_info is enough.
 	 */
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
+	if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+		return false;
+
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!offset || !get_swap_device_info(si))
 		return false;
 
 	ci = swap_cluster_lock(si, offset);
@@ -1392,10 +1396,14 @@ static bool swap_alloc_fast(struct folio *folio)
 static void swap_alloc_slow(struct folio *folio)
 {
 	struct swap_info_struct *si, *next;
+	int mask = folio_tier_effective_mask(folio);
 
 	spin_lock(&swap_avail_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+		if (!swap_tiers_mask_test(si->tier_mask, mask))
+			continue;
+
 		/* Rotate the device and switch to a new cluster */
 		plist_requeue(&si->avail_list, &swap_avail_head);
 		spin_unlock(&swap_avail_lock);
Signature · 1 line
2.34.1
Reviewed-by on v7 4/4
#6 lore raw

Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask

In reply to Youngjun Park
Quoted 74 lines · Youngjun Park

Apply memcg tier effective mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's effective mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's effective mask.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations:

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work.

---
 mm/swapfile.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
mm/swapfile.c +9 -1
index 9a86ebe992f4..1a2d29735b71 100644
@@ -1365,14 +1365,18 @@ static bool swap_alloc_fast(struct folio *folio)
 	struct swap_cluster_info *ci;
 	struct swap_info_struct *si;
 	unsigned int offset;
+	int mask = folio_tier_effective_mask(folio);
 
 	/*
 	 * Once allocated, swap_info_struct will never be completely freed,
 	 * so checking it's liveness by get_swap_device_info is enough.
 	 */
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
+	if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+		return false;
+
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!offset || !get_swap_device_info(si))
 		return false;
 
 	ci = swap_cluster_lock(si, offset);
@@ -1392,10 +1396,14 @@ static bool swap_alloc_fast(struct folio *folio)
 static void swap_alloc_slow(struct folio *folio)
 {
 	struct swap_info_struct *si, *next;
+	int mask = folio_tier_effective_mask(folio);
 
 	spin_lock(&swap_avail_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+		if (!swap_tiers_mask_test(si->tier_mask, mask))
+			continue;
+
 		/* Rotate the device and switch to a new cluster */
 		plist_requeue(&si->avail_list, &swap_avail_head);
 		spin_unlock(&swap_avail_lock);
Signature · 1 line
2.34.1

This part looks good to me, the known limitations are not regression and only for tiering, so can be improved later, and we do have plan to refine the priority / rotation / pcp cluster so they aligns well.

reviewer Reviewed-by on v7 4/4
#7 lore raw

Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask

In reply to Youngjun Park
Quoted 19 lines · Youngjun Park

Apply memcg tier effective mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's effective mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's effective mask.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations:

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

Quoted 5 lines

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

I had very similar issues when I tried hacking vswap on top of swap table too... It's even worse over there because it's not just performance - vswap needs special handling in certain cases, and in some places cannot be used at all (for e.g in zswap writeback). I ended up having to add separate caching for vswap device:

https://lore.kernel.org/all/[email protected]/

How expensive is it to add per-cpu caching for each device :(

Anyway, as a first step, this LGTM. Reviewing from swap's mechanism perspective, and leaving the cgroup side to memcg folks:

reviewer
#8 lore raw

Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask

Quoted 3 lines · Nhat Pham

How expensive is it to add per-cpu caching for each device :(

to clarify - a percpu_swap_cluster per si for every si.

Quoted 1 line

... or for each tier (assuming devices in each tier share the same performance characteristics, and could be used interchangeably?).

Basically:

struct percpu_swap_cluster {
    struct swap_info_struct *si[MAX_SWAPTIER][SWAP_NR_ORDERS];
    unsigned long offset[MAX_SWAPTIER][SWAP_NR_ORDERS];
    local_lock_t lock;
};

Seems like 4 is the default number of tier right? So the extra overhead is just (nr cpu) * 10 * 3 * (sizeof(unsigned long) + sizeof(*ptr)) or wev?

author reviewer
#9 lore raw

Re: [PATCH v7 4/4] mm: swap: filter swap allocation by memcg tier mask

Quoted 20 lines · Nhat Pham
Quoted 3 lines · Nhat Pham

How expensive is it to add per-cpu caching for each device :(

to clarify - a percpu_swap_cluster per si for every si.

Quoted 1 line

... or for each tier (assuming devices in each tier share the same performance characteristics, and could be used interchangeably?).

Basically:

struct percpu_swap_cluster {
    struct swap_info_struct *si[MAX_SWAPTIER][SWAP_NR_ORDERS];
    unsigned long offset[MAX_SWAPTIER][SWAP_NR_ORDERS];
    local_lock_t lock;
};
Quoted 3 lines

Seems like 4 is the default number of tier right? So the extra overhead is just (nr cpu) * 10 * 3 * (sizeof(unsigned long) + sizeof(*ptr)) or wev?

I agree. I actually considered the idea of a tier-centric cache as well.

You might remember that in the previous "per cgroup swap priority" patchset, I implemented per-cpu caches per priority, which is essentially the same as having them per tier.

However, as you agreed in another thread, including this optimization right now might be a bit premature. If the core swap tier idea gets merged, I plan to explore this optimization further as follow-up work.

Thanks
Youngjun Park

maintainer
#10 lore raw
In reply to Youngjun Park
Quoted 2 lines · On Wed, 27 May 2026 15:22:43 +0900 Youngjun Park <[email protected]> wrote:

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

One question from Sashiko. Minor, but easy to address.

	https://sashiko.dev/#/patchset/[email protected]

I'm reluctant to add a new feature patchset at this time - we have a lot
already and we're at -rc5. What do others think?

#11 lore raw
Quoted 11 lines · Andrew Morton
Quoted 2 lines · On Wed, 27 May 2026 15:22:43 +0900 Youngjun Park <[email protected]> wrote:

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

One question from Sashiko. Minor, but easy to address.

        https://sashiko.dev/#/patchset/[email protected]

I'm reluctant to add a new feature patchset at this time - we have a lot
already and we're at -rc5. What do others think?

This adds new user-visible interfaces and I think we didn't reach an agreement on them. I specifically recall Shakeel (and perhaps other memcg folks) having questions about the memcg interface, and I don't see any Acks on that patch. I don't think this should be included.

author reviewer
#12 lore raw
In reply to Andrew Morton
Quoted 7 lines · Andrew Morton
Quoted 2 lines · On Wed, 27 May 2026 15:22:43 +0900 Youngjun Park <[email protected]> wrote:

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

One question from Sashiko. Minor, but easy to address.

	https://sashiko.dev/#/patchset/[email protected]

Thanks, Andrew. That is a valid concern and definitely needs to be fixed. I will address it in the next version.

Quoted 2 lines

I'm reluctant to add a new feature patchset at this time - we have a lot
already and we're at -rc5. What do others think?

I will wait to hear others' thoughts on this.

Thanks,
Youngjun Park

reviewer
#13 lore raw
In reply to Youngjun Park
Quoted 126 lines · Youngjun Park

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

I revisited the design (see Design Rationale). Since our use case fits best with a memcg-based model, the implementation remains within memcg and preserves its resource accounting semantics.

Alternatives considered:

1. A separate sysfs interface under swap. (Workable. But, it would still need to reference memcg paths, and fully decoupling it would add swap-layer logic to manage memcgs, making it secondary option.)

2. Making the feature non-default.

Other interfaces were also reviewed. Aside from sysfs and BPF, the options involve trade-offs and are largely design choices. BPF was excluded due to possible disablement on our embedded platform, though future extension remains possible.

Overview
========

Swap Tiers group swap devices into performance classes (e.g. NVMe, HDD, Network) and allow per-memcg selection of which tiers to use. This mechanism was suggested by Chris Li.

Design Rationale
================

Swap tier selection is attached to memcg. A child cgroup may select a subset of the parent's allowed tiers.

This
- Preserves cgroup inheritance semantics (boundary at parent, refinement at child).
- Reuses memcg, which already groups processes and enforces hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.

Placing tier control outside memcg (e.g., via BPF, syscalls, or madvise) would allow swap preference to diverge from the memcg hierarchy. Integrating it into memcg keeps the swap policy consistent with existing memory ownership semantics. There are also real use cases built around memcg.

In the future, this can be extended to other interfaces to cover additional use cases.

I believe a memcg-based swap control is a good starting point before such extensions.

Use Cases
=========

#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)

The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers according to latency requirements.

This prevents latency-sensitive workloads from being swapped to slow devices used by background workloads.

#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)

The parent (job) has access to both zswap and SSD tiers. The child (VMM guest memory) selects SSD as its swap tier via memory.swap.tiers. In this deployment, swap device selection happens at the child level from the parent's available set.

#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)

Each child uses a different tier. Since swap paths are separated per tier, synchronization overhead between the two workloads is reduced.

Future extension
================

#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a round-robin fashion. Per-tier policy files under /sys/kernel/mm/swap/tiers/ can control how devices within a tier are selected (e.g. round-robin, weighted).

#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single tier. The current interface defines only tier assignment; it does not yet define when or how pages move between tiers. Two triggering models are possible:

(a) User-triggered: userspace explicitly initiates migration between tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at appropriate points such as reclaim or refault.

#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process swap. Or we can use it as BPF program.

Experimentation
===============

Tested on our internal platform using NBD as a separate swap tier. Our first production's simple usecase.

Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications

Quoted 8 lines

Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)

Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks

Bit late to the party - working on my review backlog right now :)

I see some parallels with this and memory tiering work being done. One future line of work could be considering how to ensure fairness when multiple cgroups share same tiers:

https://lwn.net/Articles/1073400/

I can imagine a scenario where one noisy neighbor eagerly swaps first and occupy all the space in the faster tier(s), pushing the other colocated tenants to the slower tier(s). We might need to figure out a way to ensure fairness here (while letting cgroups occupy fast swap backends opportunistically if there is no resources scarcity).

author reviewer
#14 lore raw
Quoted 136 lines · Nhat Pham
Quoted 126 lines · Youngjun Park

This is v7 of the swap tier series addressing review feedback. The cover letter has been simplified.

I revisited the design (see Design Rationale). Since our use case fits best with a memcg-based model, the implementation remains within memcg and preserves its resource accounting semantics.

Alternatives considered:

1. A separate sysfs interface under swap. (Workable. But, it would still need to reference memcg paths, and fully decoupling it would add swap-layer logic to manage memcgs, making it secondary option.)

2. Making the feature non-default.

Other interfaces were also reviewed. Aside from sysfs and BPF, the options involve trade-offs and are largely design choices. BPF was excluded due to possible disablement on our embedded platform, though future extension remains possible.

Overview
========

Swap Tiers group swap devices into performance classes (e.g. NVMe, HDD, Network) and allow per-memcg selection of which tiers to use. This mechanism was suggested by Chris Li.

Design Rationale
================

Swap tier selection is attached to memcg. A child cgroup may select a subset of the parent's allowed tiers.

This
- Preserves cgroup inheritance semantics (boundary at parent, refinement at child).
- Reuses memcg, which already groups processes and enforces hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.

Placing tier control outside memcg (e.g., via BPF, syscalls, or madvise) would allow swap preference to diverge from the memcg hierarchy. Integrating it into memcg keeps the swap policy consistent with existing memory ownership semantics. There are also real use cases built around memcg.

In the future, this can be extended to other interfaces to cover additional use cases.

I believe a memcg-based swap control is a good starting point before such extensions.

Use Cases
=========

#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)

The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers according to latency requirements.

This prevents latency-sensitive workloads from being swapped to slow devices used by background workloads.

#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)

The parent (job) has access to both zswap and SSD tiers. The child (VMM guest memory) selects SSD as its swap tier via memory.swap.tiers. In this deployment, swap device selection happens at the child level from the parent's available set.

#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)

Each child uses a different tier. Since swap paths are separated per tier, synchronization overhead between the two workloads is reduced.

Future extension
================

#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a round-robin fashion. Per-tier policy files under /sys/kernel/mm/swap/tiers/ can control how devices within a tier are selected (e.g. round-robin, weighted).

#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single tier. The current interface defines only tier assignment; it does not yet define when or how pages move between tiers. Two triggering models are possible:

(a) User-triggered: userspace explicitly initiates migration between tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at appropriate points such as reclaim or refault.

#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process swap. Or we can use it as BPF program.

Experimentation
===============

Tested on our internal platform using NBD as a separate swap tier. Our first production's simple usecase.

Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications

Quoted 8 lines

Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)

Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks

Quoted 7 lines

Bit late to the party - working on my review backlog right now :)

I see some parallels with this and memory tiering work being done. One future line of work could be considering how to ensure fairness when multiple cgroups share same tiers:

https://lwn.net/Articles/1073400/

Hi Nhat,

Thanks for bringing this up. I took a quick look at the link, and I agree with your point. We could probably use a similar proposed mechanism (e.g., setting min/max limits per tier) to handle promotion and demotion in the future.

This also suggests that keeping the swap tier limits inside the memcg interface is a reasonable approach. It aligns well with such future work. If it were implemented in other layers (e.g., prctl, sysfs, or BPF), we would likely have to revisit its integration with memcg someday anyway.

Quoted 4 lines

and occupy all the space in the faster tier(s), pushing the other colocated tenants to the slower tier(s). We might need to figure out a way to ensure fairness here (while letting cgroups occupy fast swap backends opportunistically if there is no resources scarcity).

Agreed. Ensuring fairness will be essential when we eventually expand the promotion and demotion mechanisms across swap tiers.

Thanks,
Youngjun