Skip to contentSkip to conversation

mm/swap, memcg: Introduce swap tiers for cgroup based swap control

PATCH Superseded Youngjun Park 6 participants 42 messages Jul 13 – Jul 23

Subsystems (8)

v10 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Youngjun Park

author reviewer
#0 lore raw

v10 cover mm/swap, memcg: Introduce swap tiers for cgroup based swap control

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

* Zswap tiering [2]:
Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.
* Vswap tiering [3]:
Vswap should be handled transparently to the user. Vswap itself will not be tiered. But, someday supported if there is strong and real usecase.
* Relationship with zswap.writeback [4]:
If zswap tiering is introduced, it could replace the zswap-only tier. However, since zswap cannot be tiered independently without vswap, it is still needed for non-vswap cases.
* Tier demotion [5]:
A separate interface like memory.swap.tiers.demotion might be needed. For now, we only support 0/max to enable/disable tiers. A future "auto" mode could scale a per-tier limit from swapfile size and memory.swap.max, similar to the direction memory tiering is heading in; the exact default-scaling behaviour is still under discussion.

I plan to apply the swap tier infrastructure and the first use case (cgroup-based swap control) first, and continue following up on the discussions above.

Overview
========

Swap Tiers group swap devices into performance classes (e.g. NVMe, HDD, Network) and allow per-memcg selection of which tiers to use. This mechanism was suggested by Chris Li.

Design Rationale
================

Swap tier selection is attached to memcg. A child cgroup may select a subset of the parent's allowed tiers.

This
- Preserves cgroup inheritance semantics (boundary at parent, refinement at child).
- Reuses memcg, which already groups processes and enforces hierarchical memory limits.
- Aligns with existing memcg swap controls (e.g. swap.max, zswap.writeback)
- Avoids introducing a parallel swap control hierarchy.

Placing tier control outside memcg (e.g., via BPF, syscalls, or madvise) would allow swap preference to diverge from the memcg hierarchy. Integrating it into memcg keeps the swap policy consistent with existing memory ownership semantics. There are also real use cases built around memcg.

In the future, this can be extended to other interfaces to cover additional use cases.

I believe a memcg-based swap control is a good starting point before such extensions.

Use Cases
=========

#1: Latency separation (our primary deployment scenario)
[ / ]
|
+-- latency-sensitive workload (fast tier)
+-- background workload (slow tier)

The parent defines the memory boundary.
Each workload selects a swap tier via memory.swap.tiers.max according to latency requirements.

This prevents latency-sensitive workloads from being swapped to slow devices used by background workloads.

#2: Per-VM swap selection (Chris Li's deployment scenario)
[ / ]
|
+-- [ Job on VM ] (tiers: zswap, SSD)
|
+-- [ VMM guest memory ] (tiers: SSD)

The parent (job) has access to both zswap and SSD tiers. The child (VMM guest memory) selects SSD as its swap tier via memory.swap.tiers.max. In this deployment, swap device selection happens at the child level from the parent's available set.

#3: Tier isolation for reduced contention (hypothetical)
[ / ] (tiers: A, B)
|
+-- workload X (tiers: A)
+-- workload Y (tiers: B)

Each child uses a different tier. Since swap paths are separated per tier, synchronization overhead between the two workloads is reduced.

Future extension (Follow up)
============================

#1: Intra-tier distribution policy:
Currently, swap devices with the same priority are allocated in a round-robin fashion. Per-tier policy files under /sys/kernel/mm/swap/tiers/ can control how devices within a tier are selected (e.g. round-robin, weighted).

#2: Inter-tier promotion and demotion:
Promotion and demotion apply between tiers, not within a single tier. The current interface defines only tier assignment; it does not yet define when or how pages move between tiers. Two triggering models are possible:

(a) User-triggered: userspace explicitly initiates migration between tiers (e.g. via a new interface or existing move_pages semantics).
(b) Kernel-triggered: the kernel moves pages between tiers at appropriate points such as reclaim or refault.

#3: Per-VMA, per-process swap and BPF:
Not just for memcg based swap, possible to extend Per-VMA or per-process swap. Or we can use it as BPF program.

#4: Zswap and vswap tiering:
Tiering applies to the vswap + zswap combination.

#5: Vswap on/off control:
Currently not supported. If a strong use case arises where vswap needs to be controlled by memcg, the tier interface could be used for it.

#6: Per-CPU swap allocation caching:
Per-si/per-tier per-CPU caching of allocations to reduce contention in the tier-filtered allocation path.

Experimentation
===============

Tested on our internal platform using NBD as a separate swap tier. Our first production's simple usecase.

Without tiers:
- No selective control over flash wear
- Cannot selectively assign NBD to specific applications

Cold launch improvement (preloaded vs. baseline):
- App A: 13.17s -> 4.18s (68%)
- App B: 5.60s -> 1.12s (80%)
- App C: 10.25s -> 2.00s (80%)

Performance impact with no tiers configured:
<1% regression in kernel build and vm-scalability benchmarks

Change log
===========

v10
- selftests: applied the Sashiko review fixes, and dropped redundant comments and dead code. (#5, #6 patches)
- Rebased on recent mm-new.
- v9 link: https://lore.kernel.org/linux-mm/[email protected]/

v9
- Added selftests (per Nhat's request):
- selftests/mm: swap tier configuration test for /sys/kernel/mm/swap/tiers.(#5 patch)
- selftests/cgroup: swap tier routing test for memory.swap.tiers.max. (#6 patch)
- Removed the redundant rcu_read_lock() around the memcg tier-mask tree walk; for_each_mem_cgroup_tree() already takes RCU internally and returns each memcg with a reference held. (#3 patch)
- Sashiko review: swap_sync_discard() now honors the memcg tier mask, so the discard fallback no longer drains clusters on disallowed tiers. Left as-is:
the cgroup tree walk under spinlock (bounded by cgroup.max.descendants, an admin-controlled limit, and triggered only by infrequent tier writes) and the pre-existing swap_avail_lock drop in swap_alloc_slow(). (#4 patch)
- Dropped patch #4's Reviewed-by tags (Nhat, Kairui, Baoquan): the swap_sync_discard() change above modifies that patch (the tier mask is now passed as a parameter into the alloc and discard paths), so the earlier tags no longer apply. Re-review would be welcome.
- v8 link: https://lore.kernel.org/linux-mm/[email protected]/

v8
- Changed the memcg interface to memory.swap.tiers.max.
Values are '0' (disable) and 'max' (enable). Default is 'max'.
- Addressed Sashiko's review: Update the mask value atomically at once and read the mask value while grabbing lock.
- Collected review tags from Kairui and Nhat.
- Rebase on recent mm-new
- v7 link: https://lore.kernel.org/linux-mm/[email protected]/

v7
- Collect Baoquan's review tag
- Baoquan's feedback on fixing improper comment
- Minor code adjustments per Baoquan's feedback.
- Rebase on recent mm-new
- v6 link: https://lore.kernel.org/linux-mm/[email protected]/

v6
- Sashiko AI review fixes
- Fix batch parsing error path to restore snapshot before exit
- Reject overlong tier names to prevent truncated duplicates
- Avoid restoring raw list_head via memcpy (stale pointer risk)
- Ensure early parse errors do not skip DEF_SWAP_PRIO validation
- Use (1U << TIER_DEFAULT_IDX) to avoid signed shift UB
- Defer tier mask inheritance to css_online() to close race window
- Add READ_ONCE()/WRITE_ONCE() for tier mask accesses
- Other fixes
- Fix build error reintroduced due to missing v5 change
- Fix WARNING in folio_tier_effective_mask by adding rcu_read_lock()
- default number of swap tier max (change to 32->31, for reserving last bit)
- commit message refinement.
- rebased on recently mm-new
- v5 link: https://lore.kernel.org/linux-mm/[email protected]/

v5
- Fixed build errors reported in v4
- rebased on up to date mm-new
- Minor cleanups
- Design docs with validation (by Shakeel Butt discussion)
- v4 link : https://lore.kernel.org/linux-mm/[email protected]/

v4
- Simplified control flow and indentation
- Added CONFIG option for MAX_SWAPTIER (default: 4)
- Added memory.swap.tiers.effective interface
- Reworked save/restore logic into snapshot/rollback model
- Removed tier priority modification support (deferred)
- Improved validation and fixed edge cases
- Rebased onto latest mm-new
- RFC v3 link: https://lore.kernel.org/linux-mm/[email protected]/

RFC v1 ~ v3
- Change the direction after discussion with Chris-Li
- apply some LPC feedback.
- RFC v2 - https://lore.kernel.org/linux-mm/[email protected]/
- RFC v1 - https://lore.kernel.org/linux-mm/[email protected]/

Earlier Approach (per cgroup swap priority)
- v1: https://lore.kernel.org/linux-mm/[email protected]/
- RFC: https://lore.kernel.org/linux-mm/aEvLjEInMQC7hEyh@yjaykim-PowerEdge-T330/T/#mbbb6a5e9e30843097e1f5f65fb98f31d582b973d

Reference
=========

[1] https://lore.kernel.org/linux-doc/[email protected]/
[2] https://lore.kernel.org/linux-mm/CAKEwX=Nz9SWcEVQGQjHN8P8OANJY4BG0w+iQOzoNOWuteoVjAg@mail.gmail.com/
[3] https://lore.kernel.org/cgroups/CAKEwX=O23a4iWBZoewKVb8QqODte6r3Xijckw3_oCJNoiO9M5A@mail.gmail.com/
[4] https://lore.kernel.org/linux-mm/CAO9r8zOg0OP1Ak1v7CRzSfQq0D8b4Dw+_T0Jui6YTM_KwQQNOA@mail.gmail.com/
[5] https://lore.kernel.org/linux-mm/CAO9r8zNi4-QC4sUi=xXWHt9WMeG39mbyoSf8kON9vLOZ=cbCmw@mail.gmail.com/

Youngjun Park (6):
mm: swap: introduce swap tier infrastructure
mm: swap: associate swap devices with tiers
mm: memcontrol: add interface for swap tier selection
mm: swap: filter swap allocation by memcg tier mask selftests/mm: add a swap tier configuration test selftests/cgroup: add a swap tier routing test

 Documentation/admin-guide/cgroup-v2.rst       |  20 +
 Documentation/mm/index.rst                    |   1 +
 Documentation/mm/swap-tier.rst                | 159 ++++++
 MAINTAINERS                                   |   3 +
 include/linux/memcontrol.h                    |   5 +
 include/linux/swap.h                          |   1 +
 mm/Kconfig                                    |  12 +
 mm/Makefile                                   |   2 +-
 mm/memcontrol.c                               |  67 +++
 mm/swap.h                                     |   4 +
 mm/swap_state.c                               |  75 +++
 mm/swap_tier.c                                | 477 ++++++++++++++++
 mm/swap_tier.h                                |  76 +++
 mm/swapfile.c                                 |  34 +-
 tools/testing/selftests/cgroup/.gitignore     |   1 +
 tools/testing/selftests/cgroup/Makefile       |   2 +
 tools/testing/selftests/cgroup/config         |   2 +
 .../selftests/cgroup/test_swap_tiers.c        | 509 ++++++++++++++++++
 tools/testing/selftests/mm/.gitignore         |   1 +
 tools/testing/selftests/mm/Makefile           |   1 +
 tools/testing/selftests/mm/config             |   2 +
 tools/testing/selftests/mm/run_vmtests.sh     |   5 +
 tools/testing/selftests/mm/swap_tier.c        | 337 ++++++++++++
 23 files changed, 1785 insertions(+), 11 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h
 create mode 100644 tools/testing/selftests/cgroup/test_swap_tiers.c
 create mode 100644 tools/testing/selftests/mm/swap_tier.c

base-commit: 61cccb8363fcc282d4ae0555b8739dd227f5ad0b

Signature · 1 line
2.34.1
author reviewer
#1 lore raw

PATCH v10 1/6 mm: swap: introduce swap tier infrastructure

This patch introduces the "Swap tier" concept, which serves as an abstraction layer for managing swap devices based on their performance characteristics (e.g., NVMe, HDD, Network swap).

Swap tiers are user-named groups representing priority ranges. Tier names must consist of alphanumeric characters and underscores. These tiers collectively cover the entire priority space from -1 (DEF_SWAP_PRIO) to SHRT_MAX.

To configure tiers, a new sysfs interface is exposed at /sys/kernel/mm/swap/tiers. The input parser evaluates commands from left to right and supports batch input, allowing users to add or remove multiple tiers in a single write operation.

Tier management enforces continuous priority ranges anchored by start priorities. Operations trigger range splitting or merging, but overwriting start priorities is forbidden. Merging expands lower tiers upwards to preserve configured start priorities, except when removing DEF_SWAP_PRIO, which merges downwards.

---
 MAINTAINERS     |   2 +
 mm/Kconfig      |  12 ++
 mm/Makefile     |   2 +-
 mm/swap.h       |   4 +
 mm/swap_state.c |  74 ++++++++++++
 mm/swap_tier.c  | 302 ++++++++++++++++++++++++++++++++++++++++++++++++
 mm/swap_tier.h  |  20 ++++
 mm/swapfile.c   |   8 +-
 8 files changed, 420 insertions(+), 4 deletions(-)
 create mode 100644 mm/swap_tier.c
 create mode 100644 mm/swap_tier.h
MAINTAINERS +2 -0
index 8813d5d7eb0c..e94d1af17c39 100644
@@ -17244,6 +17244,8 @@ F:	mm/page_io.c
 F:	mm/swap.h
 F:	mm/swap_table.h
 F:	mm/swap_state.c
+F:	mm/swap_tier.c
+F:	mm/swap_tier.h
 F:	mm/swapfile.c
 
 MEMORY MANAGEMENT - THP (TRANSPARENT HUGE PAGE)
mm/Kconfig +12 -0
index 69c4247306cc..0b6ff5d882dc 100644
@@ -19,6 +19,18 @@ menuconfig SWAP
 	  used to provide more virtual memory than the actual RAM present
 	  in your computer.  If unsure say Y.
 
+config NR_SWAP_TIERS
+        int "Number of swap device tiers"
+        depends on SWAP
+        default 4
+        range 1 31
+        help
+          Sets the number of swap device tiers. Swap devices are
+          grouped into tiers based on their priority, allowing the
+          system to prefer faster devices over slower ones.
+
+          If unsure, say 4.
+
 config ZSWAP
 	bool "Compressed cache for swap pages"
 	depends on SWAP
mm/Makefile +1 -1
index ab37ef428d98..6c7522d04485 100644
@@ -75,7 +75,7 @@ ifdef CONFIG_MMU
 	obj-$(CONFIG_ADVISE_SYSCALLS)	+= madvise.o
 endif
 
-obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o
+obj-$(CONFIG_SWAP)	+= page_io.o swap_state.o swapfile.o swap_tier.o
 obj-$(CONFIG_ZSWAP)	+= zswap.o
 obj-$(CONFIG_HAS_DMA)	+= dmapool.o
 obj-$(CONFIG_HUGETLBFS)	+= hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
mm/swap.h +4 -0
index b51ad3071a73..7ceab672a860 100644
@@ -32,6 +32,10 @@ struct swap_memcg_table;
 #define swap_entry_order(order)	0
 #endif
 
+#define DEF_SWAP_PRIO  -1
+
+extern spinlock_t swap_lock;
+extern struct plist_head swap_active_head;
 extern struct swap_info_struct *swap_info[];
 
 /*
mm/swap_state.c +74 -0

88 lines not on this page · raw message

mm/swap_tier.c new file +302 -0

302 lines not on this page · raw message

mm/swap_tier.h new file +20 -0
new file mode 100644
index 000000000000..a1395ec02c24
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _SWAP_TIER_H
+#define _SWAP_TIER_H
+
+#include <linux/types.h>
+#include <linux/spinlock.h>
+
+extern spinlock_t swap_tier_lock;
+
+/* Initialization and application */
+void swap_tiers_init(void);
+ssize_t swap_tiers_sysfs_show(char *buf);
+
+int swap_tiers_add(const char *name, int prio);
+int swap_tiers_remove(const char *name);
+
+void swap_tiers_snapshot(void);
+void swap_tiers_snapshot_restore(void);
+bool swap_tiers_validate(void);
+#endif /* _SWAP_TIER_H */
mm/swapfile.c +5 -3
index 72952491e9cf..ff567ad893a4 100644
@@ -48,6 +48,7 @@
 #include "swap_table.h"
 #include "internal.h"
 #include "swap.h"
+#include "swap_tier.h"
 
 static void swap_range_alloc(struct swap_info_struct *si,
 			     unsigned int nr_entries);
@@ -63,7 +64,8 @@ static void move_cluster(struct swap_info_struct *si,
  *
  * Also protects swap_active_head total_swap_pages, and the SWP_WRITEOK flag.
  */
-static DEFINE_SPINLOCK(swap_lock);
+DEFINE_SPINLOCK(swap_lock);
+
 static unsigned int nr_swapfiles;
 atomic_long_t nr_swap_pages;
 /*
@@ -74,7 +76,6 @@ atomic_long_t nr_swap_pages;
 EXPORT_SYMBOL_GPL(nr_swap_pages);
 /* protected with swap_lock. reading in vm_swap_full() doesn't need lock */
 long total_swap_pages;
-#define DEF_SWAP_PRIO  -1
 unsigned long swapfile_maximum_size;
 #ifdef CONFIG_MIGRATION
 bool swap_migration_ad_supported;
@@ -87,7 +88,7 @@ static const char Bad_offset[] = "Bad swap offset entry ";
  * all active swap_info_structs
  * protected with swap_lock, and ordered by priority.
  */
-static PLIST_HEAD(swap_active_head);
+PLIST_HEAD(swap_active_head);
 
 /*
  * all available (active, not full) swap_info_structs
@@ -3993,6 +3994,7 @@ static int __init swapfile_init(void)
 		swap_migration_ad_supported = true;
 #endif	/* CONFIG_MIGRATION */
 
+	swap_tiers_init();
 	return 0;
 }
 subsys_initcall(swapfile_init);
Signature · 1 line
2.34.1
author reviewer
#2 lore raw

PATCH v10 2/6 mm: swap: associate swap devices with tiers

In reply to Youngjun Park

This patch connects swap devices to the swap tier infrastructure, ensuring that devices are correctly assigned to tiers based on their priority.

A tier_mask is added to identify the tier membership of swap devices. Although tier-based allocation logic is not yet implemented, this mapping is necessary to track which tier a device belongs to. Upon activation, the device is assigned to a tier by matching its priority against the configured tier ranges.

The infrastructure allows dynamic modification of tiers, such as splitting or merging ranges. These operations are permitted provided that the tier assignment of already configured swap devices remains unchanged.

This patch also adds the documentation for the swap tier feature, covering the core concepts, sysfs interface usage, and configuration details.

---
 Documentation/mm/index.rst     |   1 +
 Documentation/mm/swap-tier.rst | 150 +++++++++++++++++++++++++++++++++
 MAINTAINERS                    |   1 +
 include/linux/swap.h           |   1 +
 mm/swap_state.c                |   2 +-
 mm/swap_tier.c                 | 101 +++++++++++++++++++---
 mm/swap_tier.h                 |  13 ++-
 mm/swapfile.c                  |   2 +
 8 files changed, 257 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
Documentation/mm/index.rst +1 -0
index 13a79f5d092c..6afc45cd4b3d 100644
@@ -34,6 +34,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
    page_reclaim
    swap
    swap-table
+   swap-tier
    page_cache
    shmfs
    oom
Documentation/mm/swap-tier.rst new file +150 -0

150 lines not on this page · raw message

MAINTAINERS +1 -0
index e94d1af17c39..4485a0650984 100644
@@ -17237,6 +17237,7 @@ L:	[email protected]
 S:	Maintained
 F:	Documentation/ABI/testing/sysfs-kernel-mm-swap
 F:	Documentation/mm/swap-table.rst
+F:	Documentation/mm/swap-tier.rst
 F:	include/linux/swap.h
 F:	include/linux/swapfile.h
 F:	include/linux/swapops.h
include/linux/swap.h +1 -0
index 4427c7aa16dc..2d647a8fa60f 100644
@@ -244,6 +244,7 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
+	int tier_mask;			/* swap tier mask */
 	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
mm/swap_state.c +1 -1
index c18ff741f2e0..dcec1a6c92bd 100644
@@ -1095,7 +1095,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 		}
 	}
 
-	if (!swap_tiers_validate()) {
+	if (!swap_tiers_update()) {
 		ret = -EINVAL;
 		goto restore;
 	}
mm/swap_tier.c +89 -12

165 lines not on this page · raw message

mm/swap_tier.h +12 -1
index a1395ec02c24..3e355f857363 100644
@@ -5,8 +5,15 @@
 #include <linux/types.h>
 #include <linux/spinlock.h>
 
+/* Forward declarations */
+struct swap_info_struct;
+
 extern spinlock_t swap_tier_lock;
 
+#define TIER_ALL_MASK		(~0)
+#define TIER_DEFAULT_IDX	(31)
+#define TIER_DEFAULT_MASK	(1U << TIER_DEFAULT_IDX)
+
 /* Initialization and application */
 void swap_tiers_init(void);
 ssize_t swap_tiers_sysfs_show(char *buf);
@@ -16,5 +23,9 @@ int swap_tiers_remove(const char *name);
 
 void swap_tiers_snapshot(void);
 void swap_tiers_snapshot_restore(void);
-bool swap_tiers_validate(void);
+bool swap_tiers_update(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+
 #endif /* _SWAP_TIER_H */
mm/swapfile.c +2 -0
index ff567ad893a4..f3cff586cf30 100644
@@ -3041,6 +3041,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 
 	/* Add back to available list */
 	add_to_avail_list(si, true);
+
+	swap_tiers_assign_dev(si);
 }
 
 /*
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v10 2/6] mm: swap: associate swap devices with tiers

Quoted 452 lines · On Mon, 13 Jul 2026 11:56:40 +0900 Youngjun Park <[email protected]> wrote:

This patch connects swap devices to the swap tier infrastructure, ensuring that devices are correctly assigned to tiers based on their priority.

A tier_mask is added to identify the tier membership of swap devices. Although tier-based allocation logic is not yet implemented, this mapping is necessary to track which tier a device belongs to. Upon activation, the device is assigned to a tier by matching its priority against the configured tier ranges.

The infrastructure allows dynamic modification of tiers, such as splitting or merging ranges. These operations are permitted provided that the tier assignment of already configured swap devices remains unchanged.

This patch also adds the documentation for the swap tier feature, covering the core concepts, sysfs interface usage, and configuration details.

---
 Documentation/mm/index.rst     |   1 +
 Documentation/mm/swap-tier.rst | 150 +++++++++++++++++++++++++++++++++
 MAINTAINERS                    |   1 +
 include/linux/swap.h           |   1 +
 mm/swap_state.c                |   2 +-
 mm/swap_tier.c                 | 101 +++++++++++++++++++---
 mm/swap_tier.h                 |  13 ++-
 mm/swapfile.c                  |   2 +
 8 files changed, 257 insertions(+), 14 deletions(-)
 create mode 100644 Documentation/mm/swap-tier.rst
Documentation/mm/index.rst +1 -0
index 13a79f5d092c..6afc45cd4b3d 100644
@@ -34,6 +34,7 @@ see the :doc:`admin guide <../admin-guide/mm/index>`.
    page_reclaim
    swap
    swap-table
+   swap-tier
    page_cache
    shmfs
    oom
Documentation/mm/swap-tier.rst new file +150 -0
new file mode 100644
index 000000000000..0fb4a1153a67
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+:Author: Chris Li <[email protected]> Youngjun Park <[email protected]>
+
+==========
+Swap Tier
+==========
+
+Swap tier is a collection of user-named groups classified by priority ranges.
+It acts as a facilitation layer, allowing users to manage swap devices based
+on their speeds.
+
+Users are encouraged to assign swap device priorities according to device
+speed to fully utilize this feature. While the current implementation is
+integrated with cgroups, the concept is designed to be extensible for other
+subsystems in the future.
+
+Priority Range
+--------------
+
+The specified tiers must cover the entire priority range from -1
+(DEF_SWAP_PRIO) to SHRT_MAX.
+
+Consistency
+-----------
+
+Tier consistency is guaranteed with a focus on maximizing flexibility. When a
+swap device is activated within a tier range, the tier covering that device's
+priority is guaranteed not to disappear or change while the device remains
+active. Adding a new tier may split the range of an existing tier, but the
+active device's tier assignment remains unchanged.
+
+However, specifying a tier in a cgroup does not guarantee the tier's existence.
+Consequently, the corresponding tier can disappear at any time.
+
+Configuration Interface
+-----------------------
+
+The swap tiers can be configured via the following interface:
+
+/sys/kernel/mm/swap/tiers
+
+Operations can be performed using the following syntax:
+
+* Add:    ``+"<tiername>":"<start_priority>"``
+* Remove: ``-"<tiername>"``
+
+Tier names must consist of alphanumeric characters and underscores. Multiple
+operations can be provided in a single write, separated by commas (",") or
+whitespace (spaces, tabs, newlines).
+
+When configuring tiers, the specified value represents the **start priority**
+of that tier. The end priority is automatically determined by the start
+priority of the next higher tier. Consequently, adding a tier
+automatically adjusts the ranges of adjacent tiers to ensure continuity.
+
+Examples
+--------
+
+**1. Initialization**
+
+A tier starting at -1 is mandatory to cover the entire priority range up to
+SHRT_MAX. In this example, 'HDD' starts at 50, and 'NET' covers the remaining
+lower range starting from -1.
+
+::
+
+    # echo "+HDD:50, +NET:-1" > /sys/kernel/mm/swap/tiers
+    # cat /sys/kernel/mm/swap/tiers
+    Name             Idx   PrioStart   PrioEnd
+    HDD              0     50          32767
+    NET              1     -1          49
+
+**2. Adding a New Tier (split)**
+
+A new tier 'SSD' is added at priority 100, splitting the existing 'HDD' tier.
+The ranges are automatically recalculated:
+
+* 'SSD' takes the top range (100 to SHRT_MAX).
+* 'HDD' is adjusted to the range between 'NET' and 'SSD' (50 to 99).
+* 'NET' remains unchanged (-1 to 49).
+
+::
+
+    # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+    # cat /sys/kernel/mm/swap/tiers
+    Name             Idx   PrioStart   PrioEnd
+    SSD              2     100         32767
+    HDD              0     50          99
+    NET              1     -1          49
+
+**3. Removal (merge)**
+
+Tiers can be removed using the '-' prefix.
+::
+
+    # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+When a tier is removed, its priority range is merged into the adjacent
+tier. The merge direction is always upward (the tier below expands),
+except when the lowest tier is removed — in that case the tier above
+shifts its starting priority down to -1 to maintain full range coverage.
+
+::
+
+    Initial state:
+    Name             Idx   PrioStart   PrioEnd
+    SSD              2     100         32767
+    HDD              1     50          99
+    NET              0     -1          49
+
+    # echo "-SSD" > /sys/kernel/mm/swap/tiers
+
+    Name             Idx   PrioStart   PrioEnd
+    HDD              1     50          32767       <- merged with SSD's range
+    NET              0     -1          49
+
+    # echo "-NET" > /sys/kernel/mm/swap/tiers
+
+    Name             Idx   PrioStart   PrioEnd
+    HDD              1     -1          32767       <- shifted down to -1
+
+**4. Interaction with Active Swap Devices**
+
+If a swap device is active (swapon), the tier covering that device's
+priority cannot be removed. Splitting the active tier's range is only
+allowed above the device's priority.
+
+Assume a swap device is active at priority 60 (inside 'HDD' tier).
+
+::
+
+    # swapon -p 60 /dev/zram0
+
+    Name             Idx   PrioStart   PrioEnd
+    HDD              0     50          32767
+    NET              1     -1          49
+
+    # echo "-HDD" > /sys/kernel/mm/swap/tiers
+    -bash: echo: write error: Device or resource busy
+
+    # echo "+SSD:60" > /sys/kernel/mm/swap/tiers
+    -bash: echo: write error: Device or resource busy
+
+    # echo "+SSD:100" > /sys/kernel/mm/swap/tiers
+
+    Name             Idx   PrioStart   PrioEnd
+    SSD              2     100         32767
+    HDD              0     50          99          <- device (prio 60) stays here
+    NET              1     -1          49
MAINTAINERS +1 -0
index e94d1af17c39..4485a0650984 100644
@@ -17237,6 +17237,7 @@ L:	[email protected]
 S:	Maintained
 F:	Documentation/ABI/testing/sysfs-kernel-mm-swap
 F:	Documentation/mm/swap-table.rst
+F:	Documentation/mm/swap-tier.rst
 F:	include/linux/swap.h
 F:	include/linux/swapfile.h
 F:	include/linux/swapops.h
include/linux/swap.h +1 -0
index 4427c7aa16dc..2d647a8fa60f 100644
@@ -244,6 +244,7 @@ struct swap_info_struct {
 	struct percpu_ref users;	/* indicate and keep swap device valid. */
 	unsigned long	flags;		/* SWP_USED etc: see above */
 	signed short	prio;		/* swap priority of this type */
+	int tier_mask;			/* swap tier mask */
 	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
mm/swap_state.c +1 -1
index c18ff741f2e0..dcec1a6c92bd 100644
@@ -1095,7 +1095,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 		}
 	}
 
-	if (!swap_tiers_validate()) {
+	if (!swap_tiers_update()) {
 		ret = -EINVAL;
 		goto restore;
 	}
mm/swap_tier.c +89 -12
index ac7a3c2a48cb..6b57cadb3e95 100644
@@ -38,6 +38,8 @@ static LIST_HEAD(swap_tier_inactive_list);
 	(!list_is_first(&(tier)->list, &swap_tier_active_list) ? \
 	list_prev_entry((tier), list)->prio - 1 : SHRT_MAX)
 
+#define MASK_TO_TIER(mask) (&swap_tiers[__ffs((mask))])
+
 #define for_each_tier(tier, idx) \
 	for (idx = 0, tier = &swap_tiers[0]; idx < MAX_SWAPTIER; \
 		idx++, tier = &swap_tiers[idx])
@@ -59,6 +61,26 @@ static bool swap_tier_is_active(void)
 	return !list_empty(&swap_tier_active_list);
 }
 
+static bool swap_tier_prio_in_range(struct swap_tier *tier, short prio)
+{
+	if (tier->prio <= prio && TIER_END_PRIO(tier) >= prio)
+		return true;
+
+	return false;
+}
+
+static bool swap_tier_prio_is_used(short prio)
+{
+	struct swap_tier *tier;
+
+	for_each_active_tier(tier) {
+		if (tier->prio == prio)
+			return true;
+	}
+
+	return false;
+}
+
 static struct swap_tier *swap_tier_lookup(const char *name)
 {
 	struct swap_tier *tier;
@@ -99,6 +121,7 @@ void swap_tiers_init(void)
 	int idx;
 
 	BUILD_BUG_ON(BITS_PER_TYPE(int) < MAX_SWAPTIER);
+	BUILD_BUG_ON(MAX_SWAPTIER > TIER_DEFAULT_IDX);
 
 	for_each_tier(tier, idx) {
 		INIT_LIST_HEAD(&tier->list);
@@ -149,17 +172,29 @@ static struct swap_tier *swap_tier_prepare(const char *name, short prio)
 	return tier;
 }
 
-static int swap_tier_check_range(short prio)
+static int swap_tier_can_split_range(short new_prio)
 {
+	struct swap_info_struct *p;
 	struct swap_tier *tier;
 
 	lockdep_assert_held(&swap_lock);
 	lockdep_assert_held(&swap_tier_lock);
 
-	for_each_active_tier(tier) {
-		/* No overwrite */
-		if (tier->prio == prio)
-			return -EINVAL;
+	plist_for_each_entry(p, &swap_active_head, list) {
+		if (p->tier_mask == TIER_DEFAULT_MASK)
+			continue;
+
+		tier = MASK_TO_TIER(p->tier_mask);
+		if (!swap_tier_prio_in_range(tier, new_prio))
+			continue;
+
+		/*
+		 * Device sits in a tier that spans new_prio;
+		 * splitting here would reassign it to a
+		 * different tier.
+		 */
+		if (p->prio >= new_prio)
+			return -EBUSY;
 	}
 
 	return 0;
@@ -199,7 +234,11 @@ int swap_tiers_add(const char *name, int prio)
 	if (!swap_tier_validate_name(name))
 		return -EINVAL;
 
-	ret = swap_tier_check_range(prio);
+	/* No overwrite */
+	if (swap_tier_prio_is_used(prio))
+		return -EBUSY;
+
+	ret = swap_tier_can_split_range(prio);
 	if (ret)
 		return ret;
 
@@ -226,6 +265,11 @@ int swap_tiers_remove(const char *name)
 	if (!tier)
 		return -EINVAL;
 
+	/* Simulate adding a tier to check for conflicts */
+	ret = swap_tier_can_split_range(tier->prio);
+	if (ret)
+		return ret;
+
 	/* Removing DEF_SWAP_PRIO merges into the higher tier. */
 	if (!list_is_singular(&swap_tier_active_list)
 		&& tier->prio == DEF_SWAP_PRIO)
@@ -236,13 +280,15 @@ int swap_tiers_remove(const char *name)
 	return ret;
 }
 
-static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
 /*
- * XXX: When multiple operations (adds and removes) are submitted in a
- * single write, reverting each individually on failure is complex and
- * error-prone. Instead, snapshot the entire state beforehand and
- * restore it wholesale if any operation fails.
+ * XXX: Static global snapshot buffer for batch operations. Small
+ * and used once per write, so a static global is not bad.
+ * When multiple adds/removes are submitted in a single write,
+ * reverting each individually on failure is error-prone. Instead,
+ * snapshot beforehand and restore wholesale if any operation fails.
  */
+static struct swap_tier swap_tiers_snap[MAX_SWAPTIER];
+
 void swap_tiers_snapshot(void)
 {
 	BUILD_BUG_ON(sizeof(swap_tiers_snap) != sizeof(swap_tiers));
@@ -282,10 +328,30 @@ void swap_tiers_snapshot_restore(void)
 	}
 }
 
-bool swap_tiers_validate(void)
+void swap_tiers_assign_dev(struct swap_info_struct *swp)
 {
 	struct swap_tier *tier;
 
+	lockdep_assert_held(&swap_lock);
+
+	for_each_active_tier(tier) {
+		if (swap_tier_prio_in_range(tier, swp->prio)) {
+			swp->tier_mask = TIER_MASK(tier);
+			return;
+		}
+	}
+
+	swp->tier_mask = TIER_DEFAULT_MASK;
+}
+
+bool swap_tiers_update(void)
+{
+	struct swap_tier *tier;
+	struct swap_info_struct *swp;
+
+	lockdep_assert_held(&swap_lock);
+	lockdep_assert_held(&swap_tier_lock);
+
 	/*
 	 * Initial setting might not cover DEF_SWAP_PRIO.
 	 * Swap tier must cover the full range (DEF_SWAP_PRIO to SHRT_MAX).
@@ -298,5 +364,16 @@ bool swap_tiers_validate(void)
 			return false;
 	}
 
+	/*
+	 * If applied initially, the swap tier_mask may change
+	 * from the default value.
+	 */
+	plist_for_each_entry(swp, &swap_active_head, list) {
+		/* Tier is already configured */
+		if (swp->tier_mask != TIER_DEFAULT_MASK)
+			break;
+		swap_tiers_assign_dev(swp);
+	}
+
 	return true;
 }
mm/swap_tier.h +12 -1
index a1395ec02c24..3e355f857363 100644
@@ -5,8 +5,15 @@
 #include <linux/types.h>
 #include <linux/spinlock.h>
 
+/* Forward declarations */
+struct swap_info_struct;
+
 extern spinlock_t swap_tier_lock;
 
+#define TIER_ALL_MASK		(~0)
+#define TIER_DEFAULT_IDX	(31)
+#define TIER_DEFAULT_MASK	(1U << TIER_DEFAULT_IDX)
+
 /* Initialization and application */
 void swap_tiers_init(void);
 ssize_t swap_tiers_sysfs_show(char *buf);
@@ -16,5 +23,9 @@ int swap_tiers_remove(const char *name);
 
 void swap_tiers_snapshot(void);
 void swap_tiers_snapshot_restore(void);
-bool swap_tiers_validate(void);
+bool swap_tiers_update(void);
+
+/* Tier assignment */
+void swap_tiers_assign_dev(struct swap_info_struct *swp);
+
 #endif /* _SWAP_TIER_H */
mm/swapfile.c +0 -0
index ff567ad893a4..f3cff586cf30 100644

mm/swapfile.c static void _enable_swap_info(struct swap_info_struct *si) · line 3045

Quoted 5 lines
mm/swapfile.c +2 -0
@@ -3041,6 +3041,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 
 	/* Add back to available list */
 	add_to_avail_list(si, true);
+
+	swap_tiers_assign_dev(si);

Could we move the assignment before the device is added to the active and available lists?

After patch 4, swap allocation checks si->tier_mask while holding only swap_avail_lock. But here the new device is added to swap_avail_head before swap_tiers_assign_dev() initializes its mask.

That creates a small window where reclaim can see this swap_info_struct with a stale tier_mask. swap_info_struct instances are reused across swapoff/swapon, so the stale mask can come from the previous device that occupied this slot. A memcg allowed to use the old tier could then temporarily allocate from the newly enabled device even if that device belongs to a different tier.

Quoted 7 lines
mm/swapfile.c +0 -0
 }
 
 /*
Signature · 1 line
2.34.1
author reviewer
#4 lore raw

Re: [PATCH v10 2/6] mm: swap: associate swap devices with tiers

Quoted 19 lines · Usama Arif

mm/swapfile.c static void _enable_swap_info(struct swap_info_struct *si) · line 3045

Quoted 11 lines

#endif /* _SWAP_TIER_H */

mm/swapfile.c +2 -0
index ff567ad893a4..f3cff586cf30 100644
@@ -3041,6 +3041,8 @@ static void _enable_swap_info(struct swap_info_struct *si)
 
 	/* Add back to available list */
 	add_to_avail_list(si, true);
+
+	swap_tiers_assign_dev(si);

Could we move the assignment before the device is added to the active and available lists?

After patch 4, swap allocation checks si->tier_mask while holding only swap_avail_lock. But here the new device is added to swap_avail_head before swap_tiers_assign_dev() initializes its mask.

Quoted 6 lines

That creates a small window where reclaim can see this swap_info_struct with a stale tier_mask. swap_info_struct instances are reused across swapoff/swapon, so the stale mask can come from the previous device that occupied this slot. A memcg allowed to use the old tier could then temporarily allocate from the newly enabled device even if that device belongs to a different tier.

Hello Usama.
Thanks for the review!

Good catch! You're right.
the device becomes visible to swap_alloc_slow() the moment add_to_avail_list() drops swap_avail_lock, before the mask is assigned.

Will move the assignment before the list insertions in v11.

Youngjun

author reviewer
#5 lore raw

PATCH v10 3/6 mm: memcontrol: add interface for swap tier selection

In reply to Youngjun Park

Introduce memory.swap.tiers.max, a flat-keyed file listing each tier defined in /sys/kernel/mm/swap/tiers with its state, "max" (allowed, the default) or "0" (disabled). A tier is one bit in the cgroup's tier mask, so writing "<tier> max" or "<tier> 0" sets or clears that bit.

Since the current use case lacks amount control, it only supports "max" (on) and "0" (off). Therefore, it does not track per-tier swap usage, relying instead on a fast runtime bitmask check.

We maintain both mask and effective_mask. The effective_mask is strictly bounded by the parent (e.g., if a parent is "0", the child's effective state is "0" even if its mask is "max"). Maintaining this separately avoids costly cgroup tree traversals to check ancestors at runtime.

---
 Documentation/admin-guide/cgroup-v2.rst |  20 +++++
 Documentation/mm/swap-tier.rst          |   9 +++
 include/linux/memcontrol.h              |   5 ++
 mm/memcontrol.c                         |  67 ++++++++++++++++
 mm/swap_state.c                         |   5 +-
 mm/swap_tier.c                          | 102 +++++++++++++++++++++++-
 mm/swap_tier.h                          |  57 +++++++++++--
 7 files changed, 255 insertions(+), 10 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +20 -0
index 993446ab66d0..4b0b4f00ad6e 100644
@@ -1850,6 +1850,26 @@ The following nested keys are defined.
 	Swap usage hard limit.  If a cgroup's swap usage reaches this
 	limit, anonymous memory of the cgroup will not be swapped out.
 
+  memory.swap.tiers.max
+	A read-write flat-keyed file which exists on non-root
+	cgroups.  The default is "max" for every tier.
+
+	Limits the swap tiers this cgroup may swap to.  Tiers are
+	defined globally in /sys/kernel/mm/swap/tiers and listed here,
+	one per line. When read, the values are displayed in descending
+	order of the tiers (highest tier first)::
+
+	  <tier_1> max
+	  <tier_2> 0
+	  ...
+
+	Currently, only "max" and "0" are supported. "max" allows the
+	tier, "0" disables it.  Each write sets a single "<tier> max"
+	or "<tier> 0" pair.
+
+	A child may only narrow what its parent allows. A tier an
+	ancestor disabled stays disabled regardless of the value here.
+
   memory.swap.events
 	A read-only flat-keyed file which exists on non-root cgroups.
 	The following entries are defined.  Unless specified
Documentation/mm/swap-tier.rst +9 -0
index 0fb4a1153a67..addbc495de8c 100644
@@ -15,6 +15,15 @@ speed to fully utilize this feature. While the current implementation is
 integrated with cgroups, the concept is designed to be extensible for other
 subsystems in the future.
 
+Use case
+---------
+
+Users can perform selective swapping by choosing a swap tier assigned according
+to speed within a cgroup.
+
+For more information on cgroup v2, please refer to
+``Documentation/admin-guide/cgroup-v2.rst``.
+
 Priority Range
 --------------
 
include/linux/memcontrol.h +5 -0
index 8724ff417ad4..895da2cc1a69 100644
@@ -281,6 +281,11 @@ struct mem_cgroup {
 	struct lru_gen_mm_list mm_list;
 #endif
 
+#ifdef CONFIG_SWAP
+	int tier_mask;
+	int tier_effective_mask;
+#endif
+
 #ifdef CONFIG_MEMCG_V1
 	/* Legacy consumer-oriented counters */
 	struct page_counter kmem;		/* v1 only */
mm/memcontrol.c +67 -0

91 lines not on this page · raw message

mm/swap_state.c +3 -2
index dcec1a6c92bd..ba090fa80bf1 100644
@@ -1053,6 +1053,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 	char *p, *token, *name, *tmp;
 	int ret = 0;
 	short prio;
+	int mask = 0;
 
 	tmp = kstrdup(buf, GFP_KERNEL);
 	if (!tmp)
@@ -1085,7 +1086,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 				goto restore;
 			break;
 		case '-':
-			ret = swap_tiers_remove(token + 1);
+			ret = swap_tiers_remove(token + 1, &mask);
 			if (ret)
 				goto restore;
 			break;
@@ -1095,7 +1096,7 @@ static ssize_t tiers_store(struct kobject *kobj,
 		}
 	}
 
-	if (!swap_tiers_update()) {
+	if (!swap_tiers_update(mask)) {
 		ret = -EINVAL;
 		goto restore;
 	}
mm/swap_tier.c +100 -2

125 lines not on this page · raw message

mm/swap_tier.h +51 -6

73 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#6 lore raw

PATCH v10 4/6 mm: swap: filter swap allocation by memcg tier mask

In reply to Youngjun Park

Apply memcg tier effective mask during swap slot allocation to enforce per-cgroup swap tier restrictions.

The folio's effective mask is computed once and passed to the fast, slow and discard paths as a parameter, so all of them act on the same mask even if the memcg's mask changes concurrently.

In the fast path, check the percpu cached swap_info's tier_mask against the folio's effective mask. If it does not match, fall through to the slow path. In the slow path, skip swap devices whose tier_mask is not covered by the folio's effective mask. The discard fallback honors the mask too: otherwise it would drain the discard clusters of a device outside the folio's tiers and then loop back to allocate from a tier the memcg is not allowed to use.

This works correctly when there is only one non-rotational device in the system and no devices share the same priority. However, there are known limitations:

- When non-rotational devices are distributed across multiple tiers, and different memcgs are configured to use those distinct tiers, they may constantly overwrite the shared percpu swap cache. This cache thrashing leads to frequent fast path misses.

- Combined with the above issue, if same-priority devices exist among them, a percpu cache miss (overwritten by another memcg) forces the allocator to round-robin to the next device prematurely, even if the current cluster is not fully exhausted.

These edge cases do not affect the primary use case of directing swap traffic per cgroup. Further optimization is planned for future work.

---
 mm/swapfile.c | 24 +++++++++++++++++-------
 1 file changed, 17 insertions(+), 7 deletions(-)
mm/swapfile.c +17 -7
index f3cff586cf30..967399936108 100644
@@ -1359,7 +1359,7 @@ static bool get_swap_device_info(struct swap_info_struct *si)
  * Fast path try to get swap entries with specified order from current
  * CPU's swap entry pool (a cluster).
  */
-static bool swap_alloc_fast(struct folio *folio)
+static bool swap_alloc_fast(struct folio *folio, int mask)
 {
 	unsigned int order = folio_order(folio);
 	struct swap_cluster_info *ci;
@@ -1371,8 +1371,11 @@ static bool swap_alloc_fast(struct folio *folio)
 	 * so checking it's liveness by get_swap_device_info is enough.
 	 */
 	si = this_cpu_read(percpu_swap_cluster.si[order]);
+	if (!si || !swap_tiers_mask_test(si->tier_mask, mask))
+		return false;
+
 	offset = this_cpu_read(percpu_swap_cluster.offset[order]);
-	if (!si || !offset || !get_swap_device_info(si))
+	if (!offset || !get_swap_device_info(si))
 		return false;
 
 	ci = swap_cluster_lock(si, offset);
@@ -1389,13 +1392,16 @@ static bool swap_alloc_fast(struct folio *folio)
 }
 
 /* Rotate the device and switch to a new cluster */
-static void swap_alloc_slow(struct folio *folio)
+static void swap_alloc_slow(struct folio *folio, int mask)
 {
 	struct swap_info_struct *si, *next;
 
 	spin_lock(&swap_avail_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_avail_head, avail_list) {
+		if (!swap_tiers_mask_test(si->tier_mask, mask))
+			continue;
+
 		/* Rotate the device and switch to a new cluster */
 		plist_requeue(&si->avail_list, &swap_avail_head);
 		spin_unlock(&swap_avail_lock);
@@ -1429,7 +1435,7 @@ static void swap_alloc_slow(struct folio *folio)
  * Discard pending clusters in a synchronized way when under high pressure.
  * Return: true if any cluster is discarded.
  */
-static bool swap_sync_discard(void)
+static bool swap_sync_discard(int mask)
 {
 	bool ret = false;
 	struct swap_info_struct *si, *next;
@@ -1437,6 +1443,8 @@ static bool swap_sync_discard(void)
 	spin_lock(&swap_lock);
 start_over:
 	plist_for_each_entry_safe(si, next, &swap_active_head, list) {
+		if (!swap_tiers_mask_test(si->tier_mask, mask))
+			continue;
 		spin_unlock(&swap_lock);
 		if (get_swap_device_info(si)) {
 			if (si->flags & SWP_PAGE_DISCARD)
@@ -1736,6 +1744,7 @@ int folio_alloc_swap(struct folio *folio)
 {
 	unsigned int order = folio_order(folio);
 	unsigned int size = 1 << order;
+	int mask;
 
 	VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
 	VM_BUG_ON_FOLIO(!folio_test_uptodate(folio), folio);
@@ -1759,13 +1768,14 @@ int folio_alloc_swap(struct folio *folio)
 	}
 
 again:
+	mask = folio_tier_effective_mask(folio);
 	local_lock(&percpu_swap_cluster.lock);
-	if (!swap_alloc_fast(folio))
-		swap_alloc_slow(folio);
+	if (!swap_alloc_fast(folio, mask))
+		swap_alloc_slow(folio, mask);
 	local_unlock(&percpu_swap_cluster.lock);
 
 	if (!order && unlikely(!folio_test_swapcache(folio))) {
-		if (swap_sync_discard())
+		if (swap_sync_discard(mask))
 			goto again;
 	}
 
Signature · 1 line
2.34.1
author reviewer
#7 lore raw

PATCH v10 5/6 selftests/mm: add a swap tier configuration test

In reply to Youngjun Park

This commit adds a test program for the global swap tier interface at /sys/kernel/mm/swap/tiers. It checks the add, split and remove operations and the documented error and batch atomicity rules. It also checks that a tier with an active swap device cannot be removed until the device is swapped off. That device is a zram device, and the check is skipped when zram is not available.

Assisted-by Claude:claude-opus-4-8
---
 tools/testing/selftests/mm/.gitignore     |   1 +
 tools/testing/selftests/mm/Makefile       |   1 +
 tools/testing/selftests/mm/config         |   2 +
 tools/testing/selftests/mm/run_vmtests.sh |   5 +
 tools/testing/selftests/mm/swap_tier.c    | 337 ++++++++++++++++++++++
 5 files changed, 346 insertions(+)
 create mode 100644 tools/testing/selftests/mm/swap_tier.c
tools/testing/selftests/mm/.gitignore +1 -0
index 227476d8ff1f..94ee61a9914d 100644
@@ -44,6 +44,7 @@ hmm-tests
 memfd_secret
 soft-dirty
 split_huge_page_test
+swap_tier
 ksm_tests
 local_config.h
 local_config.mk
tools/testing/selftests/mm/Makefile +1 -0
index ee8def9b4c31..4e47191e06ff 100644
@@ -104,6 +104,7 @@ TEST_GEN_FILES += guard-regions
 TEST_GEN_FILES += merge
 TEST_GEN_FILES += rmap
 TEST_GEN_FILES += folio_split_race_test
+TEST_GEN_FILES += swap_tier
 
 ifneq ($(ARCH),arm64)
 TEST_GEN_FILES += soft-dirty
tools/testing/selftests/mm/config +2 -0
index 06f78bd232e2..de3752e1bbd2 100644
@@ -14,3 +14,5 @@ CONFIG_UPROBES=y
 CONFIG_MEMORY_FAILURE=y
 CONFIG_HWPOISON_INJECT=m
 CONFIG_PROC_MEM_ALWAYS_FORCE=y
+CONFIG_SWAP=y
+CONFIG_ZRAM=y
tools/testing/selftests/mm/run_vmtests.sh +5 -0
index a60b9f9f16e7..06d53fad82ba 100755
@@ -71,6 +71,8 @@ separated by spaces:
 	tests for VM_PFNMAP handling
 - process_madv
 	test for process_madv
+- swap_tier
+	test the /sys/kernel/mm/swap/tiers configuration interface
 - cow
 	test copy-on-write semantics
 - thp
@@ -352,6 +354,9 @@ CATEGORY="process_madv" run_test ./process_madv
 
 CATEGORY="vma_merge" run_test ./merge
 
+# swap tier configuration interface (/sys/kernel/mm/swap/tiers)
+CATEGORY="swap_tier" run_test ./swap_tier
+
 if [ -x ./memfd_secret ]
 then
 if [ -f /proc/sys/kernel/yama/ptrace_scope ]; then
tools/testing/selftests/mm/swap_tier.c new file +337 -0

337 lines not on this page · raw message

Signature · 1 line
2.34.1
author reviewer
#8 lore raw

PATCH v10 6/6 selftests/cgroup: add a swap tier routing test

In reply to Youngjun Park

This commit adds a test program for the per-cgroup swap tier control memory.swap.tiers.max. It checks the default mask, toggling a tier, rejection of invalid input, and that recreating a tier resets the mask. It also checks that a cgroup's pages swap only to an allowed tier, including across the parent and child hierarchy. The routing check uses two zram devices placed in different tiers.

Assisted-by Claude:claude-opus-4-8
---
 tools/testing/selftests/cgroup/.gitignore     |   1 +
 tools/testing/selftests/cgroup/Makefile       |   2 +
 tools/testing/selftests/cgroup/config         |   2 +
 .../selftests/cgroup/test_swap_tiers.c        | 509 ++++++++++++++++++
 4 files changed, 514 insertions(+)
 create mode 100644 tools/testing/selftests/cgroup/test_swap_tiers.c
tools/testing/selftests/cgroup/.gitignore +1 -0
index 952e4448bf07..77b8e6c3e592 100644
@@ -8,5 +8,6 @@ test_kill
 test_kmem
 test_memcontrol
 test_pids
+test_swap_tiers
 test_zswap
 wait_inotify
tools/testing/selftests/cgroup/Makefile +2 -0
index e01584c2189a..a98e3c414cd5 100644
@@ -16,6 +16,7 @@ TEST_GEN_PROGS += test_kill
 TEST_GEN_PROGS += test_kmem
 TEST_GEN_PROGS += test_memcontrol
 TEST_GEN_PROGS += test_pids
+TEST_GEN_PROGS += test_swap_tiers
 TEST_GEN_PROGS += test_zswap
 
 LOCAL_HDRS += $(selfdir)/clone3/clone3_selftests.h $(selfdir)/pidfd/pidfd.h
@@ -32,4 +33,5 @@ $(OUTPUT)/test_kill: $(LIBCGROUP_O)
 $(OUTPUT)/test_kmem: $(LIBCGROUP_O)
 $(OUTPUT)/test_memcontrol: $(LIBCGROUP_O)
 $(OUTPUT)/test_pids: $(LIBCGROUP_O)
+$(OUTPUT)/test_swap_tiers: $(LIBCGROUP_O)
 $(OUTPUT)/test_zswap: $(LIBCGROUP_O)
tools/testing/selftests/cgroup/config +2 -0
index 39f979690dd3..6086bb5bba97 100644
@@ -4,3 +4,5 @@ CONFIG_CGROUP_FREEZER=y
 CONFIG_CGROUP_SCHED=y
 CONFIG_MEMCG=y
 CONFIG_PAGE_COUNTER=y
+CONFIG_SWAP=y
+CONFIG_ZRAM=y
tools/testing/selftests/cgroup/test_swap_tiers.c new file +509 -0

509 lines not on this page · raw message

Signature · 1 line
2.34.1
#9 lore raw
In reply to Youngjun Park
Quoted 21 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Quoted 8 lines

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

* Zswap tiering [2]:
Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

author reviewer
#10 lore raw
Quoted 25 lines · Yosry Ahmed
Quoted 24 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

Quoted 8 lines
Quoted 5 lines · Zswap tiering

Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

Hello Yosry!

This series does not cover zswap as a tier yet.

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Does that approach sound reasonable to you?

#11 lore raw
Quoted 40 lines · Youngjun Park
Quoted 25 lines · Yosry Ahmed
Quoted 24 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

Quoted 8 lines
Quoted 5 lines · Zswap tiering

Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

Hello Yosry!

This series does not cover zswap as a tier yet.

Quoted 7 lines

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers? I wonder if introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior, but I would check all user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

author reviewer
#12 lore raw
Quoted 47 lines · Yosry Ahmed
Quoted 46 lines · Youngjun Park
Quoted 25 lines · Yosry Ahmed
Quoted 24 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

Quoted 8 lines
Quoted 5 lines · Zswap tiering

Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

Hello Yosry!

This series does not cover zswap as a tier yet.

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Quoted 4 lines
Quoted 1 line

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers?

Yes, that's correct.

Quoted 3 lines

I wonder if introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior,

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Fair point. Let me review this more and get back to you!

Thanks,
Youngjun

maintainer
#13 lore raw

Hi Youngjun,

Thanks for keep pushing this effort.

Quoted 59 lines · Youngjun Park
Quoted 47 lines · Yosry Ahmed
Quoted 46 lines · Youngjun Park
Quoted 25 lines · Yosry Ahmed
Quoted 24 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

Quoted 8 lines
Quoted 5 lines · Zswap tiering

Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

Hello Yosry!

This series does not cover zswap as a tier yet.

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Quoted 4 lines
Quoted 1 line

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers?

Yes, that's correct.

Quoted 3 lines

I wonder if introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior,

Quoted 8 lines

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that? If that is not too much, we can make that part of this series.

Shakeel

#14 lore raw
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines

Hello Yosry!

This series does not cover zswap as a tier yet.

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Quoted 4 lines
Quoted 1 line

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers?

Yes, that's correct.

Quoted 3 lines

I wonder if introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior,

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Quoted 6 lines
Quoted 1 line

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap. We also need to disallow zswap being the only tier as that combination cannot work without vswap.

I think this should be enough to support "zswap" as a tier and allow disabling/enabling zswap per-memcg (or globally?) through tiering.

In the future, if/when swap demotion is added, we need to figure out how that would work with zswap. For example, if pages should go to swap device A then swap device B, then an entry in zswap using a swap slot in device B should not skip device A and be written back directly to B. vswap would naturally give us a solution for this problem.

Quoted 1 line

If that is not too much, we can make that part of this series.

Yeah I think it's better if we agree on the design and zswap support before landing partial support.

maintainer
#15 lore raw

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Quoted 2 lines

We also need to disallow zswap being the only tier as that combination cannot work without vswap.

Do we need to do anything explicitly for this? I am assuming in a kernel with swap tier support, there always exist a swap tier if there is even a single swap device configured i.e. a tier with the full priority range.

Quoted 9 lines

I think this should be enough to support "zswap" as a tier and allow disabling/enabling zswap per-memcg (or globally?) through tiering.

In the future, if/when swap demotion is added, we need to figure out how that would work with zswap. For example, if pages should go to swap device A then swap device B, then an entry in zswap using a swap slot in device B should not skip device A and be written back directly to B. vswap would naturally give us a solution for this problem.

This seems reasonable to me. Punting demotion/writeback to future.

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 5 lines
Quoted 2 lines

If that is not too much, we can make that part of this series.

Yeah I think it's better if we agree on the design and zswap support before landing partial support.

#16 lore raw
Quoted 24 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Quoted 6 lines
Quoted 2 lines

We also need to disallow zswap being the only tier as that combination cannot work without vswap.

Do we need to do anything explicitly for this? I am assuming in a kernel with swap tier support, there always exist a swap tier if there is even a single swap device configured i.e. a tier with the full priority range.

At least on the memcg side we need to disallow writing 0 for all tiers except zswap. Not sure about the global tier configurations.

author reviewer
#17 lore raw
In reply to Shakeel Butt
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Quoted 11 lines
Quoted 2 lines

We also need to disallow zswap being the only tier as that combination cannot work without vswap.

Do we need to do anything explicitly for this? I am assuming in a kernel with swap tier support, there always exist a swap tier if there is even a single swap device configured i.e. a tier with the full priority range.

Quoted 4 lines

I think this should be enough to support "zswap" as a tier and allow disabling/enabling zswap per-memcg (or globally?) through tiering.

Quoted 7 lines
Quoted 5 lines

In the future, if/when swap demotion is added, we need to figure out how that would work with zswap. For example, if pages should go to swap device A then swap device B, then an entry in zswap using a swap slot in device B should not skip device A and be written back directly to B. vswap would naturally give us a solution for this problem.

This seems reasonable to me. Punting demotion/writeback to future.

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

Thanks,
Youngjun

#18 lore raw
Quoted 28 lines · Youngjun Park
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

Quoted 8 lines

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Agreed.

[..]

Quoted 6 lines

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 8 lines

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

My main concern is locking ourselves into an implementation without zswap as a tier. One of the appeals of swap tiering to me, from the zswap side, is eventually unifying zswap writeback as part of a more general swap demotion, and unifying zswap controls with more general swap tier controls.

maintainer
#19 lore raw
Quoted 57 lines · Yosry Ahmed
Quoted 28 lines · Youngjun Park
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

Quoted 8 lines

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Agreed.

[..]

Quoted 6 lines

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 8 lines

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

Quoted 5 lines

My main concern is locking ourselves into an implementation without zswap as a tier. One of the appeals of swap tiering to me, from the zswap side, is eventually unifying zswap writeback as part of a more general swap demotion, and unifying zswap controls with more general swap tier controls.

There is nothing locked. The major complexity will come when zswap/vswap/Xswap/ghostswap become independent swap devices for swap tiers. We are not there yet. But if you want a bool for zswap in swap tiers world, we can make it happen trivially.

Chris

maintainer
#20 lore raw
Quoted 64 lines · Chris Li
Quoted 57 lines · Yosry Ahmed
Quoted 28 lines · Youngjun Park
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

Quoted 8 lines

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Agreed.

[..]

Quoted 6 lines

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 8 lines

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

Quoted 5 lines

My main concern is locking ourselves into an implementation without zswap as a tier. One of the appeals of swap tiering to me, from the zswap side, is eventually unifying zswap writeback as part of a more general swap demotion, and unifying zswap controls with more general swap tier controls.

Quoted 4 lines

There is nothing locked. The major complexity will come when zswap/vswap/Xswap/ghostswap become independent swap devices for swap tiers. We are not there yet. But if you want a bool for zswap in swap tiers world, we can make it happen trivially.

Yes please let's do this. Can you also please comment on Youngjun's response at
https://lore.kernel.org/all/alcebHobUCUyO70k@yjaykim-PowerEdge-T330/ as well?

author reviewer
#21 lore raw
Quoted 65 lines · Shakeel Butt
Quoted 64 lines · Chris Li
Quoted 62 lines · Yosry Ahmed
Quoted 28 lines · Youngjun Park
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

Quoted 8 lines

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Agreed.

[..]

Quoted 6 lines

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 8 lines

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

My main concern is locking ourselves into an implementation without zswap as a tier. One of the appeals of swap tiering to me, from the zswap side, is eventually unifying zswap writeback as part of a more general swap demotion, and unifying zswap controls with more general swap tier controls.

Quoted 7 lines
Quoted 4 lines

There is nothing locked. The major complexity will come when zswap/vswap/Xswap/ghostswap become independent swap devices for swap tiers. We are not there yet. But if you want a bool for zswap in swap tiers world, we can make it happen trivially.

Yes please let's do this. Can you also please comment on Youngjun's response at
https://lore.kernel.org/all/alcebHobUCUyO70k@yjaykim-PowerEdge-T330/ as well?

Hi Shakeel, Chris, and Yosry,

Thanks for the review and replies. It seems we have reached a consensus.

In the next patch series, I will update the swap.tiers.max interface to allow controlling zswap with 0 or max.

Internally, I will handle it as a not perfect tier.
Anyway since the user-visible interface is designed with future zswap tiering in mind, this approach naturally accommodates future zswap tier integration, while remaining easy to drop to make it as a tier if it proves unpromising (remaining this interface only controls zswap on/off).

As a follow-up, we can continue discussing the tiering of zswap, vswap, Xswap, and ghostswap. If we proceed, making zswap an actual logical tier entity and organizing its relationship with other interfaces (i.e., complete tiering) will be deferred to a future patchset.

I will keep you updated if further discussion is needed during implementation. Please let me know if you have any other feedback!

Thanks,
Youngjun

maintainer
#22 lore raw
In reply to Youngjun Park
Quoted 20 lines · Youngjun Park
Quoted 18 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 15 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 7 lines
Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

Maybe we can handle zswap as an internally reserved tier which is always preferred over swap devices.

See my other email. Zswap shouldn't even be in the list of swap tiers to search for swap slots, so this priority is builtin we don't need to specify it.

Quoted 8 lines

I do not think there is a strong use case for grouping zswap together with a swap device that happens to use the highest priority. Also, using a visible priority value for zswap may have a small side effect that one priority value effectively becomes unavailable to users.

That said, if zswap is represented as a tier, I agree that it should be the top tier. The exact priority value or whether it is internally reserved should be adjustable when we implement it.

Just leave it alone and the order will happen naturally.

Quoted 25 lines
Quoted 11 lines
Quoted 2 lines

We also need to disallow zswap being the only tier as that combination cannot work without vswap.

Do we need to do anything explicitly for this? I am assuming in a kernel with swap tier support, there always exist a swap tier if there is even a single swap device configured i.e. a tier with the full priority range.

Quoted 4 lines

I think this should be enough to support "zswap" as a tier and allow disabling/enabling zswap per-memcg (or globally?) through tiering.

Quoted 7 lines
Quoted 5 lines

In the future, if/when swap demotion is added, we need to figure out how that would work with zswap. For example, if pages should go to swap device A then swap device B, then an entry in zswap using a swap slot in device B should not skip device A and be written back directly to B. vswap would naturally give us a solution for this problem.

This seems reasonable to me. Punting demotion/writeback to future.

I agree this is reasonable. I can treat the demotion/writeback interaction as future work and follow up separately.

Quoted 2 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 8 lines

I sent a note with some thoughts about making zswap a tier. Since Chris also raised concerns around this area, I think it would be better to discuss it a bit more and evaluate the details carefully.

At this point, there does not seem to be an immediate use case that requires zswap tiering in this series. So my preference is to keep this as future work for now, and follow up with an RFC after verifying that introducing zswap as a tier on top of this patchset does not cause problems.

The minimal change is just giving zswap a bool to enable it in a swap tier or not. Personally I don't think it is needed. But if others want it we can add it as one off thing.

Chris

maintainer
#23 lore raw
In reply to Shakeel Butt
Quoted 19 lines · Shakeel Butt

On Tue, Jul 14, 2026 at 01:52:14PM -0700, Yosry Ahmed wrote:
[...]

Quoted 11 lines
Quoted 3 lines

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

Quoted 4 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap.

Quoted 4 lines

Default tier if zswap is configured makes sense. Should zswap be treated as having 32767 (or maybe 32768) as priority as it sits infront of all swap devices today? Also whichever swap tier has priority range containing 32767, will have zswap in it.

You shouldn't need to hard code that priority for zswap. Zswap, if you make it a tier, it is a special case. It is different than other swap tier devices because when you search for a swap slot, you don't need to search zswap as a swap device now. So the priority of the swap device does not even come into play. Internally, zswap is already hardcoded to perform before the underlying swap device. It is not an order you can change in any way.

So we don't need that priority number. Just reserve a special name for zswap.

Quoted 6 lines
Quoted 2 lines

We also need to disallow zswap being the only tier as that combination cannot work without vswap.

Do we need to do anything explicitly for this? I am assuming in a kernel with swap tier support, there always exist a swap tier if there is even a single swap device configured i.e. a tier with the full priority range.

There is no need to do anything special now. The current zswap is bind to the underlying swap device. That is unless we use vswap/Xswap/ghost swapfile.

Quoted 11 lines
Quoted 9 lines

I think this should be enough to support "zswap" as a tier and allow disabling/enabling zswap per-memcg (or globally?) through tiering.

In the future, if/when swap demotion is added, we need to figure out how that would work with zswap. For example, if pages should go to swap device A then swap device B, then an entry in zswap using a swap slot in device B should not skip device A and be written back directly to B. vswap would naturally give us a solution for this problem.

This seems reasonable to me. Punting demotion/writeback to future.

Agree as well

Chris

Quoted 9 lines

Youngjun, what do you think? Is this reasonable amount of additional work or do you envision some complexity here?

Quoted 5 lines
Quoted 2 lines

If that is not too much, we can make that part of this series.

Yeah I think it's better if we agree on the design and zswap support before landing partial support.

author reviewer
#24 lore raw
In reply to Yosry Ahmed
Quoted 29 lines · Yosry Ahmed
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines

Hello Yosry!

This series does not cover zswap as a tier yet.

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Quoted 4 lines
Quoted 1 line

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers?

Yes, that's correct.

Quoted 3 lines

I wonder if introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior,

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Quoted 6 lines
Quoted 6 lines
Quoted 1 line

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that?

Hello Yosry, Shakeel,

I have been working through this in detail at the implementation level. For now I am adding on/off control of zswap through the tier interface.

Quoted 6 lines

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

This part works as you described, with no real issues.

Quoted 5 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap. We also need to disallow zswap being the only tier as that combination cannot work without vswap.

I tried to forbid this at the implementation level as you suggested , and that is where I ran into trouble.
A few code paths can end up with zswap as the only
tier, and they are awkward to handle. Below is each case and what handling it would take:

1) A write turns off the last device tier while zswap stays on. -> rejected with -EINVAL. The write does not take effect.

2) The last device tier is removed via /sys/kernel/mm/swap/tiers. -> the file goes empty, so zswap is not shown either.

3) A cgroup has zswap and one device tier on, and that tier is removed. -> the cgroup's zswap entry is reset to 0, which the user never asked for.

4) A cgroup has zswap and device tiers on, and swapoff empties them. -> the cgroup's zswap entry is reset to 0.

So the problem is that in (3) and (4), memory.swap.tiers.max has to change on its own independently of what the user wrote and it is not like just error handling situation as (1).

There is also a consistency point. memory.swap.tiers.max already accepts a child enabling a tier that its parent has disabled: the write succeeds, no error is returned, and the difference is resolved internally. The user's setting is kept as written, and only the effective behavior is constrained. By that logic, accepting a zswap-only setting and guaranteeing only that it cannot do anything would fit how the interface already behaves.

Having thought it over, I think one of these two directions would be better than enforcing the rule as above.

1. Allow zswap-only in memory.swap.tiers.max. As you say, it cannot work without vswap, so today the setting does nothing and there is nothing to prevent. Once vswap/xswap lands it becomes meaningful on its own, with no interface change needed.

2. Expose memory.swap.tiers.max.effective, like cpuset. We already track the user-set and the effective-set separately. Exposing the effective one would show that a zswap-only setting is not in effect, giving the user visibility instead of rewriting what they wrote. It would also help the parent-off/child-on case, where the child could see from the effective value that the tier is off.

What do you think? or any other ideas?

Thanks,
Youngjun

#25 lore raw
Quoted 59 lines

Hello Yosry, Shakeel,

I have been working through this in detail at the implementation level. For now I am adding on/off control of zswap through the tier interface.

Quoted 6 lines

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

This part works as you described, with no real issues.

Quoted 5 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap. We also need to disallow zswap being the only tier as that combination cannot work without vswap.

I tried to forbid this at the implementation level as you suggested , and that is where I ran into trouble.
A few code paths can end up with zswap as the only
tier, and they are awkward to handle. Below is each case and what handling it would take:

1) A write turns off the last device tier while zswap stays on. -> rejected with -EINVAL. The write does not take effect.

2) The last device tier is removed via /sys/kernel/mm/swap/tiers. -> the file goes empty, so zswap is not shown either.

3) A cgroup has zswap and one device tier on, and that tier is removed. -> the cgroup's zswap entry is reset to 0, which the user never asked for.

4) A cgroup has zswap and device tiers on, and swapoff empties them. -> the cgroup's zswap entry is reset to 0.

So the problem is that in (3) and (4), memory.swap.tiers.max has to change on its own independently of what the user wrote and it is not like just error handling situation as (1).

There is also a consistency point. memory.swap.tiers.max already accepts a child enabling a tier that its parent has disabled: the write succeeds, no error is returned, and the difference is resolved internally. The user's setting is kept as written, and only the effective behavior is constrained. By that logic, accepting a zswap-only setting and guaranteeing only that it cannot do anything would fit how the interface already behaves.

Having thought it over, I think one of these two directions would be better than enforcing the rule as above.

1. Allow zswap-only in memory.swap.tiers.max. As you say, it cannot work without vswap, so today the setting does nothing and there is nothing to prevent. Once vswap/xswap lands it becomes meaningful on its own, with no interface change needed.

Quoted 8 lines

2. Expose memory.swap.tiers.max.effective, like cpuset. We already track the user-set and the effective-set separately. Exposing the effective one would show that a zswap-only setting is not in effect, giving the user visibility instead of rewriting what they wrote. It would also help the parent-off/child-on case, where the child could see from the effective value that the tier is off.

What do you think? or any other ideas?

Hmm yes, seems like enforcing this rule comes with a lot of complexity for little gain. It's possible today to enable zswap without adding any swap devices, which would render zswap useless anyway. I suppose in the same way, we can just do (1) and allow users to disallow all tiers except zswap for a cgroup, which would effectively disable zswap in the same way (albeit a bit less clear). We can document this clearly in the memcg docs.

Shakeel, any objections to this?

maintainer
#26 lore raw
Quoted 76 lines · Yosry Ahmed
Quoted 59 lines

Hello Yosry, Shakeel,

I have been working through this in detail at the implementation level. For now I am adding on/off control of zswap through the tier interface.

Quoted 6 lines

From zswap's perspective, we just need to skip zswap is zswap as a tier is disallowed. Could just be a check in zswap_store() similar to the check if zswap is enabled. I am assuming that if a swap tier is disabled, nothing happens to the existing swapped out pages in this tier, but new pages do not get swapped out to it. This is the same behavior that happens if zswap is disabled at runtime.

This part works as you described, with no real issues.

Quoted 5 lines

From the tiering perspective, we need to accept "zswap" as a possible tier, or maybe creating it as a tier by default if zswap is configured would be better to avoid handling the case where the user doesn't create a tier for zswap. We also need to disallow zswap being the only tier as that combination cannot work without vswap.

I tried to forbid this at the implementation level as you suggested , and that is where I ran into trouble.
A few code paths can end up with zswap as the only
tier, and they are awkward to handle. Below is each case and what handling it would take:

1) A write turns off the last device tier while zswap stays on. -> rejected with -EINVAL. The write does not take effect.

2) The last device tier is removed via /sys/kernel/mm/swap/tiers. -> the file goes empty, so zswap is not shown either.

3) A cgroup has zswap and one device tier on, and that tier is removed. -> the cgroup's zswap entry is reset to 0, which the user never asked for.

4) A cgroup has zswap and device tiers on, and swapoff empties them. -> the cgroup's zswap entry is reset to 0.

So the problem is that in (3) and (4), memory.swap.tiers.max has to change on its own independently of what the user wrote and it is not like just error handling situation as (1).

There is also a consistency point. memory.swap.tiers.max already accepts a child enabling a tier that its parent has disabled: the write succeeds, no error is returned, and the difference is resolved internally. The user's setting is kept as written, and only the effective behavior is constrained. By that logic, accepting a zswap-only setting and guaranteeing only that it cannot do anything would fit how the interface already behaves.

Having thought it over, I think one of these two directions would be better than enforcing the rule as above.

1. Allow zswap-only in memory.swap.tiers.max. As you say, it cannot work without vswap, so today the setting does nothing and there is nothing to prevent. Once vswap/xswap lands it becomes meaningful on its own, with no interface change needed.

Quoted 8 lines

2. Expose memory.swap.tiers.max.effective, like cpuset. We already track the user-set and the effective-set separately. Exposing the effective one would show that a zswap-only setting is not in effect, giving the user visibility instead of rewriting what they wrote. It would also help the parent-off/child-on case, where the child could see from the effective value that the tier is off.

What do you think? or any other ideas?

Hmm yes, seems like enforcing this rule comes with a lot of complexity for little gain. It's possible today to enable zswap without adding any swap devices, which would render zswap useless anyway. I suppose in the same way, we can just do (1) and allow users to disallow all tiers except zswap for a cgroup, which would effectively disable zswap in the same way (albeit a bit less clear). We can document this clearly in the memcg docs.

Quoted 1 line

Shakeel, any objections to this?

Nopes, no objections.

maintainer
#27 lore raw
In reply to Youngjun Park
Quoted 17 lines · Youngjun Park

I tried to forbid this at the implementation level as you suggested , and that is where I ran into trouble.
A few code paths can end up with zswap as the only
tier, and they are awkward to handle. Below is each case and what handling it would take:

1) A write turns off the last device tier while zswap stays on. -> rejected with -EINVAL. The write does not take effect.

2) The last device tier is removed via /sys/kernel/mm/swap/tiers. -> the file goes empty, so zswap is not shown either.

3) A cgroup has zswap and one device tier on, and that tier is removed. -> the cgroup's zswap entry is reset to 0, which the user never asked for.

Quoted 6 lines

4) A cgroup has zswap and device tiers on, and swapoff empties them. -> the cgroup's zswap entry is reset to 0.

So the problem is that in (3) and (4), memory.swap.tiers.max has to change on its own independently of what the user wrote and it is not like just error handling situation as (1).

This kind of complexity is exactly the reason I object to having zswap as a tier in the first place. There is very little gain in having such control for zswap in the swap tier world and it introduces a lot of corner-case complexity.

In my mind, we should introduce a separate swap tier native memory tier. e.g. xswap. which has a clean conceptual break from zswap (though code can be shared). The xswap will behave like zram in terms of tiering, but be more integrated with MM like zswap.

I think the simplest answer is just to allow zswap as a special tier, meaning it only has an on/off control on the side. Allowing zswap as the only tier means the cgroup won't be able to use zswap because there are no real devices.

Quoted 8 lines

There is also a consistency point. memory.swap.tiers.max already accepts a child enabling a tier that its parent has disabled: the write succeeds, no error is returned, and the difference is resolved internally. The user's setting is kept as written, and only the effective behavior is constrained. By that logic, accepting a zswap-only setting and guaranteeing only that it cannot do anything would fit how the interface already behaves.

Quoted 7 lines

Having thought it over, I think one of these two directions would be better than enforcing the rule as above.

1. Allow zswap-only in memory.swap.tiers.max. As you say, it cannot work without vswap, so today the setting does nothing and there is nothing to prevent. Once vswap/xswap lands it becomes meaningful on its own, with no interface change needed.

Allow it on paper and accept that zswap is not a real tier. It is just a cache in front of other swap devices.
That is what I think as well.

Quoted 7 lines

2. Expose memory.swap.tiers.max.effective, like cpuset. We already track the user-set and the effective-set separately. Exposing the effective one would show that a zswap-only setting is not in effect, giving the user visibility instead of rewriting what they wrote. It would also help the parent-off/child-on case, where the child could see from the effective value that the tier is off.

Seems more complex than it is worth.

Chris

Quoted 6 lines

What do you think? or any other ideas?

Thanks,
Youngjun

author reviewer
#28 lore raw
In reply to Shakeel Butt
Quoted 3 lines · Shakeel Butt

Hi Youngjun,

Thanks for keep pushing this effort.

Hi Shakeel
Huge thanks again for suggestion.

Quoted 6 lines
Quoted 6 lines

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

At now, the answer is there are nothing breaks or changes as a first review. I think it'll be better to introduce zswap tier after this patch. (Details are going on the below)

Quoted 9 lines
Quoted 1 line

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that? If that is not too much, we can make that part of this series.

Shakeel

I tried to verify what changes once the zswap tier is introduced. Please correct me if I'm wrong!

First observation: before the zswap tier is introduced, some of this is already possible today:

- a "zswap-only" tier equivalent on cgroup: memory.zswap.writeback = 0
- disabling zswap entirely on cgroup: memory.zswap.max = 0
- disabling swap (and zswap) entirely on cgroup: memory.swap.max = 0

Given that, my take is that introducing the tier concept itself doesn't change any user-visible zswap behavior, since on/off control for the zswap tier is basically already possible today through other knobs. So this isn't really a user-facing problem; it's more something we need to think through carefully in the patch that introduces the tier concept itself. That's what the points below are about.

The cgroup interface knobs and the zswap-global knobs need a different kind of review, so let's start with the global ones. The zswap-global knobs (compressor, max_pool_percent, accept_threshold_percent, shrinker_enabled) look fine with tiers. They're zswap-internal tuning, orthogonal to the tier concept, so tiers shouldn't change their behavior. The remaining one, "enabled", does overlap in spirit with the tier on/off knob: enabled = false is basically the same as disabling zswap for every cgroup. But like the other overlaps below, this doesn't look like a real problem.

The part that needs more thought is the overlap between memory.zswap.max / memory.zswap.writeback, memory.swap.max, and the new tier on/off knob. Since all of the above is already possible today, adding a tier on top means we should make sure it doesn't create confusing overlap.

1) memory.swap.max vs. tier on/off

memory.swap.max = 0 disables swap (and zswap) entirely, and memory.swap.max = max leaves everything enabled, so one knob can make the other a no-op. I don't think this is confusing in practice; they're just two independent ways to reach the same end state.

Conceptually, the tier knob controls on/off/auto-scale, while memory.swap.max controls pure quantity, so they cover different concerns and can coexist. Even if the tier knob later takes a numeric value, that number would apply per tier, while swap.max is a total budget that doesn't care how the tiers split it up, so they still shouldn't step on each other.

2) memory.zswap.max vs. zswap tier on/off

memory.zswap.max = 0 looks semantically equivalent to turning the zswap tier off. Could memory.zswap.max simply be folded into memory.<zswap_tier_name>.max?

Yosry pointed out:

Quoted 4 lines
Quoted 2 lines

memory.zswap.max integrates naturally (it's
memory.<tier_name>.max).

Not really. memory.zswap.max is in terms of memory usage (compressed size), not swap usage (uncompressed size).

I'm not sure where the tier limit would actually apply here: is it checked in zswap_store() against the pre-compression size, while memory.zswap.max keeps tracking the compressed size? If so, once you give the tier limit a number, wouldn't its usage end up meaning basically the same thing as memory.zswap.max? For limit = 0 the two already look identical to me (both just block zswap usage). I'd like to understand what benefit a non-zero tier limit (uncompressed) gives us over the existing memory.zswap.max (compressed), once tiers exist. Let me know if I'm missing something here.

Also note the mixed states this creates:

- memory.zswap.max has room, but the tier is off, so zswap can't be used.
- memory.zswap.max = 0, but the tier is on, so zswap still can't be used.

Both knobs can independently block zswap usage, which seems worth flagging even if it's not harmful by itself.

On top of that, the tier knob is likely to stay on/off/auto-scale without a numeric limit, while memory.zswap.max only handles the numeric side, so the two should coexist as is. If the tier ever gains its own numeric limit, we'd need to revisit unifying it with memory.zswap.max.

3) memory.zswap.writeback vs. a "zswap-only" tier

memory.zswap.writeback = 0 can act like a zswap-only tier today, because zswap still borrows a slot from the real swap device to store data; writeback = 0 just skips writing that data out to the device . A tier restriction works differently: it applies at allocation time, so a zswap-only tier would block allocating any slot from another device. Since zswap currently depends on that very slot to store anything, this would break zswap instead of giving us a working zswap-only mode, so the tier restriction can't substitute for memory.zswap.writeback here.

With vswap, this changes: tier off and writeback = 0 converge to the same effect(right now the pure zswap only tier), since zswap no longer needs to borrow a slot from another device. Even so, memory.zswap.writeback still needs to stay, since vswap itself can be disabled at compile time or runtime, and we'd still need a way to express zswap-only behavior without it. I'd like to unify the two eventually, but for now they have to stay separate. I'll keep thinking about it.

How do you think?

thanks
Youngjun

#29 lore raw
Quoted 48 lines · Youngjun Park
Quoted 3 lines · Shakeel Butt

Hi Youngjun,

Thanks for keep pushing this effort.

Hi Shakeel
Huge thanks again for suggestion.

Quoted 6 lines
Quoted 6 lines

Right, that's the plan.

Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

At now, the answer is there are nothing breaks or changes as a first review. I think it'll be better to introduce zswap tier after this patch. (Details are going on the below)

Quoted 9 lines
Quoted 1 line

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that? If that is not too much, we can make that part of this series.

Shakeel

I tried to verify what changes once the zswap tier is introduced. Please correct me if I'm wrong!

First observation: before the zswap tier is introduced, some of this is already possible today:

- a "zswap-only" tier equivalent on cgroup: memory.zswap.writeback = 0
- disabling zswap entirely on cgroup: memory.zswap.max = 0
- disabling swap (and zswap) entirely on cgroup: memory.swap.max = 0

Given that, my take is that introducing the tier concept itself doesn't change any user-visible zswap behavior, since on/off control for the zswap tier is basically already possible today through other knobs. So this isn't really a user-facing problem; it's more something we need to think through carefully in the patch that introduces the tier concept itself. That's what the points below are about.

Quoted 9 lines

The cgroup interface knobs and the zswap-global knobs need a different kind of review, so let's start with the global ones. The zswap-global knobs (compressor, max_pool_percent, accept_threshold_percent, shrinker_enabled) look fine with tiers. They're zswap-internal tuning, orthogonal to the tier concept, so tiers shouldn't change their behavior. The remaining one, "enabled", does overlap in spirit with the tier on/off knob: enabled = false is basically the same as disabling zswap for every cgroup. But like the other overlaps below, this doesn't look like a real problem.

Agreed, I don't think there's a problem there.

Quoted 15 lines

The part that needs more thought is the overlap between memory.zswap.max / memory.zswap.writeback, memory.swap.max, and the new tier on/off knob. Since all of the above is already possible today, adding a tier on top means we should make sure it doesn't create confusing overlap.

1) memory.swap.max vs. tier on/off

memory.swap.max = 0 disables swap (and zswap) entirely, and memory.swap.max = max leaves everything enabled, so one knob can make the other a no-op. I don't think this is confusing in practice; they're just two independent ways to reach the same end state.

Quoted 6 lines

Conceptually, the tier knob controls on/off/auto-scale, while memory.swap.max controls pure quantity, so they cover different concerns and can coexist. Even if the tier knob later takes a numeric value, that number would apply per tier, while swap.max is a total budget that doesn't care how the tiers split it up, so they still shouldn't step on each other.

The memory.swap.max interaction with tier on/off would be the same as memory.max with memory tiers AFAICT. We just need to make sure we remain consistent.

Quoted 8 lines

2) memory.zswap.max vs. zswap tier on/off

memory.zswap.max = 0 looks semantically equivalent to turning the zswap tier off. Could memory.zswap.max simply be folded into memory.<zswap_tier_name>.max?

Yosry pointed out:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

memory.zswap.max integrates naturally (it's
memory.<tier_name>.max).

Not really. memory.zswap.max is in terms of memory usage (compressed size), not swap usage (uncompressed size).

I'm not sure where the tier limit would actually apply here: is it checked in zswap_store() against the pre-compression size, while memory.zswap.max keeps tracking the compressed size?

As the tier limit can only be 0 or max for now, I don't think this is a problem. zswap_store() would just check that the tier is not disabled.

Quoted 2 lines

If so, once you give the tier limit a number, wouldn't its usage end up meaning basically the same thing as memory.zswap.max?

If we allow values other than 0/max, I think for zswap that would be the limit on the uncompressed size to remain consistent with other swap tiers. So it would be different from memory.zswap.max.

Quoted 5 lines

For limit = 0 the two already look identical to me (both just block zswap
usage). I'd like to understand what benefit a non-zero tier limit (uncompressed) gives us over the existing memory.zswap.max (compressed), once tiers exist. Let me know if I'm missing something here.

Yeah limit=0 and limit=max would behave similarly for both. If we do support other limits, the meaning would be different. Since most usage of zswap.max is either setting it to 0/max, we can check if we can deprecate that interface once we add zswap as a tier.

Quoted 18 lines

Also note the mixed states this creates:

- memory.zswap.max has room, but the tier is off, so zswap can't be used.
- memory.zswap.max = 0, but the tier is on, so zswap still can't be used.

Both knobs can independently block zswap usage, which seems worth flagging even if it's not harmful by itself.

On top of that, the tier knob is likely to stay on/off/auto-scale without a numeric limit, while memory.zswap.max only handles the numeric side, so the two should coexist as is. If the tier ever gains its own numeric limit, we'd need to revisit unifying it with memory.zswap.max.

3) memory.zswap.writeback vs. a "zswap-only" tier

Quoted 9 lines

memory.zswap.writeback = 0 can act like a zswap-only tier today, because zswap still borrows a slot from the real swap device to store data; writeback = 0 just skips writing that data out to the device . A tier restriction works differently: it applies at allocation time, so a zswap-only tier would block allocating any slot from another device. Since zswap currently depends on that very slot to store anything, this would break zswap instead of giving us a working zswap-only mode, so the tier restriction can't substitute for memory.zswap.writeback here.

Yes, the semantics of zswap.writeback would be different than using zswap as the only tier, but we shouldn't allow zswap as the only tier anyway as I mentioned.

Quoted 2 lines

With vswap, this changes: tier off and writeback = 0 converge to the same effect(right now the pure zswap only tier),

Yes.

Quoted 6 lines

since zswap no longer needs to borrow a slot from another device. Even so, memory.zswap.writeback still needs to stay, since vswap itself can be disabled at compile time or runtime, and we'd still need a way to express zswap-only behavior without it. I'd like to unify the two eventually, but for now they have to stay separate. I'll keep thinking about it.

The goal is to eventually make zswap depend on vswap. Once we have that, we can deprecate zswap.writeback in favor of the tiering interface.

Quoted 5 lines

How do you think?

thanks
Youngjun

maintainer
#30 lore raw
In reply to Youngjun Park
Quoted 11 lines · Youngjun Park
Quoted 3 lines · Shakeel Butt

Hi Youngjun,

Thanks for keep pushing this effort.

Hi Shakeel
Huge thanks again for suggestion.

Quoted 2 lines
Quoted 2 lines

Right, that's the plan.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

but I would check all
user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

At now, the answer is there are nothing breaks or changes as a first review. I think it'll be better to introduce zswap tier after this patch.

I am fine with that.

Quoted 54 lines

(Details are going on the below)

Quoted 9 lines
Quoted 1 line

Fair point. Let me review this more and get back to you!

Please do report back what you find.

Yosry, what is needed to enable zswap as a swap tier? What will be the minimum requirements for that? If that is not too much, we can make that part of this series.

Shakeel

I tried to verify what changes once the zswap tier is introduced. Please correct me if I'm wrong!

First observation: before the zswap tier is introduced, some of this is already possible today:

- a "zswap-only" tier equivalent on cgroup: memory.zswap.writeback = 0
- disabling zswap entirely on cgroup: memory.zswap.max = 0
- disabling swap (and zswap) entirely on cgroup: memory.swap.max = 0

Given that, my take is that introducing the tier concept itself doesn't change any user-visible zswap behavior, since on/off control for the zswap tier is basically already possible today through other knobs. So this isn't really a user-facing problem; it's more something we need to think through carefully in the patch that introduces the tier concept itself. That's what the points below are about.

The cgroup interface knobs and the zswap-global knobs need a different kind of review, so let's start with the global ones. The zswap-global knobs (compressor, max_pool_percent, accept_threshold_percent, shrinker_enabled) look fine with tiers. They're zswap-internal tuning, orthogonal to the tier concept, so tiers shouldn't change their behavior. The remaining one, "enabled", does overlap in spirit with the tier on/off knob: enabled = false is basically the same as disabling zswap for every cgroup. But like the other overlaps below, this doesn't look like a real problem.

The part that needs more thought is the overlap between memory.zswap.max / memory.zswap.writeback, memory.swap.max, and the new tier on/off knob. Since all of the above is already possible today, adding a tier on top means we should make sure it doesn't create confusing overlap.

1) memory.swap.max vs. tier on/off

memory.swap.max = 0 disables swap (and zswap) entirely, and memory.swap.max = max leaves everything enabled, so one knob can make the other a no-op. I don't think this is confusing in practice; they're just two independent ways to reach the same end state.

Quoted 6 lines

Conceptually, the tier knob controls on/off/auto-scale, while memory.swap.max controls pure quantity, so they cover different concerns and can coexist. Even if the tier knob later takes a numeric value, that number would apply per tier, while swap.max is a total budget that doesn't care how the tiers split it up, so they still shouldn't step on each other.

Ack.

Quoted 24 lines

2) memory.zswap.max vs. zswap tier on/off

memory.zswap.max = 0 looks semantically equivalent to turning the zswap tier off. Could memory.zswap.max simply be folded into memory.<zswap_tier_name>.max?

Yosry pointed out:

Quoted 4 lines
Quoted 2 lines

memory.zswap.max integrates naturally (it's
memory.<tier_name>.max).

Not really. memory.zswap.max is in terms of memory usage (compressed size), not swap usage (uncompressed size).

I'm not sure where the tier limit would actually apply here: is it checked in zswap_store() against the pre-compression size, while memory.zswap.max keeps tracking the compressed size? If so, once you give the tier limit a number, wouldn't its usage end up meaning basically the same thing as memory.zswap.max? For limit = 0 the two already look identical to me (both just block zswap usage). I'd like to understand what benefit a non-zero tier limit (uncompressed) gives us over the existing memory.zswap.max (compressed), once tiers exist. Let me know if I'm missing something here.

Quoted 6 lines

Also note the mixed states this creates:

- memory.zswap.max has room, but the tier is off, so zswap can't be used.
- memory.zswap.max = 0, but the tier is on, so zswap still can't be used.

I this is more of the zswap interal behavior, I don't have a strong preference. I will let the zswap experts comment on that.

Quoted 9 lines

Both knobs can independently block zswap usage, which seems worth flagging even if it's not harmful by itself.

On top of that, the tier knob is likely to stay on/off/auto-scale without a numeric limit, while memory.zswap.max only handles the numeric side, so the two should coexist as is. If the tier ever gains its own numeric limit, we'd need to revisit unifying it with memory.zswap.max.

Ack.

Quoted 12 lines

3) memory.zswap.writeback vs. a "zswap-only" tier

memory.zswap.writeback = 0 can act like a zswap-only tier today, because zswap still borrows a slot from the real swap device to store data; writeback = 0 just skips writing that data out to the device . A tier restriction works differently: it applies at allocation time, so a zswap-only tier would block allocating any slot from another device. Since zswap currently depends on that very slot to store anything, this would break zswap instead of giving us a working zswap-only mode, so the tier restriction can't substitute for memory.zswap.writeback here.

zswap.writeback = 0 will disable the other swap device after that. That is the incompatible part of the interface with swap tiers. I think if you want to use swap tiers other than zswap, you will have to leave the writeback = true.

Quoted 10 lines

With vswap, this changes: tier off and writeback = 0 converge to the same effect(right now the pure zswap only tier), since zswap no longer needs to borrow a slot from another device. Even so, memory.zswap.writeback still needs to stay, since vswap itself can be disabled at compile time or runtime, and we'd still need a way to express zswap-only behavior without it. I'd like to unify the two eventually, but for now they have to stay separate. I'll keep thinking about it.

How do you think?

I think with vswap/Xswap/ghostswapfile it is a totally different world. It would better to create conceptually separate entities rather than cling to the current zswap. They can share code, but the mental concept is different. The zswap behavior of borrowing a swap slot from the other swap device will likely need to change there. It is another can of worms I don't want to open right now. We should defer that to later.

Chris

maintainer
#31 lore raw
In reply to Yosry Ahmed
Quoted 10 lines · Yosry Ahmed
Quoted 7 lines

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers? I wonder if

Zswap is not part of the tiers exactly because it sits in front of all swap devices (tiers) and uses a different control to enable or disable it.
Let's not combine these two; let zswap use its own existing cgroup control interface.

Quoted 6 lines

introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior, but I would check all user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Chris

#32 lore raw
Quoted 26 lines · Chris Li
Quoted 10 lines · Yosry Ahmed
Quoted 7 lines

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers? I wonder if

Zswap is not part of the tiers exactly because it sits in front of all swap devices (tiers) and uses a different control to enable or disable it.
Let's not combine these two; let zswap use its own existing cgroup control interface.

Quoted 6 lines

introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior, but I would check all user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Quoted 4 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

An advantage of adding zswap as a tier right away is the proactive writeback use case. It naturally fits in the tiering framework as proactive demotion between swap tiers, which I expect may be useful in non-zswap use cases as well. Without zswap as a tier, we'll have to use a different interface for proactive writeback, and then if/when zswap becomes a tier, we'll have multiple ways to do proactive writeback which isn't ideal.

maintainer
#33 lore raw
Quoted 33 lines · Yosry Ahmed
Quoted 26 lines · Chris Li
Quoted 10 lines · Yosry Ahmed
Quoted 7 lines

My plan is to land the swap tier infrastructure together with the first use case (cgroup-based swap control) first, and then follow up with zswap tier support in a subsequent series, continuing the discussions we've had above.
(I mentioned on cover letter, right above the overview section)

Does that approach sound reasonable to you?

How does swap tiering work with zswap in the current series? I assume zswap is just enabled for all devices in all tiers? I wonder if

Zswap is not part of the tiers exactly because it sits in front of all swap devices (tiers) and uses a different control to enable or disable it.
Let's not combine these two; let zswap use its own existing cgroup control interface.

Quoted 6 lines

introducing zswap as a tier after the fact changes user-visible behavior. I guess if zswap will be introduced with a default "max" value it will more-or-less be the same behavior, but I would check all user-visible behaviors related to zswap (e.g. interaction with other zswap interfaces) to make sure nothing breaks or changes in a meaningful way when zswap is introduced as a tier later.

Quoted 4 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Quoted 8 lines

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

I'm afraid your zswap user interface will have to break. I don't see a way around breaking your zswap user interface to fit the swap tiering. Once we move to the swap tier world, I don't think we should continue using zswap.writeback to control the tier write back behavior. We will need to rethink this new world.

Quoted 7 lines

An advantage of adding zswap as a tier right away is the proactive writeback use case. It naturally fits in the tiering framework as proactive demotion between swap tiers, which I expect may be useful in non-zswap use cases as well. Without zswap as a tier, we'll have to use a different interface for proactive writeback, and then if/when zswap becomes a tier, we'll have multiple ways to do proactive writeback which isn't ideal.

I am looking forward to abstracting a more common write back behavior in the swap tier world. The classic zswap behavior will be preserved.

Chris

#34 lore raw
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Quoted 8 lines

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

Quoted 5 lines

I'm afraid your zswap user interface will have to break. I don't see a way around breaking your zswap user interface to fit the swap tiering. Once we move to the swap tier world, I don't think we should continue using zswap.writeback to control the tier write back behavior. We will need to rethink this new world.

I wasn't talking about the existing zswap interfaces. I want to make sure that if we introduce tiering initially without zswap as a tier, then add zswap as a tier, the semantics of tiering and user-visible zswap behavior doesn't break.

That being said, the existing zswap interfaces don't have to break with tiering, why do they? We may end up with redundant interfaces, which is unfortunate, and we can work to deprecate some of them over time. But I don't see why we have to break them?

Quoted 11 lines
Quoted 8 lines

An advantage of adding zswap as a tier right away is the proactive writeback use case. It naturally fits in the tiering framework as proactive demotion between swap tiers, which I expect may be useful in non-zswap use cases as well. Without zswap as a tier, we'll have to use a different interface for proactive writeback, and then if/when zswap becomes a tier, we'll have multiple ways to do proactive writeback which isn't ideal.

I am looking forward to abstracting a more common write back behavior in the swap tier world. The classic zswap behavior will be preserved.

Right, but this requires zswap being a tier, which you seem to be opposed to :)

maintainer
#35 lore raw
Quoted 21 lines · Yosry Ahmed
Quoted 15 lines
Quoted 6 lines
Quoted 5 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Quoted 8 lines

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

Quoted 5 lines

I'm afraid your zswap user interface will have to break. I don't see a way around breaking your zswap user interface to fit the swap tiering. Once we move to the swap tier world, I don't think we should continue using zswap.writeback to control the tier write back behavior. We will need to rethink this new world.

Quoted 4 lines

I wasn't talking about the existing zswap interfaces. I want to make sure that if we introduce tiering initially without zswap as a tier, then add zswap as a tier, the semantics of tiering and user-visible zswap behavior doesn't break.

No, the user visible part of zswap must break because zswap currently sits in front of every swapfile.
I don't see any other way around it.

If you do know how zswap can interact with swap.tiers without breaking the user interface, make a formal proposal and lay out all the details. I did that exercise myself and I my conclusion is that it is better to accept zswap is the classic behavior without burdening the unified swap tier too much.

Quoted 2 lines

That being said, the existing zswap interfaces don't have to break with tiering, why do they? We may end up with redundant interfaces,

Because zswap does not have its own swap device, it borrows the swap slot from the underlying swap device. That behavior is unique to zswap and none of the other swap tiers have that.

Quoted 2 lines

which is unfortunate, and we can work to deprecate some of them over time. But I don't see why we have to break them?

I don't want to special case the zswap for from the swap tiers.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

An advantage of adding zswap as a tier right away is the proactive writeback use case. It naturally fits in the tiering framework as proactive demotion between swap tiers, which I expect may be useful in non-zswap use cases as well. Without zswap as a tier, we'll have to use a different interface for proactive writeback, and then if/when zswap becomes a tier, we'll have multiple ways to do proactive writeback which isn't ideal.

Quoted 4 lines
Quoted 2 lines

I am looking forward to abstracting a more common write back behavior in the swap tier world. The classic zswap behavior will be preserved.

Right, but this requires zswap being a tier, which you seem to be opposed to :)

It does not need to as far as I can tell. People who want classic zswap can use zswap as it is. I am thinking the swap ops provide some interface for classic zswap to use, but zswap itself is not a tier because it does not own swap devices.

Chris

#36 lore raw
Quoted 23 lines · Chris Li
Quoted 21 lines · Yosry Ahmed
Quoted 20 lines
Quoted 6 lines
Quoted 5 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Quoted 8 lines

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

I'm afraid your zswap user interface will have to break. I don't see a way around breaking your zswap user interface to fit the swap tiering. Once we move to the swap tier world, I don't think we should continue using zswap.writeback to control the tier write back behavior. We will need to rethink this new world.

Quoted 8 lines
Quoted 4 lines

I wasn't talking about the existing zswap interfaces. I want to make sure that if we introduce tiering initially without zswap as a tier, then add zswap as a tier, the semantics of tiering and user-visible zswap behavior doesn't break.

No, the user visible part of zswap must break because zswap currently sits in front of every swapfile.
I don't see any other way around it.

If zswap becomes usable independent from a swapfile, it's mostly transparent to current users.

Quoted 5 lines

If you do know how zswap can interact with swap.tiers without breaking the user interface, make a formal proposal and lay out all the details. I did that exercise myself and I my conclusion is that it is better to accept zswap is the classic behavior without burdening the unified swap tier too much.

Today pages go to zswap, and when the limit is hit they get written back to a swap device. If zswap is a separate tier, pages will still go to zswap and then when the limit is hit they get written back to a swap device. In both cases, zswap writeback can be disabled.

Can you share the findings from your exercise, and why zswap being a tier is a burden? Any specific examples?

Quoted 7 lines
Quoted 3 lines

That being said, the existing zswap interfaces don't have to break with tiering, why do they? We may end up with redundant interfaces,

Because zswap does not have its own swap device, it borrows the swap slot from the underlying swap device. That behavior is unique to zswap and none of the other swap tiers have that.

Yes, but we are heading in the direction of removing that restriction. But I also don't see the connection between that and the current interfaces breaking. Do you have any concrete examples?

Quoted 5 lines
Quoted 3 lines

which is unfortunate, and we can work to deprecate some of them over time. But I don't see why we have to break them?

I don't want to special case the zswap for from the swap tiers.

Neither do I, but I fail to see where the special casing is, beyond the fact that zswap is not a swapfile but an in-memory swap.

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines

An advantage of adding zswap as a tier right away is the proactive writeback use case. It naturally fits in the tiering framework as proactive demotion between swap tiers, which I expect may be useful in non-zswap use cases as well. Without zswap as a tier, we'll have to use a different interface for proactive writeback, and then if/when zswap becomes a tier, we'll have multiple ways to do proactive writeback which isn't ideal.

I am looking forward to abstracting a more common write back behavior in the swap tier world. The classic zswap behavior will be preserved.

Quoted 6 lines
Quoted 1 line

Right, but this requires zswap being a tier, which you seem to be opposed to :)

It does not need to as far as I can tell. People who want classic zswap can use zswap as it is. I am thinking the swap ops provide some interface for classic zswap to use, but zswap itself is not a tier because it does not own swap devices.

A swap tier represents a class of swap devices or swap files with specific characteristics. Zswap satisfies that, although it's not really a swap device/file.

maintainer
#37 lore raw
Quoted 43 lines · Yosry Ahmed
Quoted 23 lines · Chris Li
Quoted 21 lines · Yosry Ahmed
Quoted 20 lines
Quoted 6 lines
Quoted 5 lines

Zswap will not be introduced as a tier. The existing user interface makes zswap not exactly compatible with the tier ordering because it sits in front of every swapfile. If we change that, we break the user interface. I suggest we keep zswap working as it is now.

Quoted 8 lines

The goal from making zswap a swap tier is to have a single framework to configure swapping for a cgroup, instead of configuring zswap separately. Yes, zswap currently sits in front of all swap devices/tiers, but we are heading in the direction of changing that such that zswap is standalone, at which point it becomes more obviously a swap tier. If you want us to wait until that happens before adding zswap as a tier, I don't necessarily object, but I want to make sure that nothing will break if we add zswap as a tier later.

I'm afraid your zswap user interface will have to break. I don't see a way around breaking your zswap user interface to fit the swap tiering. Once we move to the swap tier world, I don't think we should continue using zswap.writeback to control the tier write back behavior. We will need to rethink this new world.

Quoted 8 lines
Quoted 4 lines

I wasn't talking about the existing zswap interfaces. I want to make sure that if we introduce tiering initially without zswap as a tier, then add zswap as a tier, the semantics of tiering and user-visible zswap behavior doesn't break.

No, the user visible part of zswap must break because zswap currently sits in front of every swapfile.
I don't see any other way around it.

If zswap becomes usable independent from a swapfile, it's mostly transparent to current users.

Quoted 5 lines

If you do know how zswap can interact with swap.tiers without breaking the user interface, make a formal proposal and lay out all the details. I did that exercise myself and I my conclusion is that it is better to accept zswap is the classic behavior without burdening the unified swap tier too much.

Quoted 7 lines

Today pages go to zswap, and when the limit is hit they get written back to a swap device. If zswap is a separate tier, pages will still go to zswap and then when the limit is hit they get written back to a swap device. In both cases, zswap writeback can be disabled.

Can you share the findings from your exercise, and why zswap being a tier is a burden? Any specific examples?

It is all good when you have only zswap and just one other tier of swap device. As soon as you add one more, e.g. zswap -> SSD -> HDD will make things complicate when zswap using the HDD slot write back direclty to HDD. Anyway, I find that this kind of highlevel hand waving discussion of pros and cons usually doesn't produce the useful outcome. If you have a detailed solution for how zswap works with multiple layers of swap tiers, Laying out all the details and even better, providing some incremental patches, is the better way to proceed.

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

That being said, the existing zswap interfaces don't have to break with tiering, why do they? We may end up with redundant interfaces,

Because zswap does not have its own swap device, it borrows the swap slot from the underlying swap device. That behavior is unique to zswap and none of the other swap tiers have that.

Yes, but we are heading in the direction of removing that restriction. But I also don't see the connection between that and the current interfaces breaking. Do you have any concrete examples?

See above. This topic has already seen a lot of heated discussion. Show me an incrementally mergeable patch; that would be a better way to move forward.

Chris

maintainer
#38 lore raw
In reply to Yosry Ahmed
Quoted 26 lines · Yosry Ahmed
Quoted 24 lines · Youngjun Park

This is the v10 series of the swap tier patchset.

v10 folds in the Sashiko review fixes for the selftests added in v9 and rebases onto the current mm-new. There are no functional changes to the core swap or memcg code since v9; see the changelog for details.

For context, the bulk of the series is unchanged since v8, with great thanks to Shakeel Butt and Yosry for the reviews and discussions [1] that shaped it. The main change in v8 was the interface change to use memory.swap.tiers.max with '0' (disable) and 'max' (enable) values. This mechanism was suggested by Shakeel and Yosry.

This change allows for future extensions to control swap between tiers and aligns better with existing memcg interfaces. It is confined to patch #3's user-facing interface; internally, patch #3 still uses the existing mask processing method, which is implementation-efficient.

We also discussed tier extensions. Thanks to Yosry, Nhat and Shakeel for their valuable feedback.

Here is a brief summary of our tentative conclusions. Please correct me if anything is misrepresented (details in references):

Quoted 8 lines
Quoted 5 lines · Zswap tiering

Zswap can itself be a tier (typically the fastest one). But, until vswap lands, zswap cannot be the only allowed tier,
since it still needs a physical device for allocation;
that restriction can be lifted once vswap is supported.

Does this series support zswap being a tier? I cannot find any mention of zswap in the patches.

In case I wasn't clear. As it is, zswap is not part of the tier and will not be.

Chris

maintainer
#39 lore raw
In reply to Youngjun Park

Zswap is a first-order swap destination with writeback semantics. The way the discussion around zswap has been going in this thread is disappointing, and I don't feel comfortable adding permanent user interfaces on this basis.

We will not be merging a memcg swap tier interface until the swap side has a story for indirection and backend migration.

author reviewer
#40 lore raw

On Wed, Jul 22, 2026 at 09:14:10AM -0400, Johannes Weiner wrote:

Hello Johannes!

Quoted 4 lines

Zswap is a first-order swap destination with writeback semantics. The way the discussion around zswap has been going in this thread is disappointing, and I don't feel comfortable adding permanent user interfaces on this basis.

I have given this a lot of thought. First, could you clarify exactly which part of the discussion or consensus makes you hesitate? Understanding this will help me re-evaluate my proposal.

To make sure we are on the same page, I would like to share my thoughts and vision for the present and future of the tier interface semantics for your further consideration.

The interface should provide the swap amount allocated to a tier and allow the use of the swap device defined by that tier via swap.tiers.max. Currently, it would only support 0 and 'max' (essentially on/off for explicit usage). Auto-demotion is planned for the future, and specifying exact capacity limits is still to be determined. (I have also reviewed potential interface collisions and duplications based on Yosry's guidance.)

From this perspective, zswap currently cannot exist as a standalone tier. It resides in RAM when allocated and borrows slots from other swap devices. (If all tiers except zswap are turned off, it is effectively the same as zswap being off, meaning there is no zswap-only tier.) The zswap writeback interface essentially leaves only the zswap first tier enabled by using other swap slots. Virtualized swap should solve this in the near future.

On the other hand, in systems without zswap (like ours), this situation does not occur. Allocation and deallocation depend strictly on swap priority, avoiding this contradiction. (While it might be possible to implement the tier interface only for !CONFIG_ZSWAP first, I abandoned this idea as it seemed unpromising.)

In my initial review, I considered creating the swap tier interface while excluding zswap configurations, even if zswap is present. From zswap's perspective, it acts as a first-order swap, and other tiers operate well according to their original tier semantics. Once virtualized swap arrives, we can open up the interface without any contradictions.

However, virtualized swap might be toggled on or off (at runtime or compile time), we must still account for zswap without virtualized swap anyway. There is also a possibility that the initial overall picture might seem misaligned until zswap tiering is officially introduced. It looks like we must care this case on the first stage.

Therefore, I decided to follow Yosry's suggestion. Since a zswap-only setup has contradictions, we initially considered removing user visibility as possible as we can (returning an error for unsupported interfaces).

Because the implementation is a bit tricky, we agreed on an on/off toggle for this first baby step, accompanied by detailed documentation so users are fully aware of what a zswap-only tier means.

As a result, I reviewed this direction and found it does not conflict with a future zswap tier. When virtualized swap arrives, the design will align perfectly, and we can consider making zswap a proper tier.

One thing that remains undecided, however, is zswap tiering, which is not yet fully agreed upon by everyone and should be discussed at the next stage.

Quoted 2 lines

We will not be merging a memcg swap tier interface until the swap side has a story for indirection and backend migration.

Regarding this point, is your position that an indirect layer (I see
this is virtualized swap) must be implemented first?
Or do you mean we need community consensus on the overall design before proceeding?
I believe our direction is set toward virtualized swap, but there are still some discussions regarding implementation details.

I would love to hear your thoughts. In the meantime, my preference is to proceed as discussed if we can agree on the future direction. If more discussion and design verification are needed, I am happy to follow up and address any concerns you might have.

However, if you feel the tier concept is premature, I would like to hear your thoughts and others' as well on a fallback option. We could add per-cgroup swap control in debugfs for now. If we need to scale back, we could either keep the swap tier design internally or drop it entirely, implementing per-swap control strictly for our specific use case. We could then revisit this as a primary use case when swap tier discussions resume.

What do you think? I would appreciate hearing everyone's thoughts on this.

Youngjun

maintainer
#41 lore raw

Hello Youngjun,

Quoted 17 lines · Youngjun Park

On Wed, Jul 22, 2026 at 09:14:10AM -0400, Johannes Weiner wrote:

Hello Johannes!

Quoted 4 lines

Zswap is a first-order swap destination with writeback semantics. The way the discussion around zswap has been going in this thread is disappointing, and I don't feel comfortable adding permanent user interfaces on this basis.

I have given this a lot of thought. First, could you clarify exactly which part of the discussion or consensus makes you hesitate? Understanding this will help me re-evaluate my proposal.

To make sure we are on the same page, I would like to share my thoughts and vision for the present and future of the tier interface semantics for your further consideration.

Quoted 6 lines

The interface should provide the swap amount allocated to a tier and allow the use of the swap device defined by that tier via swap.tiers.max. Currently, it would only support 0 and 'max' (essentially on/off for explicit usage). Auto-demotion is planned for the future, and specifying exact capacity limits is still to be determined. (I have also reviewed potential interface collisions and duplications based on Yosry's guidance.)

This is the part that makes me uncomfortable. The interface as proposed now is just a small subset of what should eventually be "swap tiers" - with limits, hierarchies and demotion rules. It seems to be an unordered blacklist/whitelist for swapfiles right now.

And how the final form looks like depends on the indirection layer.

As Chris says upthread, virtual swap will be a whole new world. But unfortunately interfaces are permanent. So I'm sorry, but it's premature to start tying us to an ABI direction here.

Quoted 5 lines
Quoted 2 lines

We will not be merging a memcg swap tier interface until the swap side has a story for indirection and backend migration.

Regarding this point, is your position that an indirect layer (I see this is virtualized swap) must be implemented first?

It is implemented. There have been patches floating around since the beginning of 2025. Initial proposals date back to 2023.

But we need settled consensus before we can start adding interfaces that are supposed to control that very layer long-term.